DEV Community: Hari Pranav A

A DevOps approach to building, Scalable Cloud-Native Data Engineering Applications

Hari Pranav A — Tue, 24 May 2022 11:20:02 +0000

In the last blog link we created a Flask application on an AWS EC2 instance and built custom data pipelines to interact with Data Engineering tools on AWS. This method of deployment has its own pitfalls, as the entire software development lifecycle which involves the maintenance of code, development of new features, collaboration and versioning is often difficult to handle.
Hence we need to shift to a DevOps approach which helps developers create end to end pipelines from development to testing to production which can deal with collaborative changes.

In this blog post we will be.

Creating a Docker image of the Flask Application
Publishing the image to AWS Container Registry and Running the code in AWS Container Service using AWS Fargate
Add a Load Balancer for the ECS deployment
Create an AWS Code Commit repository to push code
Configure AWS Code Build to build new changes from the Code Commit Repository
Configure Code Pipeline to automatically run steps 1 to 5 once the new commit is made to the Code Commit Repository

Creating a Docker image of the Flask Application:

We can create a simple flask application which can interact with AWS as shown in the blog post below.

Using Flask to create custom data pipelines

Once we are successful with the creation of the Flask application on Ec2 instance we then need to add a Dockerfile which is used to create a Docker image from the existing Flask application.

Create two new files called Dockerfile and requirements.txt in the same parent directory as shown below

Create the requirements.txt file which has all the packages to be installed on the Docker Image.

$ cd CLOUDAUDIT_DATA_WRANGLER

$ nano requirements.txt

Then Add the two package inside the file
```
Flask
awswrangler
```

Here we don't a specify the version and this ensures that the latest version of the packages get installed.

Add the following lines into the Dockerfile.

# syntax=docker/dockerfile:1

FROM python:3.8-slim-buster

WORKDIR /python-docker

COPY requirements.txt requirements.txt

RUN pip3 install -r requirements.txt

COPY . .

CMD [ "python3", "-m" , "flask", "run", "--host=0.0.0.0"]

The first line (# syntax=docker/dockerfile:1) tells what is the syntax to be used while parsing the Dockerfile and the location of the Docker Syntax file
The second line (FROM python:3.8-slim-buster) allows us to use an already existing base image for Docker.
The third line (WORKDIR /python-docker) tells docker which directory to use.
The fourth and fifth lines tells Docker to copy the contents of requirements.txt into a container image's requirements.txt file and then run the pip install command for all the packages and dependencies.
The sixth line (COPY . .) tells Docker to copy the remainder of the folders to be copied into Dockers container.
The last line with (-m) indicates to run the Flask app as a module and to make the container accessible to the browser.

Building the Docker Image:

When we run the command as shown below, it builds a container with the name python-docker. This can then be pushed to our Container Registry on AWS.

$ docker build --tag python-docker .

$ docker run -d -p 5000:5000 python-docker

Here "-d" will run it in detached mode and "-p" will expose the specific port.

Publishing the image to AWS Container Registry and Running the code in AWS Container Service using AWS Fargate

We will be using AWS Fargate which is a serverless compute engine for Amazon ECS that runs containers without the headache of managing the infrastructure.

Open the AWS console and then search for ECR

Click on Get Started and then Create Repository

Here the repository name is flask-app

Now click on push commands as shown in the screenshot shown below.

We have a list of commands that we need to run on our Ec2 instance which will push the docker image to ECR.

For Command 1 we need to attach a policy for the Ec2 instance to get the necessary permissions to authenticate with the ECR.

The links below show how we can attach a policy to the Ec2 machine.

Attach IAM to Ec2

Add ECR execution Policy

After we add the permissions we need not run the second command as we have already created the docker image.

Now for command 3 and 4 we need to run :

$ docker tag python-docker:xxxxxxxxxx/flask-app:latest

REPLACE THE XXXXXXXX with the statements from your Console

$docker push xxxxxxxxx/flask-app:latest

REPLACE THE XXXXXXXX with the statements from your Console

Here we are tagging the image python-docker and then pushing the image to ECR with the name as flask-app.

Creating a cluster
1. Search for ECS on the AWS Console, Click on create a new cluster, give a name and choose the default VPC and the subnets. Then choose AWS Fargate as shown in the image below.

Then the left pane choose Task Definition

In this give a unique name, Enter flask-app for the name of the container and the URI which can be found in the ECR screenshot as seen below.

For the port mapping choose port 5000 and 80. Then click on next. For the Environment choose Fargate and leave the other defaults as shown below.

Finally Review and click next. This will add the Container in ECR to Fargate.

Running the Task Definitions:

As shown in the screenshot below click on on the created Task Definition and click on Run

Choose the Existing Cluster as shown below

Choose the VPC and Subnet, create a new security group and click on deploy as shown below.

Navigate back to Clusters on the left then click on the cluster which has been created. Then Click on Tasks to get the ENI which has been shown in Black in the image below.

Click on ENI and also make note of the public IP address as shown below.

Click on security Group and Edit the inbound port rules to allow traffic from Port 5000 and 80 as shown below.

Once this is done Enter the PUBLIC IP from the ENI and the port 5000

Eg: http://publicIP:5000 in the URL and once this is entered YOU HAVE SUCCESFULLY LAUNCHED A DOCKER IMAGE ON THE ELASTIC CONTAINER SERVICE !!

Add a Load Balancer for the ECS deployment:

Open the EC2 Console and then choose the application load balancer as shown below.

Make sure to shift to the OLD AWS CONSOLE while doing this step as it will help to follow with the same steps as shown below.

Choose Application Load Balancer and click Create
Next, Configure it by giving a name and select the VPC and availability zones

Click Next, select Create a new security group and then click Next
Give a name to Target group, for Target type select IP and then click Next

Click Next, Review it and click Create
Once created note down the DNS name, which is the public address for the service

Creating a Fargate Service:

We will use the same task definition to create a Fargate Service

Go to Task Definitions in Amazon ECS, tick the radio button corresponding to the existing Task definition and click Actions and Create Service.

Choose Fargate as launch type, give it a name, do not change the Deployment type (Rolling update), and click Next.

Choose the subnets that we have configured in the load balancer

Choose Application load balancer for the load balancer type, and then click Add to load balancer

Select the Target group name that we have created in the Application load balancer and then click Next, Review it and then click Create Service.

Now we need to configure the Application load balancer security group that we have created earlier, go to the created Application load balancer and click on security groups and then click Edit inbound rules and add a Custom TCP rule with port 5000, as this is the internal port our application is configured in the flask application
Now, we can check our application is running by visiting the load balancer DNS name in a browser

It will most probably not run and there are very few guides to trouble shoot this issue. The Root cause for this is as follows:

Flow of the application before adding the load balancer:

Client -> URL -> Public IP of Fargate(port 5000 and 80 is opened to a specific IP) -> Response

Once we add the new load balancer and open the ports

Client -> URL -> Load Balancer -> Public IP of Fargate(Only port 5000 and 80 is opened to a specific IP) (PORT IS NOT OPENED TO THE LOAD BALANCER SECURITY GROUP)-> Response (As port 5000 is opened for the Security Grp)

Hence this can be seen in the documentation as shown below

To trouble shoot this issue, we need to OPEN PORT for the newly created Security Grp which is attached to the load balancer.

Open ECS and select your cluster which contains the newly created service as shown below.

Click on the Task and go to the ENI as shown below

Edit the Inbound rules and add the Security Grp which was given to the load-balancer as shown below and also open port 5000

Now if we open the open the load balancer page and check the DNS address in our browser we should be getting the FLASK APPLICATION.
Hurray we have successfully deployed the application on ECS and added a load balancer !!
Another way to know if the application is running successfully is to check the load balancer Target GROUP
Open the Ec2 console, navigate to load balancer and then check the TARGET GRP, if it is showing Unhealthy or Draining refer to the links below, it will most probably be the Inbound port rules which have not been opened !. In the screenshot shown below, we can see that it shows that the instance is healthy

If we are still getting any 504, or 503 errors refer to the links below:

aws

stackoverflow

Create an AWS Code Commit repository to push code:

Open the AWS console and search for CodeCommit, click on create a new Repository and give it a name and a description as shown in the image below.

Add a new file called buildspec.yml in the parent directory.

Now our application folder structure will look as per the screenshot shown below.

This file contains the commands which can be used to compile, test and package code.

ADD A BUILD SPEC FILE and push this repo to git.

Download and change according to the image above

We can see that the "image.json" is an artifact file which contains the container name and the image URI of the container pushed to ECR. We then use these in CodePipeline.
artifacts are created from the container name which is created earlier in EC2 which is flask-app

Since this file is a Yaml file we need to check spacing and make sure we follow the right number of tabs and spaces.

Once the repository is created we need to add specific permission and create Git credentials to access the CodeCommit repository.
Go to the IAM console, choose Users and select which User you want to configure for CodeCommit, and attach AWSCodeCommitPowerUser policy from the policies list and Review and then click Add Permission.

Configure Git using the blog post below:

Configure Git Locally

Follow the documentation below to Add Git Credentials for the AWS account. Once this is added, open the repository created in Code Commit and then run the following commands.

CD into your repository and run the commands as shown

Add the files in your new local repository. This stages them for the first commit.

$ git init

Commit the files that you've staged in your local repository.

$ git add .

$ git commit -m "First commit"

At the top of your GitHub repository's Quick Setup page, click to copy the remote repository URL.

In the Command prompt, add the URL for the remote repository where your local repository will be pushed.

Sets the new remote

$ git remote add origin

Refer to the screenshot below to find the URL:

Verifies the new remote URL

$ git remote -v

Push the changes to the master branch

$ git push origin master

Configure Code Pipeline to automatically run steps 1 to 5 once the new commit is made to the Code Commit Repository:

We can then use Code Pipeline to configure builds from CodeCommit to the ECR which will inturn run the image on ECS.

1.Go to CodePipeline and click on GetStarted

Choose the Service, Repo and the Click on Next. Select the Repo name and Create a new service Role as shown below

2.Choose the Source as AWS code Commit and Repo name from the dropdown. Then Choose the Master branch and click on Next

3.Next, for the build provider select CodeBuild and create a new build project, give it a name and configure it as follows

Make sure to Check the box: Privileged and make sure that the new service role is created and the Ec2 permission is added as shown below.

4.It will create a New service role, to which we have to add ECRContainerBuilds permission. For that, open the IAM console in a new tab, go to Roles and search and select the above-created role, and click Attach policies. Search for ECR and select the policy as below and click Attach policy

For Deploy provider, select Amazon ECS, cluster, and service name. Also, add the name of the image definitions file as ‘images.json’ that we will create during the build process

Here choose the name from the dropdown and ignore the names from the images below.

Once all the configurations are you should have something like this:

You can then make changes in the code push the commit to the repository and the pipeline will run and deploy the Latest changes to ECR via the DevOps Pipeline.

If you do, You yyess, You have implemented have implemented a cloud native DevOps scalable pipeline for Machine Learning and DataEngineering .

Don't forget to leave a Like, Share and Comment !!!!!

References:

Building Data Engineering Pipelines on AWS

Hari Pranav A — Mon, 21 Feb 2022 13:50:12 +0000

Data Engineering involves building pipelines to produce data driven decisions from a variety of data sources as shown below:

Relational Databases
Non Relational Databases
Data Marts
Streaming Sources .... etc.

In this blog we will create an end to end Data Pipeline from a Flask Web application which will make use of Big Data tools like Athena to query the data on AWS and finally visualize the same on the cloud in near real time.

The architecture flow is as follows:

The flask application is hosted inside Ec2 and uses the data wrangler package to interact with AWS. A query is run in the backend which hits the Athena table.
The data returned is stored a data frame in the flask application.
The data frame is converted to a csv and pushed to S3
There is a glue crawler which runs every time a new file is pushed to s3. This is useful if we run queries multiple times which gives multiple CSV files for each fired query.
The Glue Catalog has the Database and the DDL for creating the table in Athena
The table in Athena can be connected to Quicksight which is used to visualize the data.

Flask Application on Ec2 interacting with AWS CLI :

This front end can have any number of input methods such as input boxes, radio buttons which need to be relevant to the existing query.This front end hosting will be covered below.

Here we can use a micro instance from the EC2 dashbord, from the link below:

Blogpost on micro instance deployment

After launching the instance, we need to SSH into it, we need to install the necessary dependencies:

Dependencies:

AWS cli

This provides control over most of the services on AWS. It must be installed as per the blogpost below to interact with Athena and other services:

AWS cli

Flask with Virtual environment

Link: Install flask

AWS data wrangler:

After activating the virtual environment use the link below to install AWS data wrangler
Link: Install wrangler

Once the instance is up and running we need to create a folder structure for our front end application:

Creating the index.html file:

Here we need to create multiple text boxes. Each text box will contain an input filed as well as a label which can then be referenced in the app.py file to execute the query.

**index.html**
<html>
<head><center><h3>Ornament Verification Report </h3></center></head>
<style>

.orn1
{
border: 1px outset blue;
background-color: lightblue;
padding-top: 10px;
padding-left: 10px;
padding-right: 10px;
padding-bottom: 10px;

}
</style>
<body>

<form action="`{{url_for("test")}}`" method="post">

    <div class="orn1">
    <label for="ornament1">Ornament1</label>
    <input type="text" id="ornament1" name="ornament1" placeholder="enter b1/b2/b3/b4" required>
    &nbsp;

    <label for="lowercount">Lowercount</label>
    <input type="text" id="lowercount1" name="lowercount1" placeholder="enter lowercount" required>

    &nbsp;
    <label for="uppercount">Uppercount</label>
    <input type="text" id="uppercount1" name="uppercount1" placeholder="enter Uppercount" required>

    <br>
    <br>
    <label for="grossweightlowerrange">Gross Weight Lower Range</label>
    <input type="text" id="grossweightlowerrange1" name="grossweightlowerrange1" placeholder="enter lower range" required>
    &nbsp;&nbsp;
    <label for="grossweightupperrange">Gross Weight Upper Range</label>
    <input type="text" id="grossweightupperrange1" name="grossweightupperrange1" placeholder="enter upper range" required>

</div>


    <br>

    <div class="submitbutton">
    <button type="submit">Execute</button></div>
    </form>
</body>
    </html>

The main logic will be developed in the backend where, based on the number of ornaments inputted the query will dynamically execute.

Creating the app.py file with the execution logic

Here we are referencing the ID of the label from the HTML file

Example :
first_ornament=request.form.get("ornament1")

import time
from flask import Flask, request, render_template
import awswrangler as wr
# Flask constructor
app = Flask(__name__)

@app.route('/', methods=["GET", "POST"])
def gfg():
    if request.method == "POST":
        first_ornament=request.form.get("ornament1")
        second_ornament = request.form.get("ornament1")
        uppercount2 = request.form.get("uppercount1")
        lowercount2 = request.form.get("lowercount1")
        grossweightlowerrange2 = request.form.get("grossweightlowerrange1")
        grossweightupperrange2 = request.form.get("grossweightupperrange1")
        # Get the output into a dataframe by using **wr** which is the aws datawrangler package to interact with AWS
        df = wr.athena.read_sql_query("query", database="databasename")
        # Specify the path of the bucket where the result will be stored as **file1.csv**
        bucket = 'input_bucketname'
        path1 = f"s3://{bucket}/file1.csv"
        # Write the csv file to the S3 bucket
        wr.s3.to_csv(df, path1, index=False)
return render_template("index.html")

Now we have successfully taken input from the user-> Got back the result as a dataframe -> Converted it to a CSV -> Pushed it to s3

2. AWS S3 bucket, Glue, Athena, Quicksight:

In the AWS user account we need to enable S3, Glue, Athena and Quicksight follo
wing the blogs below in sequence.

Glue

Athena

Once all three services are launched for the respective user id, we need to enable the glue crawler to read the data into Athena as soon as it is put into S3.

Glue Crawler architecture

The data flow is as follows,

Once the data sits in s3 it will trigger a glue crawler which is automatically configured to read the latest data from s3.

Then a glue catalog schema is generated on the latest data in S3 which is then read by Quicksight.

Open AWS glue and select Crawlers

In the crawler settings select the options as follows

In athena create the DDL as shown below

In quicksight create a data set and point it to athena table as shown below

Click on add a new dataset as shown below

Select the same athena table where the ddl was generated as per the image below

Apache Superset - A Swiss Army Knife for Analyzing Data

Hari Pranav A — Mon, 21 Feb 2022 13:22:27 +0000

# Apache Superset A Visualization Tool

Apache Superset is a an open source visualization tool which provides out of the box integrations with a wide variety of databases and cloud platfroms. It can be easily deployed on EC2 machines and has great features which meet the production grade requirements like

POWERFUL YET EASY TO USE

Quickly and easily integrate and explore your data, using either our simple no-code viz builder or state of the art SQL IDE.

INTEGRATES WITH MODERN DATABASES

Superset can connect to any SQL based data source through SQLAlchemy, including modern cloud native databases and engines at petabyte scale.

MODERN ARCHITECTURE

Superset is lightweight and highly scalable, leveraging the power of your existing data infrastructure without requiring yet another ingestion layer.

RICH VISUALIZATIONS AND DASHBOARDS

Superset ships with a wide array of beautiful visualizations. Our visualization plug-in architecture makes it easy to build custom visualizations that drop directly into Superset.

Why Superset is production ready and can be compared to various BI tools like Quicksight and Power BI and Metabase

ROW LEVEL SECURITY:

Superset allows us to manage access to the different rows in a particular database or table by giving the flexibility of specifying a field or column for which access is to be denied.

USE CASE :

Assuming we have a table from a database containing branch level data (1 lakh records) as the unique key, and if we have to generate analytics for each branch then that would mean we have to generate 1 lakh dashboards. Superset helps us to easily manage this scenario by allowing us to specify the various roles and the access to the data, which means that we can create a single dashboard and create views to the different users based on the roles.

GRANULARITY IN PERMISSIONS:

Superset offers us the flexibility of creating ROLES and specifying the permissions for each role. There are multiple drag and drop permissions which can be easily added to a CUSTOM role ensuring fine granularity in the permissions.

USE CASE:

When we have a large number of parties who want various views on a particular dashboard it becomes very difficult to create them. Hence Superset offers a granular approach based on ROLES which allows the admin to manage the groups and assign the users to the particular group as and when they are created.

AUTO MAILING FEATURE AND SHARING OF DATA:

Superset allows us to configure mailing services which automates the entire process of creation to sharing of data to the respective stakeholder’s email.

USE CASE:

It is often difficult to share the dashboards in a standard format when there a large number of parties involved. Hence in Superset we can give the features of auto mailing which allows the admin to automate the sending and mailing of the dashboards.

INTEGRATION WITH ACTIVE DIRECTORY

Superset allows us to integrate with various authentication services and integration with Active Directory using the LDAP protocol.

USE CASE:

If a company has > 100 employees, managing and creation of permissions with Row Level Security Manually for each of the records is a time consuming process and data entry errors are a pain to deal with

DATA REDUNDANCY AND AVAILABILITY:

There are currently three ways of deploying Superset
• DOCKER
• KUBERNETES
• PIP INSTALL

USE CASE:

Scalability is a major concern as the number of users increases which usually causes slow run times as and when the number of users increases. Hence the power of Kubernetes allows us to manage multiple deployments of docker images on the cloud and also allows up to orchestrate and spin up containers as the traffic increases, which ensures fast load times and improves the User Experience.

SQL LAB

SQL Lab allows us to directly connect the data sources from a variety of data sources and this connection allows us to prototype the various tables in real time.

USE CASE:

When we have to create a visualizations, it is often done using Query logic. Hence Superset allows us to view the data, run the query and also visualize the prototype in a single window, this helps the developers to increase the productivity as all the information is available.

Installing Superset

There are mainly three methods of installing Superset according to the documentation as mentioned above and here we are installing it on and EC2 Instance from AWS on a DOCKER IMAGE.

Let’s set up the EC2 Image on AWS

Register with a free AWS account and navigate to the respective EC2 page and launch a free tier micro instance
Let’s set up the EC2 Image on AWS

After clicking on SELECT in the image above leave all the options as default and the CONFIGURE SECURITY GROUP TAB we need to open port 8088 as Superset runs here as shown in the image below.

The next step is to install DOCKER COMPOSE and DOCKER ENGINE from the links below for an ubuntu instance

Docker Engine : https://docs.docker.com/engine/install/ubuntu/
Docker Compose: https://docs.docker.com/compose/install/

Then we can install Superset according to the documentation below: https://superset.apache.org/docs/installation/installing-superset-using-docker-compose

Now if we want to add a connector to a specific database then before the command

$ docker-compose -f docker-compose-non-dev.yml up

We need to install the respective connector as given in the link below ,

https://superset.apache.org/docs/databases/dockeradddrivers
Here in our use case we need to install the Athena driver hence in the command,

$echo “mysqlclient” >> ./docker/requirements-local.txt

We need to replace “mysqlclient” with “pyathena” and the complete list of drivers can be found in the link below :
https://superset.apache.org/docs/databases/installing-database-drivers

Once Superset is installed we can login using the default user name and password which is admin : admin

Row Level Security on User Level

Row level security provides an easy way to manage access to the various roles and permissions and access to the fine grained data. This can be implemented by going into the SETTINGS ICON and selecting the ROW LEVEL SECURITY as shown in the figure below

Then click on new (+) icon on the right side and select all the fields that need to be added as shown in the image below :

Then we can add a new user and specify this ROLE for them which ensures that the row level security is implemented and that they can see only those Records to which they have access.

Row Level Security on Data Set Level

USE CASE :

If we have a business in which there are about 10000 users and these users are a part of 100 branches and if we have to display a dashboard in such a way that only the current branch user can view their performance in the current quarter. i.e Each branch user must be able to view only their Branch performance and not other branches.

Solution:

We have to manually set the Row level security for each user based on their user name as shown in the image below. Not only this, but also we need to manually add all the tables in the ROLE section for a particular user. In the future if we have more tables then we need to manually add them to the row level security as well.

Although the documentation is not clear on how to solve this issue after a lot of research we found that there is a built in DYNAMIC FILTERING OPTION using the JINJA TEMPLATE. The line below uses the template to give the current username and this helps us to filter later in the DASHBORD LEVEL

This needs to be enabled,

We need to login to our EC2 instance and locate the superset docker folder.
Next we need to go to /superset/superset/config.py
If we open this file using nano and then search using ctrl+w ENABLE_TEMPLATE_PROCESSING, we can see that it is set to false

Even though we set it to true it does not allow us to make any changes as superset is designed in such a way that we have to override this by making changes in another file called superset_config.py.

What ever changes we make in this file is overwritten in the file called config.py.
To enable template processing we need to check the config.py

It tells us to change the FEATURE_FLAGS to TRUE. Open /superset/docker/pythonpathdev/superset_config.py and search for FEATURE_FLAGS . Then in a new line enable template processing and restart the instance.

Once we restart the instance we can create a dataset in which we have a unique user name.

Here we need to select edit dataset and then choose the legacy sql editor.

Then paste the string based on your use case. Here we want only the logged in user to view the current dashboards containing his/her username and not the dashboards of other users.

Refreshing a Dataset:

Since we have enabled Athena Driver on the superset instance, if we have created dashboards on superset and if we change the source database or reload the data it causes issues in superset as old dashboard does not reflect the new dataset.

This issue can be solved by,

Clicking on edit dataset and navigating to the columns part

Now if we click on Sync Columns from source all the old dashboards will reflect the new dataset.

Increasing Storage on EC2 as the datasets increase:

Go to the console, and choose the volume, then go to modify and change the storage. This can only be increased and once done we cant downgrade.

We then need to manually allocate the storage on the ec2 instance.

$ Df -Th

This command is used to get the increased volume in this case it is XVDA

$growpart /dev/xvda 1

This will increase the partision

$resize2fs /dev/xvda1

User Analytics on Superset:

The analytics can be viewed in the SQL lab:

It is present under the PGSQL database, Schema: Public, table ab_user

Query : SELECT * from ab_user where EXTRACT(MONTH FROM last_login )=10

Embedding Dashboards on Another Application:

First, you need to update the PUBLIC ROLE under Settings with these options.

can explore json on Superset
can dashboard on Superset, all database access on all_database_access. Second, embed your dashboard in your HTML

iframe src="localhost:8088/superset/dashboard/5/?standalone=true" iframe

Here ‘5’ in the URL specifies the dashboard number

Creation of Datasets:

To create a Dashboard we need to follow the steps below:

Go to the DATA tab
Select create a DATASET button as shown in the image below

Select the database, the schema, and the table name
Then choose the dataset and select the visualisation type as TABLE

Add all the respective fields and run the query. After that save the result in a new DASHBOARD.

Backup and Restore Superset Via Docker

A docker image does not have persistent storage hence when the image is torn down or if the image crashes the entire persistent data stored in the DB will be terminated.
We can find out more from the blog: https://docs.docker.com/storage/volumes/

Hence If we need to take backups of the data, it is crucial to identify the mounting point of the docker image on the Operating System and on Ubuntu this can be found in the /var/lib/docker.
STEPS TO BACKUP AND RESTORE A DATA IN DOCKER

Create a copy of the current volume

Cp -r [source] [destination]
Here since we already have a backup in backup_vol we are copying it into /var/lib/docker

$ sudo cp -r /home/ubuntu/backup_vol/docker /var/lib/docker

Stop the services as well:

$sudo service docker stop

Stop docker containers by using the command

$ sudo docker stop $(sudo docker ps -q)

Make sure all the services show as exited and this takes a bit of time

Recheck by running
$docker ps -a-non

Stop containers and remove containers, networks, volumes, and images created by up.

$ sudo docker container down

For testing purposes remove the current docker volume which is present in the file system by running
$ sudo su
$ cd /var/lib
$ rm -R docker
If the device is busy then we need to use umount command.

$umount overlay
By doing this all the persistent data from the disk has been removed including the row level permissions that we had set.

Here there is no user with the row level security .

Now we have a backup in the home folder, which needs to be copied to /var/lib/docker

sudo cp -r /home/ubuntu/backup_vol/docker /var/lib/

Here the name of the backup is docker in the folder backup_vol

Now we need to restart it as it will cause errors as it will be reading the data from /var/lib/docker

Now use docker compose up and restart from scratch

$ sudo service docker restart

$ sudo docker-compose up

If we get errors ctrl+c then wait for the service to go down, then go to the main superset folder and run the main command to start superset.

$ cd superset
$ sudo docker-compose -f docker-compose-non-dev.yml up

This ensures that the volumes will be read from the correct location from the backup folder.

This data needs to be backed up to S3 layer as well this can be done by using the tar command. $cd /var/lib tar -zcvf name_of_file_to_be_saved folder_name

• -z : Compress archive using gzip program in Linux or Unix

• -c : Create archive on Linux
• -v : Verbose i.e display progress while creating archive
• -f : Archive File name

$tar -zcvf prev_backup.tar.gz /var/lib/docker

Then we need to install the AWS CLI, https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html After this we need to send the data to the S3 bucket. This can be done by copying the S3 bucket URI.

To copy the files from EC2 to S3
$ aws s3 cp s3://

Hence if we have a file called prev_backup.tar.gz and s3 bucket URI then

$ aws s3 cp /home/ubuntu s3://

Now this data is sent to the respective bucket and is stored successfully.

To restore this copy from S3

$aws s3 cp s3://

Using tar unzip the file (USE SUDO SU and then UNZIP)

$ tar -zxvf prev_backup.tar.gz

$ sudo mv prev_backup/docker /var/lib

Du – display disk usage:
$du -h foldername

Df- Display file system file size
$df -f (Shows the full file system file size)

Superset Reset automation:
https://gist.github.com/pajachiet/62eb85805cee55053d208521e0bdaf13/revisions

Automation of Superset backup to AWS.

1.Create a script called automate_backup.sh. And give necessary permissions.

$nano automate_backup.sh

This script copies the entire docker folder in /var/lib to the home directory, then compresses it

HOME=/root LOGNAME=root PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin LANG=en_US.UTF-8 SHELL=bin/sh PWD=/root #!/bin/bash
sudo cp -r /var/lib/docker /home/ubuntu/prev_backup
tar -zcvf /home/ubuntu/prev_backup.tar.gz /home/ubuntu/prev_backup/docker

$sudo chmod 777 automate_backup.sh

Make sure that the aws cli is configured in the instance and the Access code and key is set. https://docs.aws.amazon.com/cli/latest/userguide/install-cliv2-linux.html

$nano automate_s3_storage.sh

This script copies the zipped data from Ec2 to S3. We have to specify the s3 URI by creating a folder

aws s3 cp /home/ubuntu/prev_backup.tar.gz s3://bucket_url

$sudo chmod 777 automate_s3_storage.sh

Schedule the cron job to run the first script (automate_backup.sh)

$ cron tab –e

#this creates a local backup and it is stored as a tar file every day at 12 AM IST or 5:30am UTC

30 5 * * _ /home/ubuntu/automate_backup_test.sh

#this sends the backup to aws on every seventh day at 12 AM IST or 5:30 am UTC

30 5 _ * 0 /home/ubuntu/automate_s3_storage.sh

#check the status of the cron tab

$ cron tab -l

By following all these steps the data is successfully backed up into s3 on a periodic basis. Due to the compression the size of the file is reduced by a factor of 4. (15 GB file is reduced to 3.7Gb)

Beginners Guide to Hackathons

Hari Pranav A — Wed, 14 Apr 2021 06:41:35 +0000

What is a Hackathon ??

A hackathon is an event in which a team of open minded ,creative people come together and try to solve a real world problem . A "Team" can consist of a mix of non-technical people , designers , thinkers, artists , developers who are diverse in their age as well as skill sets .Some of them may be in their 7th grade and some of them may have a PhD.
These events usually spans for 24 , 48 hours and they are usually conducted on the weekends .
In this limited time the problem statement is given on spot and sometimes the team formation is done in the event itself .

Why Should You Attend a Hackathon ??

To put into perspective imagine that you are attending your first "HACK" and you get paired up with a team of people who you have never met before , you are not familiar with each others skills ,strengths and weaknesses, How will you manage to come with a solution to a given problem within the time limit specified ??

Sounds challenging right , Don't worry these events will help you develop some of the core skills needed in the industry which can be learnt in the the course of a night or two. The skills that you will develop are:

Communication
Leadership
Critical Thinking
Teamwork
Coordination
Deal with conflicts
Crisis Management

We can also consider hackathons as a crash course in becoming a CREATOR or an Entrepreneur or a MAKER !!.

Sounds Wicked Right ??

Apart from this many of the attendees come to hackathons to "Network" . Networking is considered a very effective way of marketing a company , it also helps to recruit new talent and also meet new creative people with whom you may form a lifelong journey of building products.
The amazing part of a hackathon is that you can find people who have the same "Vibe" and way of thinking as you do.
Plus if you are a student you get free attendance, Free merch ,Free Swag and tons of confidence and knowledge that you can take away from the event .
What's there to lose ????
Register For one immediately !!!!

Types of Hackathons :

Internal Hackathons :

These are usually within the companies and they encourage the employees to collaborate with each other.

External Hackathons :

This is the one to look out for especially for students and people looking out for jobs ,Companies are changing their hiring strategy by putting up challenges for potential employees and students to take part in .
The Main reason is that the companies want to test the potential employees capability in solving a problem in a limited period of time with limited knowledge and resources , this model of employment also greatly reduces the burden on the company as they don't have to allocate extra budget for training and recruitment of their employees .
In today's world of startup's it is crucial to adapt to changes by doing code sprints which last all night long!
What better to experience this culture of the "Startup Culture" .

Online and Offline Hackathons:

They can be online in the various online platforms such as HackerEarh , MLH ..
Personally I really enjoy attending the offline hackathons as they are mostly held in beautiful venues like Co-Working spaces and huge office campuses such as Microsoft and Google .
The entire space is given towards the event in which you can experience the culture , the delicious food ,this opens up a new level of work culture , many venues have game centers with games like pool and table tennis , dedicated gyms and a great cafeteria .

Tools For a Hack

Github :

What is github ?? in simple terms imagine that multiple people are working on a project with new ideas and updates , How do you ensure that two people have not done the same work twice , or how do you assure that the version of the code is the latest one free of bugs , Introducing Version Control , Github helps teams to manage the entire process of writing and deploying code . The best part is Github offers their student pack which gives students with a valid ID free goodies/tools worth 200k $

Get yours here

Github student pack

Discord (Free Voice chat for Gamers)

What good is a hackathon without collaboration and communication ??
Discord is one of the most versatile tools that can be used in a hack ,
-You can create private servers
-Create voice channels
-Bots
-Music Channels

Discord

Google Suite(Slides,Docs,Tasks,Keep)

The entire GSuite is a blessing to have in a hack , The true value of real time collaboration can only be appreciated when there are only 15 minutes to submit the solution to a hack !!! XD .
These tools are a life saver , when most of the participants have different operating systems with different word processors and applications.
It takes away the pain of transferring files on usb.
One can even collaboratively write books and articles ,brainstorm ideas using these tools.
The only thing is that a stable internet is necessary as these are mostly cloud based .

Gsuite

Official Docs,StackOverflow,YouTube,Reddit,Google

The official Guides may have a lot of resources , but a beginner might find it difficult to find out how to use it.
Stackoverflow and Youtube tutorials to the rescue !! the support , solutions and videos on these platforms can be used to develop a quick prototype !!
These tools are like your personal Genie , Error in your code ,Just paste the error to get a solution :>

StackOverflow

Magic Buzz Words That Win a Hack

BAT - Blockchain , IOT , AI

In most of the hacks if the teams have implemented a solution with these technologies and if they actually make sense then it can surely secure them a spot in the top 5 hacks.

Lets consider a scenario of a problem statement related to Farming

We can easily use these technologies as shown below

IOT-Sensors send the data to the cloud where the plant health can be monitored
AI / Machine Learning- Analise the data and build models to predict , and improve the efficiency of the crop growth.
Blockchain: Immutable ledger to record the data and land records !!

Here you go you are in the top three teams for the most innovative idea :)

Your Hack will look something like this !!

Choosing Teammates

The best way to sail through a hack is to form a team a day before the event begins and always have a backup team member in case someone bails .
It is always good to have someone in the team who knows how to make really good ppts, and also people who can express the project with humour , basically find someone who knows their Memes ;>.
The above point cannot be stressed enough because the main judging criteria of a hack is usually how the presentation is done , just imagine the fate of the judges , they have to select the best team from a group of 300 odd people .
Humor and presentation skills , answering to the point will definitely help your team leave a good impression on the fatigued judges :)
Last but not least have an app / web developer who can quickly prototype a webpage/app to demonstrate the workflow of your idea.

Coming to the most important part about a hack !!

Never quit in the middle

It is understandable that a beginner who has never been or heard of a hackathon cannot come up with a fully working product / prototype in a single night ,
The best part of a hackathon is that you don't need to come up with a fully working solution !!
The judging criteria will have many parameters and most of the marks that distinguish a better solution is uniqueness of the idea , method of implementation , presentation and workflow .
It is totally understandable if you didn't finish what you intended to do , but always make sure that you can present what you intended to do with confidence to the judges , work on your idea to such an extent that it is foolproof , use diagrams/charts/illustrations to describe your workflow !!
And at the end after all the hard work even if you lose or feel disheartened do not give up ,

Take home the idea or hack and keep working on it , Upload it to git , Write an article or a blog-post about the idea .
Describe the difficulties faced and the solutions for the various problems faced, this will help future hackers on their journey as well !!
Finally you can also apply to different hacks with the same idea (if it is allowed) and hopefully Luck will strike you there !!

Some resources to kick start your journey

MLH
TechGig
DoraHacks
HackerEarth
AngelHack

Cheers !!!

Keeping all of this in mind I wanted to develop an app for beginners to start of on their hackathon journey

All of this and more in the next Article

Documentation For Projects

Hari Pranav A — Tue, 13 Apr 2021 15:59:54 +0000

What is the importance of Documentation ??

Documentation describes the version ,configuration , compatibility and the various dependencies ,prerequisites that a project requires to work .

Documentation helps other developers understand the various limitations and use cases of a project and how to use the various parts of the project to their advantage .

Hence in a nutshell Documentation provides a detailed guide to using a project and all companies big or small have documentation !!

What is markdown ??

Have you used HTML ?? It sometimes gets so messy with all those tags when we just to need to have a static page with minimal functionality

Introducing Markdown !

The main functionality of markdown can be found below

https://markdown-guide.readthedocs.io/en/latest/basics.html

Lets see some basics to bootstrap a blog page

Headings

We use a single hash before text for the largest font similar to h1 in html

Similarly if we want a h2 we use two hashes before the text we want !

Images from the web or local machine

Use an exclamation mark followed by square brackets with the text and round braces for the url

eg :

Hyperlinks

Square brackets with the text and round braces for the url

eg: BEN_Website

MkDocs

MkDocs is a fast, simple and downright gorgeous static site generator that's geared towards building project documentation. Documentation source files are written in Markdown, and configured with a single YAML configuration file. Start by reading the introduction below, then check the User Guide for more info.

Running MkDocs in Local Machine

In order to manually install MkDocs you'll need Python installed on your system, as well as the Python package manager, pip. You can check if you have these already installed from the command line:

$ python --version

$ pip --version

If you don't have python already installed download it from here

pythonDownload

run

$pip --version if it is not installed you can find it in the link below

pipDownload

Install mkdocs using pip

$ pip install mkdocs

Run

$ mkdocs --version

to check that everything worked okay.

If this does not work then python is not added to your path a neat hack around this is to run

$ py -m mkdocs --version

Creating a new project

$ mkdocs new my-project
$ cd my-project

Open a new terminal and open the "my-project" folder there you will find a file called index.md

Edit the file using a text editor then run

$ mkdocs serve

or
$ py -m mkdocs serve

Your static site is now up and running
Open up your browser and go to

http://127.0.0.1:8000, to view the webpage

Now let's add some content to it and host it on GitHub !!

Pushing the site to git

Install git if you haven't from the link below

git

Go to github and create a new repository with the same name as the project folder in this case my-project. This step is crucial as we need to deploy the site to the correct repository.

Add the name

And a short description

Leave the initialize a Readme as unticked and say create repository

Now open the command Line and

$ git init .

$ git commit -m "First Commit"

$ git remote add origin PASTE YOUR URL

Finally run

$ py -m mkdocs gh-deploy

Now your site is deployed to git Hurray !!

Publishing on Medium

Medium is a good platform to publish blogs and articles on various subjects , go to

Medium_Website

The disadvantage with Medium is that , it does not have direct support for markdown files so we can use a third party website , go to

MarkdownToMedium

and follow the steps ,

paste your markdown and convert into a GitHub gist and connect to Medium
paste the link in Medium
import the link and your site is now published to Medium!!

Before publishing to Medium make sure that all the content from the markdown file has come in the correct order, as sometimes there are bugs at the time of importing gists from git.

Decentralised Hosting on IPFS

What is IPFS ??

A peer-to-peer hypermedia protocol
designed to make the web faster, safer, and more open.

IPFS

What is fleek ??

On IPFS , Fleek provides a beautiful way to integrate with Git so that whenever we make changes to our branches it can be directly
updated on Fleek!!!!

Hosting on Fleek

Click on the link below to know more here we can directly login with our git account and host our website.
Click on sign in and sign in with github , now add the repository containing your github page and your done!!.

Fleek

Different CMS to host sites on fleek Link Below

Fleek Blog posts

We can make changes from time to time to update the webpage on git by pushing the pages to git , Fleek will automatically implement these changes from the gh-pages branch from git !!

The best part is that we can make multiple pages with a single GitHub ID!!

Awesome we now have our very own documentation website hosted on a centralized as well as a Decentralized Service!!

Decentralized Portfolio Website

Hari Pranav A — Tue, 13 Apr 2021 15:50:50 +0000

What is a Portfolio Website ??

A portfolio website is an essential tool to getting more business and building your professional brand. In today’s digital world, a portfolio is arguably more important than a resume, no matter what industry you work in. Whether you are a freelance journalist, a recent college grad looking for a job, or even an accountant, when people Google your name, your portfolio will provide the most powerful and comprehensive perspective on you.

Awesome where do I start ?

Lets choose a tech stack

It can be just pure HTML , CSS or can we can use JS frameworks like React or Angular or Vue Ember or Meteor or Mithril the list goes on and on for Javascript FacePalm But here we are going to use Flutter !!!

What is Flutter ??

Flutter is Google’s UI toolkit for building beautiful, natively compiled applications for mobile, web, and desktop from a single codebase.

Say what?? a single framework for web , mobile , desktop ?? Even on a Raspberry Pi : >

The basics of Flutter can be learned in the link below

Flutter Basics

Flutter Advanced

Without installing Flutter on Local Machine

Go to the link below and paste the code below

DartPad

import 'package:flutter/material.dart';


void main()
{
    runApp(MyApp());
}
class MyApp extends StatefulWidget {
@override
_MyAppState createState() => _MyAppState();
}

class _MyAppState extends State<MyApp> {
 @override
Widget build(BuildContext context) {
return MaterialApp(
  debugShowCheckedModeBanner: false,
  home: Scaffold(
    body: Container(
      decoration: BoxDecoration(
        image: DecorationImage(
          image: NetworkImage("https://images-wixmp-ed30a86b8c4ca887773594c2.wixmp.com/i/0052d15d-4c44-4d0f-aade-45074bff0633/d9erbf9-27735655-772d-437a-ae35-ef67e1ea9d10.jpg"),
          fit: BoxFit.fill
          ,
        ),
      ),
      child: ListView(children: [
        Center(
          child: Container(
            margin: EdgeInsets.all(40),
            height: 150,
            width: 150,
            decoration: BoxDecoration(
              shape: BoxShape.circle,
              image: DecorationImage(
                  image: NetworkImage('https://scontent.fblr2-1.fna.fbcdn.net/v/t1.0-1/c0.120.320.320a/p320x320/45144634_1855611297884897_6210030828886425600_n.jpg?_nc_cat=105&_nc_sid=7206a8&_nc_oc=AQmllP_29URGPDyNhajv3rw2YzkqMeeukAQiwgSKm4XJRBhNI533aqwYd13gJN8DdS0&_nc_ht=scontent.fblr2-1.fna&oh=2919360f49c924ee76c6c2f60fc56325&oe=5F1416AE'),
                  fit: BoxFit.contain),
            ),
          ),
        ),
        Center(
          child: Container(
            decoration: BoxDecoration(
              borderRadius: BorderRadius.all(Radius.circular(20)),
              gradient: LinearGradient(
                begin: Alignment.bottomLeft,
                end: Alignment.bottomRight,
                colors: [Colors.green[50], Colors.red[300]],
              ),
            ),
            margin: EdgeInsets.all(10),
            padding: EdgeInsets.all(5),
            child: Text(
              " Flutter | Blockchain | Forensics",
              style: TextStyle(
                color: Colors.black,
                fontSize: 20,
              ),
            ),
          ),
        ),
        Center(
          child: Container(
            decoration: BoxDecoration(
              borderRadius: BorderRadius.all(Radius.circular(20)),
              gradient: LinearGradient(
                begin: Alignment.bottomRight,
                end: Alignment.bottomLeft,
                colors: [Colors.green[50], Colors.red[300]],
              ),
            ),
            margin: EdgeInsets.all(10),
            padding: EdgeInsets.all(5),
            child: Row(mainAxisSize: MainAxisSize.min, children: <Widget>[
              InkWell(
                  child: Image.network(
                    "https://cdn.icon-icons.com/icons2/936/PNG/128/github-logo_icon-icons.com_73546.png",
                    height: 40,
                    width: 40,
                  ),
//                       onTap: () => launch('https://github.com/HariPranav')
              ),
              SizedBox(
                width: 8,
              ),
              InkWell(
                child: Image.network(
                  "https://cdn.icon-icons.com/icons2/31/PNG/128/sociallinkedin_member_2751.png",
                  height: 40,
                  width: 40,
                ),
//                     onTap: () => launch(
//                         'https://in.linkedin.com/in/hari-pranav-77b067162'),
              ),
              SizedBox(
                width: 8,
              ),
              InkWell(
                child: Image.network(
                  "https://cdn.icon-icons.com/icons2/555/PNG/128/facebook_icon-icons.com_53612.png",
                  height: 40,
                  width: 40,
                ),
    //                     onTap: () =>
    //                         launch('https://www.facebook.com/hari.pranav.1'),
              ),
              SizedBox(
                width: 8,
              ),
              InkWell(
                child: Image.network(
                  "https://cdn.icon-icons.com/icons2/122/PNG/128/twitter_socialnetwork_20007.png",
                  height: 40,
                  width: 40,
                ),
    //                     onTap: () => launch('https://twitter.com/aharipranav'),
              ),
              SizedBox(
                width: 8,
              ),
              InkWell(
                child: Image.network(
                  "https://cdn.icon-icons.com/icons2/195/PNG/128/YouTube_23392.png",
                  height: 40,
                  width: 40,
                ),
    //                     onTap: () => launch(
    //                         'https://www.youtube.com/channel/UCWYIT8-ScPTy-fWSa3ff6_A?view_as=subscriber'),
              ),
            ]),
          ),
        ),
      ]),
    ),
    ),
    );
 }
}

Here we are using the online tool called dartpad to render the UI into the browser , you might have observed that the on Tap handlers in the code are commented this is because we need to install a package which is currently not supported, so when we click a social media icon it doesn't link anywhere. Hence we need to install flutter and dart in our local environment to add hyperlinking functionality!!

With Flutter on Local Machine

Installing Flutter : Documentation

Step 1 :

After installing flutter we need to build the application on the web so change the branch to web

$flutter channel master

$flutter upgrade

$flutter config --enable-web

Step 2 :

Run

$flutter devices

Here we should see chrome as the devices available , PS : if it doesnt show up check if you have the latest version of chrome

Step 3 :

Run:

$flutter create

Step 4 :

Lets get coding !!!

Open the pubspec.yaml and go to the commented section under dependencies

    dependencies:

    flutter:
        sdk: flutter

        // paste this code below

    url_launcher: ^5.4.10

Go to the terminal and type

$pub get

This installs the dependencies

Next Open the main.dart under the lib directory in your project

Then paste the code below

// importing the libraries

import 'package:flutter/material.dart';
import 'package:url_launcher/url_launcher.dart';

// creating a stateful widget
// type stf and it auto creates one (Works in vs code and android studion with )

void main()
{
    runApp(MyApp());
}
class MyApp extends StatefulWidget {
@override
_MyAppState createState() => _MyAppState();
}

class _MyAppState extends State<MyApp> {
@override
Widget build(BuildContext context) {
    return MaterialApp(
        // we check the debug banner to be false
    debugShowCheckedModeBanner: false,
    // the scaffold basically provides predefined //material components to be used in UI
    home: Scaffold(
        // we use a container to display the Image
        body: Container(
        decoration: BoxDecoration(
            image: DecorationImage(
                // create a folder called assets in the parent directory of your project and put your profile photo to be displayed
            image: AssetImage("assets/ss2.jpg"),
            fit: BoxFit.cover,
            ),
        ),
        child: ListView(children: [
            Center(
            child: Container(
                margin: EdgeInsets.all(40),
                height: 150,
                width: 150,
                decoration: BoxDecoration(
                shape: BoxShape.circle,
                image: DecorationImage(
                    image: AssetImage('assets/HP.png'), fit: BoxFit.contain),
                ),
            ),
            ),
            Center(
            child: Container(
                // we use the famous Linear Gradient
                decoration: BoxDecoration(
                borderRadius: BorderRadius.all(Radius.circular(20)),
                gradient: LinearGradient(
                    begin: Alignment.bottomLeft,
                    end: Alignment.bottomRight,
                    colors: [Colors.green[50], Colors.red[300]],
                ),
                ),
                margin: EdgeInsets.all(10),
                padding: EdgeInsets.all(5),
                child: Text(
                " Flutter | Blockchain | Forensics",
                style: TextStyle(
                    color: Colors.black,
                    fontSize: 20,
                ),
                ),
            ),
            ),
            Center(
            child: Container(
                decoration: BoxDecoration(
                borderRadius: BorderRadius.all(Radius.circular(20)),
                gradient: LinearGradient(
                    begin: Alignment.bottomRight,
                    end: Alignment.bottomLeft,
                    colors: [Colors.green[50], Colors.red[300]],
                ),
                ),
                margin: EdgeInsets.all(10),
                padding: EdgeInsets.all(5),
                child: Row(mainAxisSize: MainAxisSize.min, children: <Widget>[
                    // here we add the links to our varoius social media websites
                InkWell(
                    child: Image.asset(
                        "assets/github.png",
                        height: 40,
                        width: 40,
                    ),
                    onTap: () => launch('https://github.com/HariPranav')),
                SizedBox(
                    width: 8,
                ),
                InkWell(
                    child: Image.asset(
                    "assets/linkedin-square-color.png",
                    height: 40,
                    width: 40,
                    ),
                    onTap: () => launch(
                        'https://in.linkedin.com/in/hari-pranav-77b067162'),
                ),
                SizedBox(
                    width: 8,
                ),
                InkWell(
                    child: Image.asset(
                    "assets/facebook-round-color.png",
                    height: 40,
                    width: 40,
                    ),
                    onTap: () => launch(
                        'https://www.facebook.com/hari.pranav.1'),
                ),
                SizedBox(
                    width: 8,
                ),
                InkWell(
                    child: Image.asset(
                    "assets/twitter-color.png",
                    height: 40,
                    width: 40,
                    ),
                    onTap: () => launch('https://twitter.com/aharipranav'),
                ),
                SizedBox(
                    width: 8,
                ),
                InkWell(
                    child: Image.asset(
                    "assets/youtube-color.png",
                    height: 40,
                    width: 40,
                    ),
                    onTap: () => launch(
                        'https://www.youtube.com/channel/UCWYIT8-ScPTy-fWSa3ff6_A?view_as=subscriber'),
                ),
                ]),
            ),
            ),
        ]),
        ),
    ),
    );
        }
    }

go to the terminal and type

$flutter run -d chrome

Voila your Single Page Website is Ready on the web!!! This can also be run as a standalone app on your android phone or IOS device !!

To do so change the branch to stable by using the command

$flutter channel stable

Anyways Lets get to hosting our Website on the web

Adding Project to Github

Go to github and create a new repository

Add the name

And a short description

Leave the intitalize a Readme as unticked and say create repository

Now open the command Line and

Run

$git init

$git add .

$git commit -m "First commit"

$git remote add "URL of repo"

$git push -u origin master

Now go to settings and enable the githubpages option

Then Run the following Commands in your flutter project

$flutter pub global activate peanut

$flutter pub global run  peanut

$git push origin --set-upstream gh-pages

Finally you are done !!!!

The page is now hosted on github as the centralised version !!

Fell free to share this url with your friends family and employer's

Hoorah Lets Decentralize It

Decentralization

What is IPFS ??

A peer-to-peer hypermedia protocol
designed to make the web faster, safer, and more open.

IPFS

What is fleek ??

On IPFS , Fleek provides a beautiful way to integrate with Git so that whenever we make changes to our branches it can be directly
updated on Fleek!!!!

Hosting on Fleek

Click on the link below to know more here we can directly login with our git account and host our website

Fleek

Different CMS to host sites on fleek Link Below

Fleek Blog posts

Find Me !!

I have added more pages on for my porfolio such as an animated bottom navigation bar with custom images , a blog section and a short demo of my hackathon projects which will be added soon

Find me on the web below !!

My Portfolio

My Decentralized Portfolio