DEV Community: Mahmoud Galal

Why you Should join Hackathons

Mahmoud Galal — Sat, 11 Feb 2023 19:08:36 +0000

MLH is a platform where you could join their programs and Hackathons

They have launched the Global Hack Week 6 days ago

And its about AI/ML, and It has a lot of sessions and challenges

There's tons of hackathons per year and you definitely should join them

Why you should participate

Swags
Prizes
Learning new things will boost your career
connections All these benefits come together in a one package just by joining a Hackathon

They will add A lot to you and your knowledge

What I learnt in AI/ML week ?

I built many projects with different categories
I learnt how to use NLP and speech to text algorithms
I explored how to clean Data

even all of that was not part of my career but they opened a lot of doors to new Ideas

I'm now ranked the top 7% of the total hackers, and I hope to be the 1%

and You should definitely check their website all the time
MLH

and you should register for the coming hacks
MLH-Hack

Happy Hacking

Puppeteer: Crawling data for newbies

Mahmoud Galal — Thu, 15 Sep 2022 14:04:05 +0000

Puppeteer is a NodeJs library which provide a High-level API to Chrome over DevToolsProtocol

It's do a lot

Crawl data from webapps
Generate a screenshots and PDFs
Create an up-to-date, automated testing environment. Run your tests directly in the latest version of Chrome using the latest JavaScript and browser features.
Automate form submission, UI testing, keyboard input, etc.

Let's do a tutorial about Automation and Crawling

Puppeter setup

Is common to know how the structure look to run any lib
so Puppeter rely on promises

const puppeteer = require('puppeteer');

(async () => {
  const browser = await puppeteer.launch();
  const page = await browser.newPage();
  await page.goto('https://example.com');
  await page.screenshot({path: 'example.png'});

  await browser.close();
})();

we call puppeteer.launch() method to open a browser for us
then asking it to open a page which we will do all stuff there.
and goto() to navigate to a website
all methods that we'll use is provide to the page object

Crawling data

if we want to crawl data, we need to know
what we deal with, So we want to know that we select using the id's or tags or even attributes we deal with DOM

so let's make the environment

mkdir book-scraper
cd book-scraper
npm init -y
npm install --save puppeteer

setup the browser instance

When you open browser you can interact with it like doing clicks and type etc. Headless chrome will do the same thing, but programmatically.

we will need

four .js file
browser.js
main.js
pageScraper.js
PageController.js

let's start with browser.js

const pupt= require('puppeteer');

async function startBrowser(){
    let browser
    try{
        browser = await pupt.launch({
            headless:false, // defualt
            'ignoreHTTPSErrors':true
        })
    }
    catch(err){
        console.error(err)
    }

    return browser;
}

module.exports = {
    startBrowser
};

Making the headless:false that's mean you will see the interface while your script is running to see the process

let's do the main.js file to run our browser instance

const browser= require('./bowser')
const scraperController= require('./pageController'); // we will make it later

let browserInstance = browser.startBrowser();

// we want to to controle this browser, so we will do a function to do that
// so we have to pass it there

scraperController(browserInstance);

let's do the 'pageController.js' file to see how to interact with our browser

const pageScraper= require('./pageScraper');

async function scrapeAll(browserInstance){
    let browser;

    try{
        browser=await browserInstance;
        await pageScraper.scraper(browser);
    }
    catch(err){
        console.error("There's a problem with browser Instance => ", err)
    }
}

module.exports=(browserInstance) => scrapeAll(browserInstance)

we export the method to take the browserInstance as argument

let's make the pageScraper.js file to scrape the data

const scrapeObject={
    url:'http://books.toscrape.com',
    async scraper(browser){
        let page= await browser.newPage();
        console.log(`navigating to ${this.url}..`)
        await page.goto(this.url);
    }
}
module.exports=scrapeObject

This the basic structure to build your web crawler

you can clone it through github pupt starter

Scrape the data from a page

First, if you inspect the source code for the homepage using the Dev Tools inside your browser, you will notice that the page lists each book’s data under a section tag. Inside the section tag every book is under a list (li) tag, and it is here that you find the link to the book’s dedicated page, the price, and the in-stock availability.

let's edit our files, reopen pageScraper.js file

const scrapeObject={
    url:'http://books.toscrape.com',
    async scraper(browser){
        let page= await browser.newPage();
        console.log(`navigating to ${this.url}..`)
        await page.goto(this.url);

        // wait fot the DOM to render
        await page.waitForSelector('.page_inner')
        //Get the link to all selected books
        let urls= await page.$$eval('section ol > li',links =>{
            // Make sure the book is in stock 
            links=links.filter(link => link.querySelector('.instock.availability > i').textContent !== "In stock")
            // Extract the links from the data
            links=links.map(el => el.querySelector('h3 > a').href)
            return links;
        })
        console.log(urls);
    }
}
module.exports=scrapeObject;

// run this command
node main.js

it should work clearly

let's scrap all data about the books

modifie the pageScraper.js file

const scraperObject = {
    url: 'http://books.toscrape.com',
    async scraper(browser){
        let page = await browser.newPage();
        console.log(`Navigating to ${this.url}...`);
        // Navigate to the selected page
        await page.goto(this.url);
        // Wait for the required DOM to be rendered
        await page.waitForSelector('.page_inner');
        // Get the link to all the required books
        let urls = await page.$$eval('section ol > li', links => {
            // Make sure the book to be scraped is in stock
            links = links.filter(link => link.querySelector('.instock.availability > i').textContent !== "In stock")
            // Extract the links from the data
            links = links.map(el => el.querySelector('h3 > a').href)
            return links;
        });


        // Loop through each of those links, open a new page instance and get the relevant data from them
        let pagePromise = (link) => new Promise(async(resolve, reject) => {
            let dataObj = {};
            let newPage = await browser.newPage();
            await newPage.goto(link);
            dataObj['bookTitle'] = await newPage.$eval('.product_main > h1', text => text.textContent);
            dataObj['bookPrice'] = await newPage.$eval('.price_color', text => text.textContent);
            dataObj['noAvailable'] = await newPage.$eval('.instock.availability', text => {
                // Strip new line and tab spaces
                text = text.textContent.replace(/(\r\n\t|\n|\r|\t)/gm, "");
                // Get the number of stock available
                let regexp = /^.*\((.*)\).*$/i;
                let stockAvailable = regexp.exec(text)[1].split(' ')[0];
                return stockAvailable;
            });
            dataObj['imageUrl'] = await newPage.$eval('#product_gallery img', img => img.src);
            dataObj['bookDescription'] = await newPage.$eval('#product_description', div => div.nextSibling.nextSibling.textContent);
            dataObj['upc'] = await newPage.$eval('.table.table-striped > tbody > tr > td', table => table.textContent);
            resolve(dataObj);
            await newPage.close();
        });

        for(link in urls){
            let currentPageData = await pagePromise(urls[link]);
            // scrapedData.push(currentPageData);
            console.log(currentPageData);
        }
        await browser.close();
    }
}

module.exports = scraperObject;

This is all about crawling the data from a webpage

You can find this on github Full code

How I automated git workflow!

Mahmoud Galal — Sat, 06 Aug 2022 16:36:00 +0000

everyday we use git, and we do a lot of git add -A and a lot of commits also pushing and pulling, so I was thinking to make this easy for me, and I said why not for all developers!

so I come up with CLI for doing that!

What this CLI do?

let's describe your flow when you want to start a project

make a dir
init git
make a remote so with gite you can make this with just gite repo

what if you want to push?

add -A to the stage
commit with signed or not signed commit, I will mention it later
then push, but whooo, there's stuff to fetch! with gite push you just have to write where you want to push and which branch you want to push it and the biggest thing here we handle fetching error, with automated one

So how can I install this CLI?

you just want to have

NPM
nodejs
GPG (optional for signed commit)

this CLI tested on Linux ecosystem, and it will be available for windows later, it will work for macOS

installation guide

open your terminal sudo npm i -g cli-gite

after download it install it
sudo ~/../../usr/local/lib/node_modules/cli-gite/install.sh

if you want to use gpg key settings, you have to know more about it
so read my documentation

commands

gite repo
gite push
gite -v
gite
gite update

what is GPG Key

This key used to make a verified commit in GitHub, it's important for people who work on company to make sure the make the push authorized commit

you can setup it and make your on signature

contribution

feel free to contribute with anything you want
repo

Why JavaScript is confusing?, Learn it by knowing the difference

Mahmoud Galal — Fri, 01 Jul 2022 16:32:41 +0000

JavaScript is a superstar those days with its friend typescript

They have a lot of libs and their community is really big, and this made it very useful as there are a lot of frameworks.

So you can create everything with JavaScript, mobile apps, desktop, web apps, even AI models, IoT, Backend and Frontend.

Those reasons maybe make you learn it.

Why it’s confusing for us?

really this has a lot to know, but mainly the big difference is the syntax, and some other functionality, I will cover all of them here.

Declaring a variable

In other languages you know there’s a primitive Data Types

like int,char,string,float,etc.

but JavaScript have something else to say about Declaring Variables

they called it dynamically typed, and that means JavaScript will auto-understand what is inside this variable, like Python.

let x=5;// int
let str='hi' // string with single quote
let Str="Hi" // string with double quote

There’s no char here the string can be one character as you want.

Other datatypes like boolean

let status=true;
let isWorking=false;

What is confusing in Declaring a variable

there’s an old-school keyword to declare a variable var
This keyword make a confusion
let’s see

{
// making a local scope with brackts
let x=6;
var y=5;
}
console.log(x);// undfinded 
console.log(y);// 5

that’s happened because var is a globally-scoped variable

it’s seen everywhere

to fix something like that we use ‘use strict’ I will talk about it soon.

'use strict'
{
// making a local scope with brackts
let x=6;
var y=5;
}
console.log(x);// undfinded 
console.log(y);// undfinded

Functions

declare a function is everywhere but here’s the difference

you can pass any type of value to the function

function foo(x,y){
 return x+y;
}
foo("Hi"," Kro") //Hi Kro
foo(2,4) // 6

maybe you see that’s cool but in many cases isn’t cool cause it will make your function fall into bugs, you have to be careful when you deal with it.

The ==

really there’s another comparison operator is ===
yeah you see

the difference is really because the variable

as you know we can declare anything using let

so when we compare we may compare with different data types

let x=5;
let y="6";
(x==y)// true
(x===y)//false

the == operator is just to see if they are the same without begin aware if they are different data types or not

but the other one, === want the exact match in the look and the data type

Things to avoid

As we know JavaScript does not require a semicolon to end your statement

but it must sometimes,

Why?

really because JavaScript parser or engine ( similar to the compiler )

put a semicolon on every good ending expression

alert(5+
6
+2); // 13

As you can see here Js parser didn’t put a semicolon in this case, cause when he sees no good ending expression will ignore it.

for more tutorials about JavaScript

The Modern JavaScript Tutorial

Master of Linked Lists data structure

Mahmoud Galal — Sat, 04 Jun 2022 21:17:22 +0000

Master anything however it's so confusing

I'm going to tell you the secret of Mastering anything you want however is so hard to understand.

The story of how I realized this powerful method

I started a challenge on one of problem-solving websites
for solving 2 or 3 problems each day and after 2 months I realized that I'm a Master ⚡ of using and solving problems that have Linked list and I solve it very quick without any hinting, otherwise on any problem with any data structure
like graph I start struggling and google stuff and the topic is so confusing in my head, So I started to learn graph with the same way I studied linked list

Know the science first 🎓

When you try to learn anything you should start understand the concept and the science of this thing.

and those some tricks that will help you find out.

Head up to google and try to search why they did this thing.
start compare it to something that do the same like linked list VS arrays.
look for difference between them.

Baby steps 🐣

When you do anything for your first time you should go slow as possible as you can.
It's will help you to clear your thinking of many things you will face while learning.

Imagine it

While you try to understand something confusing to you, The best thing to do it's to imagine how it would be like in the real world.

Relate it to thing you know very will

Teach someone 📖

even you didn't learn a lot you have to do multiple teaching

in the early step

Teach someone know the topic will
Talk to your rubber duck (any cute thing) and explain what you learned
try to explain it to one who has the intention to learn this topic

Practice on the topic a lot

innovate when you practice when you solving a problem and you got a solution try to solve it again later days when you forgot the other solution you will find this so impressive,

Because your mind will be more different and more smarter than the before days

If you know a trick don't miss to share it with us in the comments

Custom Commands in your Linux or Mac

Mahmoud Galal — Wed, 01 Jun 2022 14:48:35 +0000

Terminals are the superpower of developers

I have been in love with using terminals all the time even on windows.
The true love comes after I understand most of the commands and how to write Bash Scripts, and If you are not a Linux or Mac user using the terminal, you missed a lot.

How can I make my own commands?

Open your terminal using CTRL + ALT + T
sudo vim ~/.bashrc

if you get something else so, you should install Vim

sudo apt install vim

and run the command before

Go down and write your bash script click I to insert in the file write alias [your command]='[ the executable command in Linux ] '

then ESC and :wq Enter

if you want to make clean aliases

create a new file in the home ~ dir `touch .bash_aliases'
go into it using vim .bash_aliases
put your aliases there
save it by ESC and :wq Enter

close the current terminal to refresh your bashrc

open a new terminal and test your new commands

*this is the end of the tutorial *

How does it works?

when we open a terminal there's a bash script run to provide a path of everything we install,
when you write code . this opens your VScode so this command is saved in the bashrc
this makes a configuration for everything in the current terminal

so this script begins to execute every time you open the terminal, what if I want to print something every time a open a terminal

log to your bashrc again and write what you want using
echo Hi KroKing
save and restart the terminal

*Follow for more Tricks in Linux *

Low-Level Trick to solve problems

Mahmoud Galal — Sun, 29 May 2022 18:59:24 +0000

Solve programming problems with one of the most popular Low-Level Technics

Bit Manipulation & Bitwise Operators

When you compile anything with your favorite programming language it’s converted to machine code

There’s a very impressive trick that could help you to solve problems 😎

what are the Bitwise operators?

The & (bitwise AND) in C or C++ takes two numbers as operands and does AND on every bit of two numbers. The result of AND is 1 only if both bits are 1.
The | (bitwise OR) in C or C++ takes two numbers as operands and does OR on every bit of two numbers. The result of OR is 1 if any of the two bits is 1.
The ^ (bitwise XOR) in C or C++ takes two numbers as operands and does XOR on every bit of two numbers. The result of XOR is 1 if the two bits are different.
The << (left shift) in C or C++ takes two numbers, the left shifts the bits of the first operand, and the second operand decides the number of places to shift.
The >> (right shift) in C or C++ takes two numbers, the right shifts the bits of the first operand, and the second operand decides the number of places to shift.
The ~ (bitwise NOT) in C or C++ takes one number and inverts all bits of it.

Let’s know how it works by example

Put in your mind that every number you will deal with will be converted into binary
and If you are familiar with digital circuits, it will be so easy to understand how they work

Example #1


int main(){
        // a = 5(00000101), b = 9(00001001)
    int a = 5, b = 9;
    // The result is 00000001
    cout<<"a = " << a <<","<< " b = " << b <<endl;
    cout << "a & b = " << (a & b) << endl;

}

let’s put a sample rule 0 is false 1 is true every number will be reconverted into decimals again when printing it.

Explanation #1

The & operator work like if the two cases are 1 it will return 1
so
1&1 = 1
1&0 = 0
0&1 = 0
0&0 = 0

Example #1

a = 5 , in binary it will be 00000101

b = 9 , in binary it will be 00001001

so we take each digit and apply our & rule

1 & 1 = 1, and so on..

so the result will be a & b = 00000001

Example #2

// The result is 00001101
    cout << "a | b = " << (a | b) << endl;

Explanation #2

The | operator work if one of the cases is 1 it will return 1
1|1 = 1
1|0 = 1
0|1 = 1
0|0 = 0

Example #2

a = 5 , in binary it will be 00000101

b = 9 , in binary it will be 00001001

so we take each digit and apply our | rule

1 | 1 = 1, and so on..

so the result will be a | b = 00001101

Example #3

// The result is 00001100
    cout << "a ^ b = " << (a ^ b) << endl;

Explanation #3

The ^ operator it’s a little bit confusing, it works if the two numbers are different it will return 1
1^1 = 0
1^0 = 1
0^1 = 1
0^0 = 0

Example #3

a = 5, in binary it will be 00000101

b = 9, in binary it will be 00001001

so we take each digit and apply our ^ rule

1 ^ 1 = 0, and so on..

so the result will be a ^ b = 00001100 = 12 in decimal

Example #4

// The result is 11111010
    cout << "~a = " << (~a) << endl;

Explanation #4

The ~ operator gets the opposite of the number so,
~1 = 0
~0 = 1

Example #4

a = 5, in binary it will be 00000101

so we take each digit and apply our ~ rule

~1 = 0 , and so on..

so the result will be ~a = 11111010 = -6 in decimal it’s too way hard to convert it to decimal with basic rules so search for 2’s complement, 1’s complement and Signed magnitude

keep in mind - the last left digit is to represent if it is positive = 0 or negative = 1, when you apply this operator you should add +1 to the result if you want to get the negative value of the number

Example #5

// b=9 (00001001)
// The result is 00010010
    cout<<"b << 1" <<" = "<< (b << 1) <<endl;

    // The result is 00000100
    cout<<"b >> 1 "<<"= " << (b >> 1 )<<endl;

Explanation #5

The << operator shift the bit pattern n time to The Left which in our case n=1 and append 0 at the end of the number, the Left shift is equivalent to multiplying the bit pattern with $2^n$
( if we are shifting n bits ).
1 = 00000001
1 << 1 = 00000010 = 2 = $(1*2^1)$ and so on ..
in the example
9 << 1 = 00010010 = $(9 * 2^1)$ = 18

The >> operator shifts the bit pattern n time to The Right which in our case n=1 and append 1 at the end of the number, the Left shift is equivalent to dividing the bit pattern with $2^n$
( if we are shifting n bits ). Keep in mind Binary has no float points in this case
4 = 00000100
4 >> 1 = 00000010 = 2 = $(4/2^1)$

5 = 00000101
6 >> 1 = 00000010 = 2 = $(5/2^1)$ in decimal and so on ..

in the example

9 >> 1 = 00000100 = 4 = $( 9/2^n )$ in decimal

Let’s know how Bit manipulation works with Example

and why is so helpful for an optimized solution, its works like a magic 🪄

let’s start with an easy example

we want to know if x is a power of 2

implementation

bool isPowerOfTwo(int x)
    {
        if(x == 0)
            return false;
        else
        {
            while(x % 2 == 0) x /= 2;
            return (x == 1);
            }
    }

The above code has a time complexity O(log N)

Is there an optimal approach?

The same problem can be solved using bit manipulation.

Consider a number x that we need to check for being a power for 2.

Now think about the binary representation of (x-1).

(x-1) will have all the bits the same as x, except for the rightmost 1 in x and all the bits to the right of the rightmost 1.

Let, x = 4 = (100)2 x - 1 = 3 = (011)2

Let, x = 6 = (110)2 x - 1 = 5 = (101)2

It might not seem obvious with these examples, but the binary representation of (x-1) can be obtained by simply flipping all the bits to the right of rightmost 1 in x and also including the rightmost 1.

Now think about x & (x-1).

x & (x-1) will have all the bits equal to the x except for the rightmost 1 in x.

Let, x = 4 = (100)2 x - 1 = 3 = (011)2 x & (x-1) = 4 & 3 = (100)2 & (011)2 = (000)2

Let, x = 6 = (110)2 x - 1 = 5 = (101)2 x & (x-1) = 6 & 5 = (110)2 & (101)2 = (100)2

in x= 6, 6 is not a power of 2 so when x & (x-1) = (100)2 that mean it’s not what we want

Properties for numbers which are powers of 2, is that they have one and only one bit set in their binary representation.

If the number is neither zero nor a power of two, it will have 1 in more than one place. So if x is a power of 2 then x & (x-1) will be 0.

Implementation:

bool isPowerOfTwo(int x)
    {
        // x will check if x == 0 and !(x & (x - 1)) will check if x is a power of 2 or not
        return (x && !(x & (x - 1)));
    }

and the above code has time complexity O(1)

Another Hard Example to see how much bit manipulation is powerful

Given a string array of words
, return the maximum value of
length(word[i]) * length(word[j])
where the two words do not share common letters
. If no such two words exist, return 0

Idea is to create a bitmask of each string and compare it with the bitmask of other strings.

Bitmask Explanation

-Let's say we have three strings "abc", "def" and "abde", corresponding bitmasks of these strings will be "111", "111000", and "11011". how?

For each character, we'll need to find the index where we need to set 1 in our bitmask. So for character "a" index will be 0, for "b" it will be 1, for "c" it will be 2 and so on.

The index can be easily found by subtracting each character by 'a' Ex.

'a' - 'a' = 0
'b' - 'a' = 1
'c' - 'a' = 2

Next step is to left shift 1 by the index value.

for 'a', index will be 0
1 << 0 = 1

'b', index will be 1
1 << 1 = 10

'c', index will be 2
1 << 2 = 100

After shifting, the last step is to OR the shifted value of current character with the bitmask of current string. So if we OR the shifted values of character "a", "b" and "c".

001 | 010 | 100 = 111

Check for common characters -If two strings s1 and s2 have common characters, then they will have 1 at the same index in the bitmask and if we do AND of the bitmasks of s1 and s2, it will result in a value greater than 0.

bitmask[s1] & bitmask[s2] = 0, if no common characters
                            >0, otherwise

Ex. bitmask of "abc" and "def" is 111 and 111000, respectively.

111 & 111000 = 0, hence no common characters

bitmask of "abc" and "abde" is 111 and 11011, respectively.

111 & 11011 = 00011 > 0, hence common characters found

Finally, code -

class Solution {
    public int maxProduct(String[] words) {
        int n = words.length;
        int[] bitmask = new int[n];
        int max = 0;

        for(int i=0; i<n; i++) {
            // Calculate bitmask for current word
            for(int j=0; j<words[i].length(); j++) {
                // index will be - for a -> 0, b -> 1, c -> 2 and so on
                int index = words[i].charAt(j) - 'a';

                // left shift 1 by index and OR it with the current bitmask
                bitmask[i] |= (1 << index);
            }

            // Compare bitmask of current string with previous strings bitmask
            for(int j=0; j<i; j++) {
                /* If there is a 1 at the same index of current string {i} and {j},
                then bitmask of i and j string will result in a number greater than 0,
                else it will result in 0 */
                if( (bitmask[i] & bitmask[j]) == 0 )
                    max = Math.max(max, words[i].length()*words[j].length());
            }
        }

        return max;
    }
}

Time Complexity = O(n.(k+n))