DEV Community: Ahmed Magdy

Become a Better programmer by Detaching Your Emotions from work

Ahmed Magdy — Sun, 11 Dec 2022 14:44:09 +0000

Intro

I wanna share and discuss something with all my Dev colleagues. In the Last few months about I felt like I have lost most interest for programming in general and this has lead to me becoming Lazy even though I am not a lazy person at all. I tried multiple stuff. but nothing worked. In my mind I knew that I was good programmer. I can't simply leave something I am good at and make good money from It to go work at something I am not good at and pays less money.

Me saying this doesn't mean to just drop your dreams and stick to becoming programmer ..etc. Fast forward to the future I am trying something new which is working for me perfectly which is Detaching my emotions away from programming. meaning I don't have to like it to do it and let me tell you this is the most correct way to get things done.

But what leads to losing interest in the first place?

In some experiment some scientist gathered around bunch of kids who
liked to draw with out giving them any rewards then they started giving them rewards for painting and drawing as soon as they stopped giving them these rewards all of the kids have lost interest in drawing.

So if you apply this scenario on fresh grads who are very passionate about programming.. You will find that the reward mentioned here is The Salary and after sometime the salary just doesn't feel the same anymore because baseline of dopamine is elevated. You can check Dr. Andrew Huberman on Youtube he has some great videos on how dopamine affect you.

Conclusion

If you have lost some interest like me in programming try not doing it for fun for a while and just do it regradless of how you feel and this can just work around for anything in life.

How to Make a major change to all your foreign keys (Mysql, Laravel)

Ahmed Magdy — Thu, 28 Oct 2021 21:03:37 +0000

I've recently stumbled upon this issue where I wanted to add ON DELETE CASCADE to every foreign key i had on my tables.

I couldn't afford to do this manually because the project I am working on had over 106 tables and it's still expanding, so the main idea here is to write a general script that can be applied to any project.

1- You need to fetch every foreign key for you DATABASE

SELECT TABLE_NAME, COLUMN_NAME, CONSTRAINT_NAME, REFERENCED_TABLE_NAME, REFERENCED_COLUMN_NAME FROM INFORMATION_SCHEMA.KEY_COLUMN_USAGE WHERE TABLE_SCHEMA = "<DATABASE_NAME>" AND REFERENCED_COLUMN_NAME IS NOT NULL AND CONSTRAINT_NAME LIKE "%_foreign"

Note I added CONSTRAINT_NAME LIKE "%_foreign" because when Laravel creates a foreign key it follows this convention below

CONVENTION:


Schema::table('student_grades', function(Blueprint $table){
  // adding a foreign key 
   $table->foreign("student_id")->references("id")->on("students");
});

The created foreign key will be named student_grades_student_id_foreign.

hence the condition CONSTRAINT_NAME LIKE "%_foreign".

If you apply the previous SQL Query you will get data like this

2- Re-create your foreign key

make a migration file php artisan make:migration alter_foreign_keys_on_tables

Now let's get to the code

  public function up() {

       // The SQL Query
        $data = DB::table("INFORMATION_SCHEMA.KEY_COLUMN_USAGE")
            ->select([
                "TABLE_NAME",
                "COLUMN_NAME",
                "CONSTRAINT_NAME",
                "REFERENCED_TABLE_NAME",
                "REFERENCED_COLUMN_NAME"
            ])
            ->whereRaw("TABLE_SCHEMA =  '<DATABASE_NAME>' 
                AND REFERENCED_COLUMN_NAME IS NOT NULL 
                AND CONSTRAINT_NAME LIKE '%_foreign'")
            ->get();
        // you need to write the next line to get back to your ..
        // Original Database 

        DB::statement("USE <DATABASE_NAME>");
        foreach ($data as $single) {
           // Don't forget to add protected $single = null; in migration class
            $this->single = $single;
            Schema::table($single->TABLE_NAME, function (Blueprint $table) {
               // drop the previous foreign key 
                $table->dropForeign($this->single->CONSTRAINT_NAME);
                // New Foreign key 
$table->foreign($this->single->COLUMN_NAME)->references($this->single->REFERENCED_COLUMN_NAME)
                    ->onDelete("cascade")->on($this->single->REFERENCED_TABLE_NAME);
            });
        }
    }

Final Note:

The purpose of this article was to show you how to make a major change to all your foreign keys.

feel free to contact me on my email ahmed.magdy.9611@gmail.com.

Thanks for coming to my Ted Talk.

Building Cache Layer Using Redis and Mongoose

Ahmed Magdy — Wed, 17 Feb 2021 01:58:52 +0000

Introduction.

If you ever built an api you will find that you will be needing to cache some GET requests that repeat alot and a find (if you are using mongoose) or select (sql) queries can be expensive over time. We are going to introduce a solution to this problem in this article.

Solution.

We will be following a very simple strategy here, But before we start you need to be familiar with mongoose and node.js

Strategy

Imagine we are working with a Query to fetch all dev.to blogs and The model will be called Blogs

Blogs Model

const blogSchema = new mongoose.Schema({
    owner : {
        // user in the database 
        type: mongoose.Types.ObjectId,
        required: true,
        ref: "User"
    },
    title: {
        type : String,
        required: true
    },
    tags: {
        type : [mongoose.Types.ObjectId],
    },
    blog: {
        type : String
    }
});

now the request to fetch all the blog

app.use("/api/blogs",(req,res,next)=>{
         const blogs = await Blogs.find({}); 
          res.send(blogs);
});

now after we get the image of what we are working with lets get back to the strategy

send a query to the database to ask for a certain thing
if this query has been fetched before aka exists in cache (redis)?
if yes, Then return the cached result
if no, Cache it in redis and return the result

The trick here is that there is a function in mongoose that is automatically executed after every operation
The function is called exec.

so we need to overwrite this exec function to do the caching logic.

first step to overwrite

const exec = mongoose.Query.prototype.exec;
mongoose.Query.prototype.exec = async function (){
    // our caching logic
    return await exec.apply(this, arguments);
}

now we need to make a something that tells us what gets cached and what doesn't. Which is a chainable function.

making the chainable function

mongoose.Query.prototype.cache = function(time = 60 * 60){
    this.cacheMe = true; 
    // we will talk about cacheTime later;
    this.cacheTime = time;
    return this;
}

So Now if i wrote

Blogs.find({}).cache(); // this is a valid code

Now if you are not familiar with Redis GO GET FAMILIAR WITH IT. there are thousands of videos and tutorials and it won't take that much time.

We need some data structure or types for the cached results. After some thinking, I've found out this is the best structure and I will explain why.

Blogs is the collection name;

let's say you are doing Blogs.find({"title" : "cache" , user : "some id that points to user" })

then Query will be { "title" : "cache" , "user" : "some id ... " , op : "find" // the method of the query } ;

result is the result we got from database;

This structure is called NestedHashes.

Why we are doing Nested Hashes like this

we need to say if Blogs got a new Update or Insert or Delete operation delete the cached result. Because the cached result is old and not updated by any of the new operations.

NOW back to code.

mongoose.Query.prototype.exec = async function(){
    const collectionName = this.mongooseCollection.name;

    if(this.cacheMe){   
      // You can't insert json straight to redis needs to be a string 

        const key = JSON.stringify({...this.getOptions(),
             collectionName : collectionName, op : this.op});
        const cachedResults = await redis.HGET(collectionName,key);

      // getOptions() returns the query and this.op is the method which in our case is "find" 

        if (cachedResults){
          // if you found cached results return it; 
            const result = JSON.parse(cachedResults);
            return result;
        }
     //else 
    // get results from Database then cache it
        const result = await exec.apply(this,arguments); 

        redis.HSET(collectionName, key, JSON.stringify(result) , "EX",this.cacheTime);
       //Blogs - > {op: "find" , ... the original query} -> result we got from database
        return result;
    }

    clearCachedData(collectionName, this.op);
    return exec.apply(this,arguments);
}

Remember the part where I said we need to clear cached data in case of Update, Insert or Delete.

clear the cached data

async function clearCachedData(collectionName, op){
    const allowedCacheOps = ["find","findById","findOne"];
    // if operation is insert or delete or update for any collection that exists and has cached values 
    // delete its childern
    if (!allowedCacheOps.includes(op) && await redis.EXISTS(collectionName)){
        redis.DEL(collectionName);
    }
}

Expected Results

Much faster find queries.

What to Cache

Don't Cache large data Imagine if you have a find query that return 20 MB or even 100 MB worth of data you will be slowing down your whole application.
Don't Cache Requests that don't get a lot of traffic and that is highly dependent on your application.
Don't Cache Important Data like users or transactions.

Final Notes

My redis setup.
cacheTime paramter is option I put a default of 1 hour but you can edit it as you wish, i suggest 1 or 2 days.

Handling Mongoose Dublication Errors

Ahmed Magdy — Mon, 08 Feb 2021 16:19:20 +0000

Introduction

If you ever wrote Node.js code and decided to have a document based DB, your main goto will be MongoDB ofc and you will be using mongoose as your ODM then you have met this error before.

MongoError: E11000 duplicate key error collection: testDB.users index: name_1 dup key: { : "some random name" }.

the problem is there are multiple ways to handle it. one of them is using a library called mongoose-unique-validator. but we are not going to use an external library which I don't know how it works under the hood.

Before we keep going

there is some stuff that needs to be clarified
1- name { type :string , unqiue: true} unique param in mongoose is not a validator meaning doing const myUser = new User(data) will not throw an error in case of duplication.
it will only throw and error when doing myUser.save()

2- when trying to add a user I suggest using either insertOne or create function. and keep using it through your whole application because We are about to overwrite one of them.

Note: I will be using `create` here.

Why?

why we wanna handle duplication error globally anyway?
because you might have 10 or 20 collections where each one has 2 or 3 unique keys and you are not going to check for every one manually.

Implmentation

you can easily overwrite mongoose function by doing

const mongoose = require("mongoose");
// old create function 
const create = mongoose.Model.create;
// overwriting
// it takes this arguments according to mongoose documentation
mongoose.Model.create = async function (doc, options, callback){
     // your logic here;
     // return original function 
    return create.apply(this, arguments);
}

my logic here is when I am using create function I will insert a new option which is some keys to check if they are duplicated or no.

const data = { name : "ahmed"} 

User.create(data, {checkForDublication : ["name"] }).then(console.log).catch(err=> console.error(err));

I am going for this format where checkForDublication is a new option I created and will be sending the keys as array format.

Logic

check if options has a checkForDublication param.
check if its values exist in the schema and are unique.

the last step (checking if the key is unique) is very important, Because we are going to use findOne({$or: searchQuery}) ... and as you know searchQuery is going to be an array, If one Element in this array is not unique or index it's going to perform collectionScan instead of indexScan which is very slow.

filter the checkForDublication array meaning remove every key that doesn't exist in the schema or is not unique.
generating the search query
checking if the result of the search query exist.

Code

mongoose.Model.create = async function (doc, options, callback){
    if (options && options.checkKeysForDublication){
        const searchQuery = getSearchQuery(doc,this.schema.obj, options.checkKeysForDublication);
        await checkForDublication(this, searchQuery);
    }


    return create.apply(this, arguments);
}

getSearchQuery function

function getSearchQuery(doc,schema, keys){
    if (!Array.isArray(keys)||keys.length === 0){
        return;
    }
    const filteredKeys = filterKeys(doc,schema,keys);
    return  makeSearchQuery(doc,filteredKeys);

};
function filterKeys (doc,schema,keys){
    const filteredKeys = keys.filter(key=>{
        if (!schema[key] || !schema[key].unique || !doc[key]){
            console.warn(`${key} key either doesn't exist in this schema or not unique so it will filtered`);
        }
        return schema[key] && schema[key].unique && doc[key];
    });
    return filteredKeys;
}
function makeSearchQuery(doc,keys){
    const query = [];
    keys.forEach(key=>{
        const obj = {};
        obj[key] = doc[key];
        query.push(obj);
    });
    return query;
}

output of getSearchQuery

[{"name" : "ahmed"} // and every key that was common between insterted document and checkForDublication arr].

another example

User.create({name: "ahmed ,
 email :"anymail@gmail.com" , password : "123" }, {checkForDublication : ["name" , "email"] }.then(console.log);

output of getSearchQuery

[{ "name" : "ahmed" , {"email": "anymain@gmail.com"}]

checkForDublication function

async function checkForDublication (Model, searchQuery){
    const result = await Model.findOne({$or: searchQuery});
    if (!result){
        return;
    }
    searchQuery.forEach(singleObject=>{
        //every singleObject has only one keyl
        const key = Object.keys(singleObject)[0];
        if (result[key] === singleObject[key]){
            throw new Error(`${key} already exists`);
        }
    });

}

output `Error: name already exists`

important note: don't forget to put this line of code require("./fileThatHasOverWrittenCreateFunction.js") at the very start of your project so changes can take effect.

NOTE: you can throw your custom error as well... but this one is for another article.

Finally

The main goal in this article was to make a global way to handle duplication errors.
if You have any feedback feel free to send me on this email ahmedmagdy@creteagency.com.

Enjoy~

In case You didn't know, you don't need to install any npm package more than once.

Ahmed Magdy — Tue, 02 Feb 2021 14:51:37 +0000

Introduction

Many of us have faced the problem of having to install the same package a lot of times because we are working on multiple projects locally, And as we all know your node_modules can get pretty out of hands in terms of size on disk.

PNPM

I've stumbled upon this project recently which helps you manage and solve that problem of having to install the same package more than one time. It's using CAS to store node_modules and later refer to them on your project.

Usage

you will find a lot of commands in their Documentation.

here are the ones I am using for now

-pnpm install instead of npm install, What will this do is install the packages in your package.json; if they were already installed before it will just link them to your project directory.
also, pnpm install will generate a file called pnpm-lock.yaml which contains information about the installed packages.

pnpm add <package name> instead of npm install <package name> also works that same as pnpm install if the package doesn't exist it will download it and link it to your node_modules

NOTE: you can use the same flags of npm.

example: pnpm add <package name> --save-dev and it will it add to your dev dependencies.

I recommend that you read their Documentation as it contains a lot more details on how to use and the behavior of each command.

Good luck and Enjoy.

Building a cronjobs server with NodeJs

Ahmed Magdy — Sat, 24 Oct 2020 16:27:16 +0000

Note: This article is made for people who are familiar with Nodejs, if you are coming from another language this can be beneficial as well.

What is a cronjob, It's basically a job or a function that will be executed after a certain amount of time aka scheduled.

In the project we are building right now we needed to have scheduled jobs to update certain parameters into our Database. so this is easy, Right? you can just use this npm package , so where is the problem?

The problem

While developing the project we found that some of the jobs are being scheduled but not Executed, why? because when we push new code into our server we have to restart. and every scheduled job into the memory is deleted forever. so what is the solution?

The solution

We have two options here but the core idea is still the same.
The cronjob should run as a stand-alone instance meaning it's independent of our main application.

1- to schedule jobs based on the OS that our server running on which is a Linux distribution. While this solution might work, for now, the problem is we don't have full control over our server and we might remove the whole project into another server in the future.
2- to make a cronjob server and have a record of these jobs in our database

Important Note: I am not going to share the full code in this article, I am just sharing the core idea.

Making the Server.

first, we need to make a jobs model

a more simplified version of this model will be

   ` time:{
        type: Date,
        required: true
    },
    done:{
        type: Boolean,
        default: false
    },
    type:{
     type: "Whatever you want it to be"
    }
    canceled:{
        type: Boolean,
        default:false
    }`

Ofc you can add or customize that model as you want but keep in mind that time, done, canceled are the most important parameters in this model.

second install express and mongodb and node-schedule.
third make a simple server that starts after connecting to The DB.

here is a simple configuration for this

DB.js Config

const mongodb= require('mongodb');
const dbService = {

    db:null,
    connect: async function connection (){
        return new Promise ((resolve,reject)=>{

            mongodb.MongoClient.connect(process.env.MONGODB_URL,{
                useUnifiedTopology:true
            },(err,client)=>{
                if (err){
                    return reject(err);
                }
                this.db = client.db(process.env.DB);
                resolve(true);
            })
        });
    }
}

Now make a server and end-point to receive job requests, And another one to cancel jobs if you want to.

const express = require('express');
const dbConfig = require('./DB');

dbConfig.connect().then(()=>{
    app.listen(5000,()=>{
        console.log("server is listening on port 5000");
    });
   // re-schedule jobs (that are not Done yet) if the server restarts 
   onServerRestart();

}).catch((e)=>{
    console.log("couldn't connect to database Restart the server");
});

end-point to schedule a job and another to cancel.

router.post('/',(req,res)=>{

    const job = req.body.job;
    // job is a Document that is created in the main application  
   // and sent to this server to be scheduled 

    scheduleJob(job);
    return res.sendStatus(200);
});

router.get('/cancel',(req,res)=>{
  // job id is sent from the main application
   const jobID = req.query.id;

  // this will be explained later
   Emitter.emit(jobID);
   return res.sendStatus(200);
}

inside job schedule function

const sched = require("node-schedule");
// this will also be explained later 
const Emitter = require("../Emitter/cutomEmitter");

async function scheduleJob(job){
   const newJob = sched.scheduleJob(job.time,()=>{
      // do the job after a certain amount of time 
      and now this job is in memory 
   });
}

now, what if you want to cancel the job? node-schedule gives you a way to do that by calling newJob.cancel(). But how will you do that from another server? remember this server is just made to schedule jobs. here comes the Event Emitter API.

refactoring the function to cancel jobs.

async function scheduleJob(job){
   const newJob = sched.scheduleJob(job.time,()=>{
      // do the job after a certain amount of time 
      and now this job is in memory 
   });
   function cancelJob(j){
      j.cancel();
     //access the DB and find Job by ID and cancel it 
        db.db.collection("jobs").updateOne(
            {
                _id: ObjectID(_id),
            },
            {
                $set: {
                    cancelled: true,
                },
            }
    );
   }
// now how the hell are we going to access this function? 
//using emitter -> don't worry i will show u how to configure one 
    Emitter.once(_id, cancelJob);
 // using Emitter.once cause this removes the whole Event which is "_id" -> referring to the job u want to cancel. 
// If the job is executed after a certain amount of "time" then you don't need that event, time to remove it. 
    sched.scheduleJob(job.time,()=>{
         Emitter.removeListener(_id,cancelJob);
   });
}

here is the Emitter.js config

const EventEmitter = require('events');

class jobEmitter extends EventEmitter{}
const Emitter = new jobEmitter();

module.exports = Emitter;

yes, it is that easy.

now let's use our CronServer

Usage

The scenario is in server 1 you need to schedule a job

first, if you are using mongoose just export the Jobs model and jobs.Create({ time: new Date(anytime that you want) note you might want to add certain parameters here to specify the job // which I suggest you do });
send a post request to CronServer with the job to be scheduled.

axios.post("http://localhost:5000/,job,{ //might want to config that request });

look into your Database to see if the job is scheduled or no.

time to test the cancellation request.

axios.get(http://localhost:5000/cancel?id=<jobID>);

Check if the job is canceled or no you should see in the terminal the console.log(job with ${_id} is canceled);.

try hitting the same request again you won't get anything because the emitter has been removed and the job has been canceled.

Final Notes

onServerRestart function is made to reschedule jobs if anything happened and you need to restart the CronServer just search for the jobs that have (done: false, canceled: false) if the time is less than now Date execute them IMMEDIATELY without re-scheduling, else just re-schedule.

if you have any questions you can contact me via ahmed.magdy.9611@gmail.com. Thanks for coming to my TED talk.

DEV Community: Ahmed Magdy

Become a Better programmer by Detaching Your Emotions from work

Intro

But what leads to losing interest in the first place?

Conclusion

How to Make a major change to all your foreign keys (Mysql, Laravel)

Building Cache Layer Using Redis and Mongoose

Introduction.

Solution.

Strategy

Blogs Model

first step to overwrite

making the chainable function

NOW back to code.

clear the cached data

Expected Results

What to Cache

Final Notes

Handling Mongoose Dublication Errors

Introduction

Before we keep going

Note: I will be using create here.

Why?

Implmentation

Logic

Code

getSearchQuery function

output of getSearchQuery

output of getSearchQuery

checkForDublication function

output Error: name already exists

Finally

In case You didn't know, you don't need to install any npm package more than once.

Introduction

PNPM

Usage

Building a cronjobs server with NodeJs

The problem

The solution

Making the Server.

Usage

Final Notes

Note: I will be using `create` here.

output `Error: name already exists`