DEV Community: zaman shovon

Laravel Resource or Customized Repo?

zaman shovon — Wed, 21 Aug 2024 04:10:10 +0000

Earlier this week, when working with the Laravel Rest API, I encountered an annoyance in the form of a timeout error. It leads to end-user frustration with development concerns. Let me brief the total scenario:

I needed to load data from an external data source, filter it, then prepare it for a json return. The amount of data was not large, only around 10K in a single request. The main problem occurred when I tried to format them after retrieving and filtering them. So, i started to debug using following step:

Check the query is optimized and columns are indexed as well.
Make sure use of chunk method
Check the formatting repo doesn’t use any unnecessary method/reference/implementation/unused functions/external api calling.

All checks are done but still it shows Gateway Timeout Error as it exceeds more than 1 minute. The service class looks like below:

The repo class looks like below:

In naked eye, It shouldn’t throw timeout error for 10K+ data processing and manipulating. We will discuss at the end why it happens(mayn’t be actual concrete reason but probable) and now discuss how i resolve it using Laravel Api Resource.

Its simple to implement. First, generate Laravel Api Resource from command line:

php artisan make:resource DataFormatterResource

Then, send your model object to resource and format/manipulate your data as per requirement given below:

Surprisingly, It only took 3.7 seconds to response 😮!
I tried to dig out the real issue here and found some probable cases that mentioned on top to define at the end. The cases are given:

Laravel API resources provide a consistent interface for accessing and manipulating data where i use repo with some dependency injection in it. This makes it easier to write efficient code and to avoid common performance bottlenecks.
Laravel API resources are optimized for performance as they use caching and other techniques to improve the speed of data retrieval and processing where i only go for chunk of array raw formatting.
Laravel API resources automatically serialize the results of database queries to JSON or XML, depending on the request headers. This saves you the hassle of having to write your own serialization code.

At most of my project’s services, I utilized repo or functional formatter at the service layer, but in this case, I had a difficulty where there may be other causes for this issue to occur.
What I wanted to emphasize is that Laravel Resources might come in handy in some tricky situations when working with models.

If you like this article leave a clap or comment. <?= “Happy coding!….” ?>

Cache driver switch in Laravel while failing connection

zaman shovon — Thu, 02 Mar 2023 14:27:36 +0000

Common Usage of Cache Driver

While working with small projects ex. SPA(Single Page Application), Module based feature(Payment Gateway), Small service provider etc. no need to deal with big sets of data while rendering. But while working with large scale application with proper scalability, Less rendering time and faster data loading we often face with loading issue and database connection issue.

To mitigate those issues, We often use cache driver to store data for certain time to process data faster and there no need to fetch from database or query from there.

Laravel cache driver is used in some large-scale application to maintain data readability, faster execution and data filtering. But when comes the worst scenario mentioned below what we have to do is to deal with some exceptions with bare hand:

Cache driver failed to connect ex. Redis
Cache driver stop responding or shut down for some issue.
Data storing or data fetching unknown exception raised.

For those, Application falls and client disappointment is raised too high. To prevent this kind of situation, We have to switch immediately from one cache driver to another without delay and not manually.

In Laravel, There is a tricky but much efficient to do switch cache drivers.

Switch cache driver in Laravel

Suppose, We set Redis as cache driver but we have file driver for cache storing as well. If Redis shut down accidently, We have to switch to file driver by using below operation:

class CacheConnectionValidator // Cache connection checker
{
    /**
     * @return void
     */
    public static function checkAndUpdateCacheDriver() : void
    {
        $redis_host = env("REDIS_HOST"); // primary set cache host
        $redis_port = env("REDIS_PORT"); // primary set cache host port
        if(!self::checkPortConnection($redis_host,$redis_port)){
            config(['cache.default' => 'file']);
        }
    }

    /**
     * @param string $redis_host
     * @param string $redis_port
     *
     * @return bool
     */
    private static function checkPortConnection($redis_host,$redis_port) // check host return response
    {
        try{
            $connection = fsockopen($redis_host, $redis_port, $errno, $errstr, 20);
            if( !$connection ){
                return false;
            } else {
                return true;
            }
        } catch(Exception $e){
            return false;
        }
    }
}

Here, CacheConnectionValidator class will check, validate and set the storage for default cache driver for data processing.

fsockopen() is used here to open the external source socket connection to check connection establishment during any application runtime. We can use pfsockopen() which is identical to fsockopen() but fsockopen() is closing after script finishing but pfsockopen() is running even after script finishing.
If connection returns true, It means the host is responding but if the connection returns false immediately set the file as default cache driver.

    /**
     * Bootstrap any application services.
     *
     * @return void
     */
    public function boot()
    {
        // While boot the class will be called
        CacheConnectionValidator::checkAndUpdateCacheDriver();
    }

The CacheConnectionValidator class should be registered in boot function in AppServiceProvider to load the script.

The whole process is sketched in following:

There are lot more packages ex. Predis, PhpRedis to catch the Redis connection exception but they sometimes unable to catch exception while falling in runtime. So, It will be secure and safe to use php built in functions to catch the exception and take immediate action

Have a good lara-day 😎

Laravel Worker with super-visor

zaman shovon — Mon, 10 Oct 2022 07:01:02 +0000

Many of us struggled to setup supervisor with Laravel Worker in our docker container or web-server. If you don't then you are lucky not as me. 😉

If you are struggling to do or you want to know about what is job, how supervisor maintain the jobs inside a docker-container or web-server, how to setup all these stuffs then you are in right post i guess.

Follow my journey that let you know something informative.

Pre-requisites

Setup laravel queue:

php artisan queue:table
php artisan migrate

After migrating, You should find 2 tables named failed_jobs & jobs. If you don't then follow the procedure again or check laravel queue for better understanding.

Setup job batches to keep track of jobs:

php artisan queue:batches-table
php artisan migrate

You should see table named job_batches. To learn more about batch jobs, Check the beautiful documentation of batch jobs laravel

Setup some test methods for batch jobs:

    public function testJob() {
        $path = storage_path('test');
        $files = glob("$path/user.csv"); // Parse the stored csv
        $header = [];
        foreach ($files as $key => $file) {
            $data = array_map('str_getcsv', file($file));
            if($key === 0) {
                $header = $data[0];
                unset($data[0]);
            }
        }
        $batch = Bus::batch([new TestCsvForSupervisor($header, 
        $data)->dispatch();
        return $batch->id;
    }

What we do here?
Ok, we've imported a csv with 100 thousand users information ex. email, username, address etc. TestCsvForSupervisor is the job class for handling some checks or operation like insert, update or comparing datas with any table ex. users. Simple as writing artisan command. 😉

In TestCsvForSupervisor Job Class:

    public function handle()
    {
        try{
            foreach ($this->data as $row) {
                $result = array();
                $result = User::whereName($row[0])->first() ? 'Found' : 'Not Found';
                Log::info('User checked successfully with email: '.$row[1]);
            }
        } catch (Exception $err) {
            Log::info('Error occured with message: '.$err);
        }
    }

Tough one, I have 20 thousand users information in my users table and what I've done here? Compare csv data with users table data and store the checking result in log (You might find it clueless but it's just a test case actually not good test case😒). Then dispatch the job to store the pending jobs to be done in table.

Setup supervisor

Now it's time for setup supervisor into docker container where you can follow 2 steps:

You can access docker bash and install supervisor by command scripts (It is for local testing not for global deployment or team collaboration)
You can integrate the command scripts in Dockerfile.

Before digging into both please let me inform you some details about supervisor

Supervisor is basically linux based daemon that runs in the background. You have to configure your laravel app so that supervisor know about background jobs. It monitors the executing or executable jobs and if for any interference the jobs accidentally stop ex. server down after the restarting the supervisor automatically starts the job from where it stops and monitor the failed, pending and success jobs through this time and can deliver us jobs report.

Install from docker bash:

sudo apt-get install supervisor

Install from Dockerfile using Daemon

In your app/docker-conf/php/Dockerfile, Write the following script:

RUN apt-get install -y supervisor

Then, build and start the docker:

docker compose build
docker compose up -d // -d for running the docker in background

I will suggest you to follow the docker daemon-based (step-2) supervisor because it can be used globally. So, we will be continuing with that.

In your root directory make file named laravel-worker.conf and write and save the file:

[program:laravel-worker]
process_name=%(program_name)s_%(process_num)02d
command=php /app/artisan queue:work // you may use extra attributes like --sleep=3
autostart=true
autorestart=true
# stopasgroup=true
# killasgroup=true
;user=
numprocs=8
redirect_stderr=true
stdout_logfile=/app/storage/logs/worker.log
stopwaitsecs = 100

In the docker bash, you will find file named supervisord.conf in /etc/supervisor/ directory. Overwrite the last line of the supervisord.conf file by following script and save it:

[include] files = /etc/supervisor/conf.d/*.conf /app/laravel-worker.conf

After that, Execute the following commands to initiate supervisor:

# Start the service
  service supervisor start

# Read the new config from supervisord file
  supervisorctl reread

# Activate the configuration
  supervisorctl update

# Start queue command for running jobs
  supervisorctl start laravel-worker:* or laravel-worker

# Check the status of new config operation
  supervisorctl status

Congrats 🥳, You've just did a great job not just laravel-job but the real automated jobs using supervisor.

You may find the job list with status and unique id from job_batches table after starting supervisor like below shot:

There is one interesting fact about supervisor. Do you know what language used to develop supervisor?

It's Python 🐍😲

         Thanks for reading with so much patience. ❤️

DSA vs Generic System Dev

zaman shovon — Mon, 25 Apr 2022 16:08:23 +0000

I am a new one here(Dev.to) and in my 2+ year of dev career I have faced a lot of challenges over many micro and robust project.

Mostly the challenge I have faced that maintaining system architecture and handling more data in less time and in a convenient way(bugless/non-conflict).

Some points we usually use Generic coding over DSA are mentioned below:

Because of limited deadline, Most of us use template which may be open-source or already in our git storage(Previous project reference). To meet the tight deadline and make sure less buggy product, We often follow this trend and almost all of them are written in generic method calling and packed by sometimes oop's(Object Oriented Programming System) or redundant method calling! So, we have less scope to implement or even fix where to use DSA.😐
If we develop the project from the scratch, Most of us always love to use some built-in methods, open source packages or as usual array-looping execution over DSA. This is because we feel comfort to use those or we are very used to use those. We think of using recursive loop over binary search because we feel ok with that but we barely think about memory-optimization/time-complexity/space-complexity!!

Those mentioned points can be different for others but the motive is almost same for everyone.

I have been a team member of a robust project since 2021 and I have faced a lot of issues while handling huge amount of data at a time. We have used Django and we thought that as Django is faster and can handle more requests per second. But we were wrong. This million dollar project is becoming slower while database is becoming heavier day by day. While digging the main reason, We have found that if we used DSA at some modules, System won't be slow so earlier! 😓