DEV Community: Güven Atbakan

CLI Applications Tips & Tricks – Run the command in background

Güven Atbakan — Thu, 03 Feb 2022 20:24:03 +0000

Sometimes commands takes too much time to finish. And during its running process, your terminal should always be online! If you close your terminal or you lose your internet connection, process will interrupt.

To avoid this problem, there are many different approaches. If your command takes too much time and you afraid it gets terminated because of "you", you can use nohup to run command in background.

nohup php artisan command:name &

It will run the command in background and you can close your connection. When you run the command, you'll see such message:

nohup: ignoring input and appending output to 'nohup.out'

This means, all output will be stored under nohup.out file. You can tail it while your command is running. tail -f nohup.out

I prefer to save logs in specific folder and file.

nohup php artisan command:name &> /path/to/your/specific/file.log &amp;

CLI Applications Tips & Tricks – Lock the command

Güven Atbakan — Thu, 03 Feb 2022 20:21:37 +0000

Locking the command is basically a "double run prevention".

Assume that you have a hourly email command and you're sending e-mails to users one by one. If the command doesn't finish until next hour, you may send multiple e-mails to some users. Or your colleague can try to start same command after you run it.

// RUN A:
$users = User::where('hourly_email_sent', false)->get();
// On above query, 10.000 users have been found. 
// The command started and already processed 8.000 users in one hour. There are 2.000 users remaining.
// If you start command again, you'll send 2 emails for remaining 2.000 users. Because a command is already running and you started another one.

// RUN B:
$users = User::where('hourly_email_sent', false)->get();
// On above query 2000+ users have been found.
// RUN A and RUN B can contain same users!

I generally prefer to put a value to cache and check it before actually running the command. Here's an example:

<?php

$cacheLockKey = 'command_name';
if (Cache::has($cacheLockKey)) 
{
    $this->output->writeln('Another process is already running. Exiting...');

    return 1;
}

Cache::put($cacheLockKey, date('Y-m-d H:i:s'));

// do some logic

Cache::forget($cacheLockKey);

return 0;

This simple control will prevent you to run a command twice with mistake. It's easy, isn't it?

Downsides with putting lock into cache

You can use database or file storage too. I didn't get trouble just because I'm using cache, but of course you can get problems somehow.

If cache gets flushed for all values, your cache key will also be removed. You have to protect your "lock" until command is really finished.

If you have multiple servers, you have to make sure all servers are using same cache storage.

Failure Case

If the command fails for a reason, you have to be sure cache key was removed. Otherwise command won't start again.

You can handle this situation manually, but if you're running command on cronjob, your next run may not be able to succeed.

This failure case of course depends whatever you want from a job.

CLI Applications Tips & Tricks - dry-run

Güven Atbakan — Thu, 03 Feb 2022 20:13:40 +0000

Testing command line applications is not easy. Dry Run, is a testing process usually used for command line applications. Also it allows you to simulate what will happen when you run the command.

It's like running "SELECT" statement on SQL before running DELETE/UPDATE statements and see which records will be affected.

I'm going to share an example use case. Assume that you have a command which deletes unused files from your S3 bucket.

<?php

$files = File::all();

$this->output->writeln('There are '.count($files).' file to be checked');

foreach ($files as $file) {
    if (!$this->fileService->isFileInUse($file)) {
        $this->fileService->deleteFile($file);
        $this->output->writeln('File has been deleted from storage: file_id_'.$file->id);
    }
}

This command is a bit risky! If there is an unexpected behavior on isFileInUse method, then we can delete wrong files. I prefer to simulate data before actually running these kind of commands.

So --dry-run option is with us!

<?php

// In Laravel, we use `option` method to get an option.
$dryRun = $this->option('dry-run');

$files = File::all();

$this->output->writeln('There are '.count($files).' file to be checked');

foreach ($files as $file) {
    if (!$this->fileService->isFileInUse($file)) {
        if ($dryRun) {
            $this->output->writeln('File will be deleted from storage: file_id_'.$file->id. ' URL: '.$file->url);
            continue;
        }

        $this->fileService->deleteFile($file);
        $this->output->writeln('File has been deleted from storage: file_id_'.$file->id);
    }
}

php artisan s3:delete-unused-files --dry-run

Simple! We're just passing --dry-run option to command line and it's telling us which files will be deleted. In the example, I've also added the url to output. You can add whatever you want to make spot-check. If everything seems good, you can run actual command without dry-run.

CLI Applications Tips & Tricks - Logging Outputs

Güven Atbakan — Thu, 03 Feb 2022 20:10:53 +0000

In this series, I'm going to tell you about my experience and best practices with command line applications. I'm sure these will help you to write better command line applications.

Log output or it didn't happen!

Logging is fundamental for any kind of application. For a command, it's too important to log whatever the command does and its results in detail.

I generally output to a file so that in future you can check when the command ran and what it did.

/var/www/artisan some:command >  $HOME/logs/some-command/`date +%Y-%m-%d-%H-%M-%S`.log 2>&amp;1

I'm saving output to a file with named its creation date. This is really helpful for debugging if command ran or not. And of course running date will be helpful!

So what will the file contain? It depends on what command does, but you can print literally anything! Don't hesitate to write something into logs. Anything will be useful!

Example command:

Assume that you have a command which checks current subscription status of users. And disables users if they're no longer paying. Why we're doing this: You can use webhook for any kind of subscription status update, but sometimes it may not work as expected and you may need command line job to check everything is ok. So let's do this!

<?php

$users = User::where('is_active', true)->get(); // TODO: use cursors instead of getting whole data. This will be fixed in the next chapters!

$this->output->writeln('There are '.$users->count().' users to be checked.');

foreach ($users as $user) {
    if (!$this->stripeService->isUserActive($user)) {
        $this->userService->disableUser($user);
        $this->output->writeln('User have been disabled: user_id_'.$user->id);
    }
}

$this->output->writeln('Done');

return 0;

Output file will be like this:

There are 1989 users to be checked. 
User have been disabled: user_id_100
User have been disabled: user_id_157
...
...
Done.

Advantage of this method over using a logging system is you can watch output (with tail) anytime. And of course, you can connect your output files to any kind of log management system like ELK, Datadog, Coralogix, Papertrail etc.

Find data on log files

Most of time, you may not have a log management system and you may need to find something on logs. You can definitely use grep for this!

grep -Ril -e 'user_id_100' $HOME/logs/some-command/

Ta da! You'll see on when user have been disabled. If you wouldn't log this info, you wouldn't easily understand why user have been disabled.

Log cronjob output on Laravel Framework

I'm using Laravel's scheduler for running my cronjobs. With scheduler, you can send output to a file easily.

https://laravel.com/docs/8.x/scheduling#task-output

$schedule->command('emails:send')
         ->daily()
         ->appendOutputTo(storage_path('cronjob-logs/emails_send_'.date('YmdHis'));

Conclusion

Logging is most critical thing on an application. Saving output is easy. But you have find whatever is precious for you, what you want to log and a structure for yourself. Then you don't have to do nothing, unless debug is required. I hope it won't be required any time, but let's be prepared anyway :)

Calculate distance between 2 points (latitude-longitude) with Mysql ST_Distance_Sphere method

Güven Atbakan — Mon, 22 Nov 2021 20:05:54 +0000

If you want to measure distance between 2 points, you can use Mysql’ ST_Distance_Sphere method – introduced with Mysql 5.7+.

Assume that, you want to get closest user to given point: 41.0049823,28.7319855

$latitude = 41.0049823;
$longitude = 28.7319855;

$sql =  "SELECT 
    users.id, 
    users.username, 
    ST_Distance_Sphere(point(users.longitude, users.latitude), point({$longitude}, {$latitude})) as distance 
FROM users"

So you’ll get distance as meter unit. Note that, this method uses "longitude-latitude" pair. Not "latitude-longitude". It’s important :) I don’t know why but I always use "lat-lng" convention.

Let’s make another query: Get users within 100 meters for given point.

$latitude = 41.0049823;
$longitude = 28.7319855;

$sql = "SELECT 
    users.id, 
    users.username, 
    ST_Distance_Sphere(point(users.longitude, users.latitude),point({$longitude}, {$latitude})) as distance 
FROM users 
WHERE distance < 100"

I was using some sinus, cosinus, tangant and so other functions before ST_Distance_Sphere. That query was running slow and sometimes it wasn’t result correct. Thanks ST_Distance_Sphere for making our life easier :)