Generating PO files with Laravel & translating Blade templates

You surely know that the process of translating a whole website can be pretty difficult. For many years, I have used POEdit and .po files for the translation of the static texts of my templates. So when I started using Laravel, I naturally looked for something to use .po files, and I found the laravel-gettext package which did everything that I needed.
But I quickly came upon two different problems that made my life harder with .po files.

First, the POEdit parser for PHP files doesn't understand the blade syntax. I found two ways on the Internet to handle that, but none of them was perfect:

• Manipulate the configuration of the parser to make it believe that files are written in Python. Well, yes it's cheating, but it works quite well. Except for one case : if you have translations inside HTML attributes, for instance : <a href="" alt="{{ _i('Alternative text') }}"> => this text wouldn't be catched by the parser.
• Parse the views generated by laravel in the storage/framework/views folder. It works fine too, but it has a few issues. You have to use the views:cache and views:clear commands to update this folder and I don't really know how it works, but I sometimes have old texts that are still there, or new texts that are not. Therefore, when they are, they are referenced in the PO file with a hash, so it's not very easy to find in which file the original string is located :

 #: ../storage/framework/frontend/16769eb3358dd95b21b188464a877c8b196fabec.php.php:11
msgid "Bonjour"
msgstr "Hello"

Secondly, I have started to work remotely, on a development server accessed throught SSH. And when I want to update my .po files, I now have to download all my files locally to be able to parse them with POEdit.

And thirdly, POEdit parsing is REALLY slow. So everytime you change a few static strings in your project, you have to download all your files, and then wait for POEdit to parse all of them.

So I took a little time to think about it and I found a way to handle all of this myself with just a simple artisan command.
It now just takes a few seconds to parse my blade files, update translations and update my PO files with all my project's translations.

1 - Parsing blade templates to make "blade translations files"

If you are not using blade, you can skip this part and go to the second part.
I will not explain here all of my script, as it is a little bit complicated, so I will just give the whole file for you.
I'm just going to explain a few things here :

• I made the parsing customizable for the functions you are using. I only use '_i()' and '_n()' functions but if you use others, you can configure them on line 29.
• Directory structure : I use different folders at the root of my views templates (one for frontend and one for backend). I have separated them in the resulting files too so it can be used in different PO domains. But you are free to modify it on lines 88-99 and 129.
• You'll see that I put the resulting files in a "blade-translations" folder at the root of the resources folder. You can also update it and put it in the "lang" folder if it makes more sense to you.

Enough talk, here is the file! Just put it in the app/Console/Commands directory and you're good to go :

// app/Console/Commands/TranslationsParseBladeTemplatesCommand.php

<?php

namespace App\Console\Commands;

use Illuminate\Console\Command;
use Illuminate\Support\Facades\File;

class TranslationsParseBladeTemplatesCommand extends Command
{
    /**
     * The name and signature of the console command.
     *
     * @var string
     */
    protected $signature = 'translations:parse-blade';

    /**
     * The console command description.
     *
     * @var string
     */
    protected $description = 'Parse blade templates to find translations and insert it in lang files';

    /**
     * The functions used in your blade templates to translate strings
     *
     * @var string
     */
    protected $functions = ['singular' => ['_i'], 'plural' => ['_n']];

    /**
     * Execute the console command.
     *
     * @return int
     */
    public function handle()
    {
        $path_directories = resource_path('views');

        $directories = File::directories($path_directories);


        // Parse all directories within the resources/views folder
        foreach ($directories as $dir):

            // Init translations global container
            $translations = [];
            foreach ($this->functions['singular'] as $strFunction):
                $translations[$strFunction] = [];
            endforeach;
            foreach ($this->functions['plural'] as $strFunction):
                $translations[$strFunction] = [];
            endforeach;

            // Parse all files from folder
            $files = File::allFiles($dir);
            foreach ($files as $file):
                // Parse all lines from file
                $lines = File::lines($file);
                foreach ($lines as $index => $line):
                    // Get strings from line
                    $strings = $this->parseLine($line);

                    // Insert strings into global translations container
                    foreach ($this->functions as $type => $functions):
                        foreach ($functions as $strFunction):
                            if (!empty($strings[$strFunction])):

                                foreach ($strings[$strFunction] as $val):
                                    if (is_array($val)) {
                                        $string = $val[0].','.$val[1];
                                    } else {
                                        $string = $val;
                                    }
                                    if (isset($translations[$strFunction][$string])) {
                                        $translations[$strFunction][$string][] = $file->getPathname().':'.($index+1);
                                    } else {
                                        $translations[$strFunction][$string] = [$file->getPathname().':'.($index+1)];
                                    }
                                endforeach;

                            endif;
                        endforeach;
                    endforeach;
                endforeach;
            endforeach;

            // Create directories if not exists
            $language_dir = resource_path('blade-translations');
            if (!File::isDirectory($language_dir)):
                File::makeDirectory($language_dir, 0755);
            endif;

            $path = explode('/', $dir);
            $interface = array_pop($path);
            $language_dir_interface = resource_path('blade-translations/'.$interface);
            if (!File::isDirectory($language_dir_interface)):
                File::makeDirectory($language_dir_interface, 0755);
            endif;

            // Create file content
            $content = "<?php\n\n";
            $content .= "return [\n\n";
            $content .= "    /*\n";
            $content .= "    |--------------------------------------------------------------------------\n";
            $content .= "    | STATIC STRINGS TRANSLATIONS\n";
            $content .= "    |--------------------------------------------------------------------------\n";
            $content .= "    |\n";
            $content .= "    |  !!! WARNING - This is a file generated by the 'translations:parse-blade' command. !!!\n";
            $content .= "    |  You should not modify it, as it shall be replaced next time this command is executed.\n";
            $content .= "    |\n";
            $content .= "    */\n\n";

            $i = 0;
            foreach ($this->functions as $type => $functions):
                foreach ($functions as $strFunction):
                    foreach ($translations[$strFunction] as $translation => $paths):
                        foreach ($paths as $p):
                            $content .= "    // $p \n";
                        endforeach;
                        $content .= "    $i => $strFunction($translation), \n";
                        $i++;
                    endforeach;
                endforeach;
            endforeach;
            $content .= "\n];";

            // Generate file
            File::put($language_dir_interface.'/static.php', $content);

        endforeach;

        $this->info('Strings have been exported from blade templates successfully !');
    }

    /**
     * Return translated strings within the line
     */
    private function parseLine($line) {
        $return = [];

        // First let's see if the line has to be parsed
        $found = false;
        foreach ($this->functions as $type => $functions):
            foreach ($functions as $strFunction):
                $pos = strpos($line, $strFunction.'(');
                if ($pos !== false) {
                    $found = true;
                    break 2;
                }
            endforeach;
        endforeach;

        // if not, return
        if (!$found) return [];

        // Then parse for each function
        foreach ($this->functions as $type => $functions):
            foreach ($functions as $strFunction):
                $return[$strFunction] = [];
                $pos = strpos($line, $strFunction.'(');

                // if not found, head to next function
                if ($pos === false) continue;

                while ($pos !== false):
                    $arr = $this->getNextString($line, $pos, $type);
                    if (!$arr):
                        // Error findind string, leave loop
                        $pos = false;
                    else:
                        $line = $arr['line'];
                        $return[$strFunction][] = $arr['string'];
                        $pos = strpos($line, $strFunction.'(');
                    endif;
                endwhile;
            endforeach;
        endforeach;

        return $return;
    }

    /**
     * Return first string found and the rest of the line to be parsed
     */
    private function getNextString($subline, $pos, $type) {

        $substr = trim(substr($subline, $pos+3));

        $separator = $substr[0];
        $nextSeparatorPos = $this->getNextSeparator($substr, $separator);

        if (!$nextSeparatorPos) return [];

        if ($type == 'singular'):
            $string = substr($substr, 0, $nextSeparatorPos+1);
            $rest = substr($substr, $nextSeparatorPos+1);

            // security check : string must start and end with the separator => same character
            if ($string[0] != $string[strlen($string)-1]):
                return [];
            endif;

            return ['string' => $string, 'line' => $rest];
        else:
            $first_string = substr($substr, 0, $nextSeparatorPos+1);
            $rest = substr($substr, $nextSeparatorPos+1);

            // security check : string must start and end with the separator => same character
            if ($first_string[0] != $first_string[strlen($first_string)-1]):
                return [];
            endif;

            $comma_pos = strpos($rest, ',');
            $rest = trim(substr($rest, $comma_pos+1));

            $separator = $substr[0];
            $nextSeparatorPos = $this->getNextSeparator($rest, $separator);

            if (!$nextSeparatorPos) return [];

            $second_string = substr($rest, 0, $nextSeparatorPos+1);
            $rest = substr($rest, $nextSeparatorPos+1);

            // security check : string must start and end with the separator => same character
            if ($second_string[0] != $second_string[strlen($second_string)-1]):
                return [];
            endif;

            return ['string' => [$first_string, $second_string], 'line' => $rest];
        endif;
    }

    /**
     * Return first unescaped separator of string
     */
    private function getNextSeparator($str, $separator) {
        $substr = substr($str, 1);
        $found = false;
        preg_match_all('/'.$separator.'/', $substr, $matches, PREG_OFFSET_CAPTURE);
        foreach($matches[0] as $match):
            $pos = $match[1];
            if ($substr[$pos-1] != '\\'):
                $found = true;
                break;
            endif;
        endforeach;

        if ($found) return $pos+1;
        return false;
    }
}

Now you can just launch a php artisan translations:parse-blade to run this script, and you should see the result in the blade translations folder.
You should find there "static.php" files, where you can find all your static texts and where they come from :

// /var/www/html/resources/views/front/auth/account.blade.php:10
// /var/www/html/resources/views/front/auth/addresses.blade.php:11
0 => _i('Bonjour, %s'),

2 - Generating and updating .po files with xgettext

Now we run into the most important part. How to generate .po files with all of the static texts that are currently written in our project ?
Well, we just have to use exactly the same tool POEdit uses : the xgettext linux command.
For this, you have to install it on your development environment. I will not explain it here for every case, if you are using ubuntu server just like me, simply use this command : apt-get install gettext
Once this is done, you have access to xgettext commands. We will be able to use it with PHP's exec function.
I will also assume you are already using the laravel-gettext package, and you have configured your translations files in its config file.

Let's create a command by using the php artisan make:command command, and call it TranslationsUpdatePoCommand for instance.
The signature of the command will be translations:update-po.
So, let's write the handle function.

Before all, if you are using blade templates, you can call the previous command to have fresh blade translations :

Artisan::call('translations:parse-blade');

Let's define variables we are going to need : the base path where will be located translations, and list of sources files from the laravel-gettext package :

$base_path = 'resources/lang/i18n/';

// First, let's establish which files will need to be parsed
$sources = config('laravel-gettext.source-paths');

Now we can loop on the $sources array we get:

foreach ($sources as $domain => $paths) {

In this loop, we can first list all the files we are going to parse, and put the list in a temporary file we will use later.
For this, we can use the linux command find : (watch out, it only works if you have the same relative path in your laravel-gettext config file : 'relative-path' => '../../../../../app',)

$command = "find ";
foreach ($paths as $path) $command .= "app/".$path.' ';
$command .= '-name "*.php" >pot'.$domain;
exec($command);

At this point, the script will only create at the root of your project multiple files containing file paths.
We can now create the xgettext command. This command will parse the files and create .pot files containing the source texts to be translated.

$command = "xgettext -L PHP --from-code=UTF-8 --add-comments --force-po -o $base_path/$domain.pot --files-from=pot$domain";

There we tell the function that it should create a .pot file per domain and put the resulting file into the folder $base_path. And we use the --files-from to give it the list of paths we created before. But it is not enough : we still have to tell the xgettext command which PHP functions it should be looking for. We can use the --keyword option to add it. Luckily, these functions are already defined in the laravel-gettext config file :

$keywords = config('laravel-gettext.keywords-list');
foreach ($keywords as $keyword) $command .= ' --keyword='.$keyword;

Optionnaly, we can also add metadata in the file:

$command .= " --package-name=".config('app.name');
$command .= " --package-version=".config('app.after_version');
$command .= " --msgid-bugs-address=[e-mail]";

This will add information about the project and owner of the project.
We can now execute it :

exec($command);

Ok, so now we have .pot files, so we can use it to update or create .po and .mo files.
For this we can use 2 xgetext functions : msginit, for creating, or msgmerge for updating.
It is important to use msgmerge for updating as it will keep all the previous existing translations in your files.
And finally, we can use the msgfmt function to generate the .mo from the newly updated .po file.
So come on, let's do it :

// Initialize PO/MO files from POT for each lang
$locales = config('laravel-gettext.supported-locales');
foreach ($locales as $locale) {
    $output_path = $base_path.$locale.'/LC_MESSAGES/';
    $output_po = $output_path.$domain.'.po';
    $output_mo = $output_path.$domain.'.mo';

    if (is_file($output_po)) {
        $command = "msgmerge --update $output_po $base_path/$domain.pot";
        exec($command);
    } else {
        $command = "msginit --locale $locale --input $base_path/$domain.pot --output $output_po --no-translator";
        exec($command);
    }

    $command = "msgfmt $output_po --output-file=$output_mo";
    exec($command);
}

We are just about to be good, just add this line before the loop ends to remove the temporary files created :

exec("rm pot$domain");

And that's it ! You have now .po and .mo files that contains every single text in your project. You can now open a .po file and update one translation, you just execute the php artisan translations:update-po again and the .mo will be updated with it. Or you can download the .po file, open it in POEdit, enter all your translations, save, and replace the .po/.mo by those generated by POEdit.