DEV Community: Arturo Linares

Easy VRT testing

Arturo Linares — Mon, 13 Apr 2020 04:09:08 +0000

I normally work on different and very active projects in several testing environments. I needed a simple and fast way to generate visual regression tests between them, or even to just to compare my local environment to an integration server. However, it was cumbersome to edit backstop configuration files frequently just to change servers, thus I created a tool that simplifies the task a lot:

https://github.com/arturolinares/easy-vrt

It is a Symfony console application that wraps Docker backstop containers to run the tests. The application reads a list of URLs from a CSV file to generate the JSON configuration files for BackstopJS.

Usage

Once the project is installed you will want to compare two sites. Follow these steps:

Generate the backstop scenarios using a CSV file containing the routes to visit.
Run the tests.
Browse the results to visualize the differences.

To generate the scenarios use the command vrt:gen. It will use a CSV file with the routes to generate them:

bin/console vrt:gen path/to/routes.csv --ref-domain=http://prod.com --url=http://my-qa.com

You can find a sample of the CSV file in the sample directory. The file has two fields: the route and an optional label. For example:

/,"The Homepage"
/category/tag,
"/about-us", "About US"

You can set the reference domain using the flag --ref-domain. This is normally production. Use the flag --url to specify the domain you want to compare.

Sometimes you need to create several scenarios for the same route but different prefix. For example, when using multilingual sites. To specify the prefixes you can use the --prefix (-p) option:

bin/console vrt:gen path/to/routes.csv -u [http://my-qa.com](http://my-qa.com) -r prod.com -p es -p en -p fr

The above will generate scenarios for these urls:

/es/
/en/
/fr/
/es/category/tag
/en/category/tag
/fr/category/tag
/es/about-us
/en/about-us
/fr/about-us

Until now, you only have configured backstop. To actually run the tests use vrt:run. This command will run backstop using a Docker container. Also, if you want to see what the script is doing, you can change the verbosity with -v, -vv and -vvv.

Finally, to see the report , use vrt:server to browse the diffs.

Git log filtering and how to show a better history tree

Arturo Linares — Mon, 04 Nov 2019 09:00:00 +0000

Some useful commands to find commits using Git:

Filter by content

Filter by the contents of a commit. Think of it as if you do a git log -p (shows the diff of each commit) and then you search the output.

git log -S var_dump

Filter by commit message

git log --grep "FRB-666"

Filter by author

git log --author "Arturo"

You can use emails and several values:

git log --author "outlook.com|hotmail.com"

Filter by date

Conveniently accepts --before and --after

git log --before "2017-06-11"

Git log format

I've found that git log is powerful, but sometimes difficult to read. Normally I use this git alias to show a nice tree in the console:

$ git config --global alias.lg
log --color --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)%Creset' --abbrev-commit

This will show a tree like this when you type git lg

Windows running on a Linux kernel?

Arturo Linares — Mon, 07 Oct 2019 09:00:00 +0000

Windows running on a Linux kernel?adminMon, 10/07/2019 - 04:00

Body

This week Microsoft unveiled their Surface Phone, called Surface Duo. It looks great and I think it puts the company back in the innovators list.

But the fact that the Duo runs on Android is not easy to digest. Some people think that Microsoft soon will make a Windows version that runs on a Linux kernel. Although it might be nothing more than crazy theories, they make sense when Nadella says that the operative system is not that important for Microsoft any more:

The operating system is no longer the most important layer for us. What is most important for us is the app model and the experience.

To add more fuel to the conspiracy theory, Intel has been working on a highly optimized Linux distro (for Intel architecture of course) called Clear Linux.

Remember when I've said that my money is on Microsoft moving Windows to a Linux kernel? Well, Android is based on Linux so this might be testing the waters with a mobile device first. Remember, Microsoft wants unified architecture

🤔

-- @Barnacules

Intel has been focusing mainly in performance, and while you still run Gnome or other Desktop Environment on it, they have stated that their priority is performance and security.

What if... they are waiting for a Windows that runs on a Linux kernel to finally integrate it in Clear Linux?

Personally, a Windows that runs on a Linux kernel doesn't make sense to me. The main reasons why these rumors started was because Microsoft started opening source code, bought GitHub and, specially, WSL. But with the last case (WSL2), which allowed a Linux kernel to run on Windows, is a Windows technology, also used in Windows Sandbox. Why would they invest in this awesome technology that runs on Windows if you're about to jump off the boat.

Another reason is that it would break backwards compatibility of many applications, something Windows has respected for many years. I know open source technologies like Wine have been making possible to run windows applications on Linux, but they are not perfect, and they certainly would have a limit. The same limit that Microsoft hit when trying to write their reversed Wine: WSL v1. Some issues like performance, networking and file system access are simply not easy to emulate.

I guess only time will say, meanwhile it is worth to keep an eye on Clear Linux and, of course, the new foldable devices that are about to hit the market 😁

Drupal development on Windows

Arturo Linares — Thu, 26 Sep 2019 13:00:00 +0000

Recently I wrote on how I replaced my MacBook Pro with a PC and still being productive when developing on several Drupal environments.While Pop_OS! is a wonderful Linux distro, I decided to give Windows a try. The primary reasons were the usual: hardware.

The drivers are optimized to work on Windows and some small features that are nice-to-have only work on this controversial OS: Login with an infrared camera (Windows Hello), a solid workspace with several monitors (maybe this is the most important reason), better Dolby sound from the speakers and battery optimizations. Nothing critical really, but things that improve the experience.

What means to have a productive development environment?

This:

Bash command line.
Fast docker environment (without unison, I've had enough problems with it already).
Working git
Be able to launch several development environments, fast, from command line, and use custom domain names to access them and its related services.
A good editor, like Sublime Text or Visual Studio Code. (I know, I don't use PHP Storm and don't want to).

Since I had tried WSL before (and failed), I decided to get rid of it and move directly to install a VM.

An Ubuntu server checks first three items in the above list, but the next two can get messy. Sharing source code files with the Windows host as a Samba share defeats the purpose because its bad performance, symlinks weirdness, among other things. Sharing it using sshfs could be an option, but then how could I control docker?

Editor

The solution came on a recent iteration on Visual Studio Code: Remote code extensions , specifically, Remote Coding via SSH. This extension allowed me to open remote directories through SSH and work with them. This is not like running Dreamweaver to edit files on a server (remember?). It works splitting the editor in two pieces, a client (the UI) and a server, where the indexing and language processing extensions run. The result is a very responsive editor, while working remotely with all the extensions I would normally use. Even the integrated terminal in the editor opens automatically a bash prompt in the server, a really useful feature.

This remote feature in Visual Studio Code gives me the full Linux experience right from the editor.

However, I normally access docker websites using custom URLs, a pretty common practice. If using docksal, for example: super-awesome-client.docksal, db.super-awesome-client.docksal, etc., you get the idea. I needed to find a way to re-route custom URLs to my VM.

Networking

On a Mac or on Linux this would work out of the box after installing docksal (or lando or whatever docker wrapper you use). I wanted this to work the same way:

I didn't want to configure each custom website manually
This had to work on new sites created with docksal
Should work on any network

To achieve this transparent routing, I needed a static IP for the VM, so I could always reference it if I was at home, at a coffee shop, or in an airport. I'm using Hyper-v, so I created a NAT using the instructions below:

Windows 10: How to setup NAT network for Hyper-V guests?

Then in the VM, configured the static IP address using netplan, created file /etc/netplan/01-netcfg.yaml (I used the same IP address as the guide to create the NAT above).

network:
    ethernets:
        eth0:
            dhcp4: no
            dhcp6: no
            addresses: [192.168.200.11/24]
            gateway4: 192.168.200.1
            nameservers:
                addresses: [8.8.8.8]

After enabling the ssh server in the VM , and configuring my ssh keys I was able to jump into the server from Windows very quickly, or even configure ConEmu or the new Windows Terminal to automatically connect to ssh.

To access the web servers running in docker containers created with docksal, I installed a DNS proxy : Acrylic DNS Proxy, and used this hosts configuration:

192.168.200.11 *.docksal

Visual Studio Code Configuration

In Visual Studio Code I installed the Remote Development extensions for SSH. The configuration was straight forward and worked really well (the extension guides you until you are successfully connected to a SSH server).

Huge Drupal projects were indexed really fast and the performance when browsing them was like working on Linux. Debugging worked out of the box (this thanks to Docksal configs, but still, using docker-compose shouldn't be a problem).

The only downside I've found is that I can't commit, push or pull using the editor, which is not a big deal since I can use the integrated terminal. From the documentation:

If you clone a Git repository using SSH and your SSH key has a passphrase, VS Code's pull and sync features may hang when running remotely. Either use an SSH key without a passphrase, clone using HTTPS, or run git push from the command line to work around the issue.

I ended up creating an SSH key for my repos without a passphrase.

Recap

Working on PHP natively on Windows it is still impossible, but using a VM and Visual Studio Code with Remote Code extensions works very well, with the price of more memory usage, but memory is cheap now, and that is one of the reasons I left Apple in the first place: to have an upgradable development computer.

One unexpected bonus is that I can recover from suspend and the VM will load up with all services that were running, something that in Linux didn't happen; I had to to run fin up or docker-compose up again after the laptop woke.

Aside: WSL2

The Windows Subsystem for Linux v2 is about to be released, which uses a custom Linux Kernel in a very lightweight VM (it is already in Windows Internals fast ring, but expected to reach public release until 2020).

In theory, it has all the same good points of this setup with a VM, and seems so be a great setup for a future development environment because of these:

The VM file system can be accessed directly from Windows using a p9 file system (that alone is awesome).
The performance is very close to native one.
Shared memory and processor cores.
Much tighter integration, it is possible to run windows programs from within the VM.

This technology can really change the development experience on Windows, lets see if it is up to the expectations when it is released (because I won't subscribe to the fast ring in my main development computer).

Stranger things

While setting the environment I found an error that didn't allow me to clone big repos or to push changes. It turned out to be a bug in my WiFi drivers. So if you find the error below and have an Intel WiFi card, update your drivers.

$ git clone git@somrerepo-bla-bla.git
Cloning into 'somedir'...
error: RPC failed; curl 56 GnuTLS recv error (-12): A TLS fatal alert has been received.
fatal: The remote end hung up unexpectedly

Git errors when using Hyper-V

Arturo Linares — Wed, 18 Sep 2019 00:20:25 +0000

I started having some issues when using git inside a Linux VM while setting up an experimental Drupal development environment. I wasn't able to clone repositories because I was getting some errors saying the remote host hanged unexpectedly, or couldn't push changes. However I was able to clone shallow repos, but strangely Bitbucket and Gitlab seemed to be more prone to these errors.

$ git clone git@....
Cloning into 'somedir'...
error: RPC failed; curl 56 GnuTLS recv error (-12): A TLS fatal alert has been received.
fatal: The remote end hung up unexpectedly

At some point, disabling offloading solved the issue: Disabling Offloading on Ubuntu 18.04 with Netplan and/or systemd-networkd

It apparently worked, but soon I found that only worked when using Github. Of course, that didn't make any sense. When I tried to push changes to Bitbucket or Gitlab I had again the same strange errors saying the remote host hung up unexpectedly.

I tried to debug git using GIT_SSH_COMMAND, tried using https and even recompiled git to use OpenSSL instead of GnuTLS. Nothing seemed to work... Until I found an answer from this guy who found the solution): I needed to update my Wi-Fi drivers.

Strange error indeed.

TLDR : A bug between Hyper-V and my Wi-Fi card (Intel 5060) was causing this. Updating the drivers solved it:

https://downloadcenter.intel.com/download/28876/Windows-10-Wi-Fi-Drivers-for-Intel-Wireless-Adapters?v=t

How to make a title block with a custom background

Arturo Linares — Fri, 16 Mar 2018 06:00:00 +0000

How to make a title block with a custom background

A common request I’ve received lately when building sites in Drupal is to make the background of the page title configurable to allow the user upload and change it.

Of course, this could be solved with a new block type with two fields, one for the image and one for the title. But then we would need to add the block for each page where we want to show the title.

How does Drupal handle page titles? Looking in the block admin page, I found that actually it is a block. Well, we need the same functionality of that block in our new one.

So looking for the class that contains the block machine name page_title_block I found Drupal\Core\Block\Plugin\Block\PageTitleBlock which implements TitleBlockPluginInterface. This is all the interface:


    interface TitleBlockPluginInterface extends BlockPluginInterface {

      /**
       * Sets the title.
       *
       * @param string|array $title
       *   The page title: either a string for plain titles or a render array for
       *   formatted titles.
       */
      public function setTitle($title);

    }

Searching what uses that interface, I found that BlockPageVariant uses it to assign it a title:

    foreach ($this->blockRepository->getVisibleBlocksPerRegion($cacheable_metadata_list) as $region => $blocks) {
          /** @var $blocks \Drupal\block\BlockInterface[] */
          foreach ($blocks as $key => $block) {
            $block_plugin = $block->getPlugin();
            if ($block_plugin instanceof MainContentBlockPluginInterface) {
              $block_plugin->setMainContent($this->mainContent);
              $main_content_block_displayed = TRUE;
            }
            elseif ($block_plugin instanceof TitleBlockPluginInterface) {
              $block_plugin->setTitle($this->title);
            }
      //...

This means that if I create a custom block that implements TitleBlockPluginInterface I will get the page title automatically (thanks to BlockPageVariant)? Apparently, yes.

Lets start writing the custom block extending PageTitleBlock to use the title field implementation:

    /**
     * @Block(
     *   id = "my_image_title_block",
     *   admin_label = "Image title block"
     * )
     */
    class ImageTitleBlock extends PageTitleBlock {

      /**
       * The image field.
       *
       * @var mixed
       */
      protected $image;

      public function defaultConfiguration() {
        return [
          'image' => [
            'value' => '',
          ],
          'label_display' => FALSE,
        ] + parent::defaultConfiguration();
      }

      /**
       * {@inheritdoc}
       */
      public function build() {
        $build = [];

        if (isset($this->configuration['image'])
          && !empty($this->configuration['image'])) {

          $image_field = $this->configuration['image'];
          $image_uri = File::load($image_field[0]);

          $build['image'] = [
            'uri' => ImageStyle::load('title_bar')->buildUrl($image_uri->getFileUri()),
          ];
        } else {
          $build['image']['#markup'] = '[' . t('Picture') . ']';
        }


        return $build + parent::build();
      }

      /**
       * {@inheritdoc}
       */
      public function blockForm($form, FormStateInterface $form_state) {

        $form['image'] = array(
          '#type' => 'managed_file',
          '#upload_location' => 'public://images/',
          '#title' => $this->t('Image'),
          '#description' => $this->t("Background image"),
          '#default_value' => $this->configuration['image'],
          '#upload_validators' => array(
            'file_validate_extensions' => array('gif png jpg jpeg'),
            'file_validate_size' => array(25600000),
          ),
          '#states' => array(
            'visible' => array(
              ':input[name="image_type"]' => array('value' => t('Upload New Image')),
            )
          )
        );

        return $form;
      }

      public function validateConfigurationForm(array &$form, FormStateInterface $form_state) {
        if (!$form_state->getValue('image')) {
          $form_state->setError($form['image'], $this->t('Select an image'));
        }
      }

      /**
       * {@inheritdoc}
       */
      public function blockSubmit($form, FormStateInterface $form_state) {
        /* Fetch the array of the file stored temporarily in database */
        $image = $form_state->getValue('image');

        $this->configuration['image'] = $image;

        /* Load the object of the file by it's fid */
        $file = \Drupal\file\Entity\File::load($image[0]);

        /* Set the status flag permanent of the file object */
        $file->setPermanent();

        /* Save the file in database */
        $file->save();
      }
    }

After clearing Drupal cache, you will find your new shiny block waiting for you in the Add block list. The build method returns the processed image URL (using title_bar style) instead of the render array for an image so we can use it later.

Let’s create title_bar image style and configure it accordingly (in my case, resize and crop to a predefined size will do).

Then, update your theme to make sure that image url shows in a style attribute, or a data-* if you’re using javascript to load the image.

block—image-title-block.html.twig

    <div{{ attributes.setAttribute('style',
        'background: #787c8a url(' ~ content.image.uri ~ ') no-repeat center;') }}>
      {{ title_prefix }}
      {% if label %}
        <h2{{ title_attributes }}>{{ label }}</h2>
      {% endif %}
      {{ title_suffix }}
      {% block content %}
        {{ content }}
      {% endblock %}
    </div>
    ...

Lets test it!

Setup Solr and Drupal using Docker

Arturo Linares — Wed, 07 Mar 2018 04:59:16 +0000

Setup Solr and Drupal using Docker

Last time I tried to create from scratch a Drupal instance that used a Solr server wasn’t as easy as I expected. Maybe command line tools for installing modules have spoiled me and I just expected that running something like drupal module:install solr would configure everything for me.

Here is how I made it work, but please consider I’m not a Solr expert. If you know a better way to do this please let me know.

Setup Drupal

Let’s suppose we want to create a blog that uses a Solr for searching its contents.

First, to create a basic Drupal install we can use a recently released project from WeKnow to quickly create a Drupal instance using Docker. Follow the README file keeping the default .env values so we can install Drupal using this command:

$ docker-compose exec --user=82 php drupal site:install standard --db-host=mariadb --db-name=drupal --db-user=drupal --db-pass=drupal --quiet

Don’t forget to add the hosts entry, as mentioned in the README file, or use a proxy like hotel. The objective is to use a custom domain name that the web server can identify (if we use localhost and in the .env file we used HOST_NAME=drupal.vm the web server won’t serve the request).

You can verify it is working if you see a fresh Drupal install when you visit http://drupal.develop.

Install Solr

Let’s install Solr now. We will need some config files found in the module to setup our Solr core, so lets download it inside the container:

$ docker-compose exec --user=82 php composer req drupal/search_api_solr

We will create a sample core named blog. Copy the files located at web/modules/contrib/search_api_solr/solr-conf/5.x/ to env/solr/blog/conf/ and create this file in env/solr/blog/core.properties (you will need to create the env dir):

env/solr/blog/core.properties

name=blog
config=solrconfig.xml
schema=schema.xml
dataDir=/var/lib/solr/books

Update docker-compose.yml to include the solr container and config its port and a way to access it using traefik (using http://solr.drupal.develop if using the defaults in .env):

docker-compose.yml

services:
  # ...
  solr:
    image: solr:5.5
    labels:
      - 'traefik.backend=solr'
      - 'traefik.port=8983'
      - 'traefik.frontend.rule=Host:solr.${HOST_NAME}'
    volumes:
      - ./env/solr/blog:/opt/solr/server/solr/blog
      - solrdata:/var/lib/solr

volumes:
  mysqldata:
    driver: "local"
  solrdata:
    driver: "local"

Don’t forget to add the solr.drupal.develop to your hosts file).

After running docker-compose up -d (and adding to your hosts file the entry for solr.drupal.development) you will see this error in solr:

This happened because the user and the uid has to match with the user the solr process runs as in the solr container. To fix this we have to create another container that fixes the perms where the core will write the index.

We will use just-containers/base-alpine image, which has utilities to do this (s6).

Add the docker file to update fix the perms and update docker-compose to use this new container:

./dockerfiles/solr_data/Dockerfile

FROM just-containers/base-alpine
VOLUME ["/var/lib/solr"]
ADD root /

./dockerfiles/solr_data/root/etc/fixattrs.d

/var/lib/solr false solr,8983 0755

And update the docker-compose.yml to include this image:

services:
  solr:
    image: solr:5.5
    labels:
      - 'traefik.backend=solr'
      - 'traefik.port=8983'
      - 'traefik.frontend.rule=Host:solr.${HOST_NAME}'

    # add the volumes from the new container
    volumes_from:
      - solr_data
    # the core configs
    volumes:
      - ./env/solr/blog:/opt/solr/server/solr/books

  # added container to fix perms
  solr_data:
    build: ./dockerfiles/solr_data
    volumes:
      - real_solr_data:/var/lib/solr

volumes:
  mysqldata:
    driver: "local"
  real_solr_data:
    driver: "local"

Now if you run docker-compose up -d and you visit the Solr page you should see your errors fixed.

Configure Solr

Now that we have the server running, lets configure Drupal so we can use it as a search engine. Fortunately, most of the hard work is solved by enabling search_api_solr_default module.

$ docker-compose exec --user=82 php drupal moi search_api_solr_defaults

Go to the Search API configs (/admin/config/search/search-api) and you will see a new Solr server entry, although with errors because it is missing the server name and the core it will use.

Click edit and set the host (ie. the solr container name, solr) and the core we created before (blog):

Click save and you should be able to to see all green when you go back to the Search API config page. You can now start using your solar server for searching.

The defaults module contains a sample search page to easily get you started in /solr-search/content. It is implemented as a View.

References

You might find this links useful:

Permalink: Setup Solr and Drupal using Docker