DEV Community: Adamo Crespi

How to debug ANY Symfony command simply passing `-x`

Adamo Crespi — Thu, 04 Apr 2024 16:46:10 +0000

Debugging a Symfony console command requires setting some environment variables (depending on your actual configuration of xDebug):

XDEBUG_SESSION=1 (Docs)
XDEBUG_MODE=debug
XDEBUG_ACTIVATED=1

You can check the purpose of these environment variables on the xDebug's Docs.

TL;DR

So, launching a Symfony console command in debug mode would look like this:



XDEBUG_SESSION=1 XDEBUG_MODE=debug XDEBUG_ACTIVATED=1 php bin/console my:command --an-option --an argument

Using the listener below, instead, you can debug any Symfony command this way:



bin/console my:command --an-option --an argument -x

Really shorter and faster, also to debug a command on the fly, without having to move the cursor to the beginning of the command (that is so boring to do on the CLI! 🤬).

The -x option starts the "magic" and the listener actually performs the trick.

Using this listener, you can actually debug ANY Symfony command, also if it doesn't belong to your app, but belongs to, for example, Doctrine, a third party bundle or even Symfony itself.

It's really like magic!

WARNING

The command will not work to debug the Symfony's container building: in that phase, in fact, the listeners are not still active.

If you need to debug the configuration of a bundle or any other part of Symfony that is before the container is built and services configured, then you still need to go the old way, with full declaration of env variables directly in the command call in the command line.

The `RunCommandInDebugModeEventListener` listener

The listener works thanks to the ConsoleEvents::COMMAND event.

It simply searches for the flag -x (or --xdebug) and, if it finds it, then restarts the command setting the environment variables required by xDebug to work.

The restart of the command is done through the PHP function passthru().

The rest of the code is sufficiently self explanatory, so I'm not going to explain it.

This is the listener: happy Symfony commands debugging!



<?php

declare(strict_types=1);

namespace App\EventListener;

use Symfony\Component\Console\Command\Command;
use Symfony\Component\Console\Command\HelpCommand;
use Symfony\Component\Console\ConsoleEvents;
use Symfony\Component\Console\Event\ConsoleCommandEvent;
use Symfony\Component\Console\Input\ArgvInput;
use Symfony\Component\Console\Input\InputOption;
use Symfony\Component\EventDispatcher\Attribute\AsEventListener;

#[AsEventListener(ConsoleEvents::COMMAND, 'configure')]
class RunCommandInDebugModeEventListener
{
    public function configure(ConsoleCommandEvent $event): void
    {
        $command = $event->getCommand();

        if (false === $command instanceof Command) {
            throw new \RuntimeException('The command must be an instance of ' . Command::class);
        }

        if ($command instanceof HelpCommand) {
            $command = $this->getActualCommandFromHelpCommand($command);
        }

        $command->addOption(
            name: 'xdebug',
            shortcut: 'x',
            mode: InputOption::VALUE_NONE,
            description: 'If passed, the command is re-run setting env variables required by xDebug.',
        );

        if ($command instanceof HelpCommand) {
            return;
        }

        $input = $event->getInput();
        if (false === $input instanceof ArgvInput) {
            return;
        }

        if (false === $this->isInDebugMode($input)) {
            return;
        }

        if ('1' === getenv('XDEBUG_SESSION')) {
            return;
        }

        $output = $event->getOutput();
        $output->writeln('<comment>Relaunching the command with xDebug...</comment>');

        $cmd = $this->buildCommandWithXDebugActivated();

        \passthru($cmd);
        exit;
    }

    private function getActualCommandFromHelpCommand(HelpCommand $command): Command
    {
        $reflection    = new \ReflectionClass($command);
        $property      = $reflection->getProperty('command');
        $actualCommand = $property->getValue($command);

        if (false === $actualCommand instanceof Command) {
            throw new \RuntimeException('The command must be an instance of ' . Command::class);
        }

        return $actualCommand;
    }

    private function isInDebugMode(ArgvInput $input): bool
    {
        $tokens = $this->getTokensFromArgvInput($input);

        foreach ($tokens as $token) {
            if ('--xdebug' === $token || '-x' === $token) {
                return true;
            }
        }

        return false;
    }

    /**
     * @return array<string>
     */
    private function getTokensFromArgvInput(ArgvInput $input): array
    {
        $reflection     = new \ReflectionClass($input);
        $tokensProperty = $reflection->getProperty('tokens');
        $tokens         = $tokensProperty->getValue($input);

        if (false === is_array($tokens)) {
            throw new \RuntimeException('Impossible to get the arguments and options from the command.');
        }

        return $tokens;
    }

    private function buildCommandWithXDebugActivated(): string
    {
        $serverArgv = $_SERVER['argv'] ?? null;
        if (null === $serverArgv) {
            throw new \RuntimeException('Impossible to get the arguments and options from the command: the command cannot be relaunched with xDebug.');
        }

        $script = $_SERVER['SCRIPT_NAME'] ?? null;
        if (null === $script) {
            throw new \RuntimeException('Impossible to get the name of the command: the command cannot be relaunched with xDebug.');
        }

        $phpBinary = PHP_BINARY;
        $args      = implode(' ', array_slice($serverArgv, 1));

        return "XDEBUG_SESSION=1 XDEBUG_MODE=debug XDEBUG_ACTIVATED=1 {$phpBinary} {$script} {$args}";
    }
}

Moving Symfony files in the Docker container

Adamo Crespi — Thu, 22 Nov 2018 11:22:48 +0000

Once you have created the first configuration of your basic Docker stack, it is time to move the Symfony’s files into Docker’s web server service.

This will make us able to run the symfony application from inside the web server container, getting the real benefits of using a system as Docker (same environment on any machine, above all).

So, we need a way to move in the container all the files required to make our project run.

How do we do this?

The answer to our question is a really simple word: “Dockerfile” (never heard of it? Ok, it is a not so simple word! But it is simple to use it anyway! )

From the Docker documentation about “Dockerfile”:

Docker can build images automatically by reading the instructions from a Dockerfile

A Dockerfile is a text document that contains all the commands a user could call on the command line to assemble an image.

Using docker build users can create an automated build that executes several command-line instructions in succession.

So, basically, all what you write in the console to configure your environment has to be put in a Dockerfile to automate the process.

Anytime someone wants to configure the environment to run our app, (s)he will not have to take care of the details as they are all in the Dockerfile: easy and fast!

So, let’s prepare our app to use Dockerfiles.

Create the folder structure and the `Dockerfile`

First of all we need to create some folders.

So, in the root of your app, create a folder called docker, inside it create another folder called build and inside it another folder called apache, then inside it create a file called Dockerfile.

The final folder structure will be like this:

The next steps are:

Put in the Dockerfile the commands required to build the apache container;
Configure docker-compose.yaml to use this new Dockerfile.

As first thing, instruct the Dockerfile about the Apache image we want to use:

# docker/build/apache/Dockerfile

FROM httpd:2.4

The httpd:2.4 is exactly the same we used in our docker-compose.yaml file.

Now, edit the docker-compose.yaml file so it will use this Dockerfile: we will do this using the build key node:

version: '3.7'
services:
  # "php" was "language" in previous example
  ...

  # Configure the database
  ...

  # Configure Apache
  apache:
    # image: httpd:2.4 <- Remove this node
    build: docker/build/apache
    ports:
      - "8100:80"

Once edited the docker-compose.yaml file, first run docker-compose down to stop all the containers, then run docker-compose build:

MacBook-Pro-di-Aerendir:app-aragog-www Aerendir$ docker-compose down
Stopping app-aragog-www_mysql_1 ... done
Stopping app-aragog-www_apache_1 ... done
Removing app-aragog-www_php_1 ... done
Removing app-aragog-www_mysql_1 ... done
Removing app-aragog-www_apache_1 ... done
Removing network app-aragog-www_default
MacBook-Pro-di-Aerendir:app-aragog-www Aerendir$ docker-compose build
php uses an image, skipping
mysql uses an image, skipping
Building apache
Step 1/1 : FROM httpd:2.4
 ---> fb2f3851a971

Successfully built fb2f3851a971
Successfully tagged app-aragog-www_apache:latest

As you can see, the php and mysql containers are skipped as they use an image, while the apache container is built as it uses a Dockerfile.

Try to start the containers with docker-compose up -d:

MacBook-Pro-di-Aerendir:app-aragog-www Aerendir$ docker-compose up -d
Creating network "app-aragog-www_default" with the default driver
Creating app-aragog-www_mysql_1 ... done
Creating app-aragog-www_php_1 ... done
Creating app-aragog-www_apache_1 ... done

And access the URL http://127.0.0.1:8100: you see the message “It works!”.

Ok, the first step is done: the Apache web server is continuing to work also if we are using a different method to configure it.

Now we need to do the second step: add the Symfony files to the document root of Apache.

To do this, we first need to understand what is the Docker Context as it will be extremely useful.

The Docker Context: what is it, why we need to understand it and how to use it

Before diving in the files and their copy in the container, we need to understand what is Docker Context.

From the documentation about build:

The docker build command builds Docker images from a Dockerfile and a “context”.

A build’s context is the set of files located in the specified PATH or URL.

The build process can refer to any of the files in the context.

For example, your build can use a COPY instruction to reference a file in the context.

So, to copy our Symfony files to the container, we need to make them available in the context.

We are using a docker-compose.yaml file, so we need to understand how to specify the context when using it.

The solution? The context key of the build node.

The documentation says:

Either a path to a directory containing a Dockerfile, or a url to a git repository.

When the value supplied is a relative path, it is interpreted as relative to the location of the Compose file.

This directory is also the build context that is sent to the Docker daemon.

But there is more: we can also set a specific Dockerfile to use when setting the context.

This is useful as because

By default the docker build command will look for a Dockerfile at the root of the build context

So, trying to recap what we have read until now in the documentation:

We can copy or add files in the container from our machine only if they are in the “context”;
Using the build key in the docker-compose.yaml file with a path (build: path/to/folder):
1. Uses the Dockerfile found in this path;
2. Uses the content of this path as “context” (so we can access only files in it)
In our docker-compose.yaml file, we can:
1. Specify a specific “context” to use to build the service/container;
2. Specify a specific Dockerfile to use to build the service/container.

Interesting, isn’t it? Let’s put all those information at work to understand how to copy our Symfony files to the Apache container.

As usual, we will start small.

Trying to copy only a single file `test.txt`

First, we need a file to copy: create an empty file called test.txt and put it in the root directory of the project: we will try to copy it in the container.

Now, let’s edit our docker-compose.yaml file to specify the desired context and Dockerfile:

# docker-compose.yaml

version: '3.7'
services:
  # "php"
  ...

  # Configure the database
  ...

  # Configure Apache
  apache:
    # build: docker/build/apache <- Remove this node
    # And recreate it as follows
    build:
      context: .
      dockerfile: docker/build/apache/Dockerfile
    ports:
      - "8100:80"

As you can see, we removed the path to the apache folder and, instead, added two more keys:

context, that sets the context to the current folder (that is the root folder, as the docker-compose.yaml file is in the root);
dockerfile, that specifies which Dockerfile to use to build the container/service that runs Apache

This has two effects:

Sets the Docker Context to our root folder as the docker-compose.yaml file is in the root (so we can now use – so, we can copy, too – any file in our root folder);
Explicitly specifies the Dockerfile to use, without relying on the auto-discovery capabilities of Docker.

Before this modification, instead, the context was the folder docker/build/apache (and so we were able to copy only files and folders in that folder) and the Dockerfile was found thanks to the auto-discovery capabilities of Docker in finding Dockerfiles in the provided paths.

Now that we have our entire root folder in the Docker Context, edit the Dockerfile that builds Apache to make it able to copy the file test.txt:

# docker/build/apache/Dockerfile

FROM httpd:2.4

# Copy Symfony files: COPY [FROM_MACHINE] [TO_CONTAINER]
COPY test.txt copied-test.txt

We are using the COPY instruction.

As the context is our root folder, we simply use the file name test.txt and copy it to the container as copied-test.txt.

Now run the command docker-compose build:

MacBook-Pro-di-Aerendir:app-aragog-www Aerendir$ docker-compose build
php uses an image, skipping
mysql uses an image, skipping
Building apache
Step 1/2 : FROM httpd:2.4
 ---> fb2f3851a971
Step 2/2 : COPY test.txt copied-test.txt
 ---> 287d3e3b6a04

Successfully built 287d3e3b6a04
Successfully tagged app-aragog-www_apache:latest

As usual, php and mysql are skipped, BUT apache has now two steps, and the second one is exactly the COPY instruction.

Let’s see if the file was really copied.

First, recreate the containers running the up command:

MacBook-Pro-di-Aerendir:app-aragog-www Aerendir$ docker-compose up -d
Recreating app-aragog-www_apache_1 ... done
Starting app-aragog-www_php_1 ... done

Then enter the container (use docker ps to find the ID of the apache container) and list the files in it:

MacBook-Pro-di-Aerendir:app-aragog-www Aerendir$ docker exec -it e4ed20c531ae bash
root@e4ed20c531ae:/usr/local/apache2# ls
bin build cgi-bin conf copied-test.txt error htdocs icons include logs modules

BINGO! Our test.txt file is there with the name copied-test.txt!

Now that we can use the context to copy files in the container from it, we are ready to further explore the copying of Symfony files to the container.

Moving Symfony’s files into the container

Now that we know how to move files to the container, we are ready to move our entire app to the container.

We can use a simple instruction like this:

COPY . ./htdocs

This will move the entire context (that is our entire root folder) in the creating container.

WARNING: This is not the best way of doing this. For the moment we make simple things, so it is ok to copy the entire root folder.

Later in this series of posts, we will learn how to refine the files copied in the container to speed up and optimize the process.

For the moment, let’s try this quick and dirty solution.

Edit the docker/build/apache/Dockerfile, adding the above instruction:

FROM httpd:2.4

# Copy Symfony files
# COPY test.txt copied-test.txt <- Remove this line (and also the text.txt file!)

COPY . ./htdocs

Rebuild the containers:

MacBook-Pro-di-Aerendir:app-aragog-www Aerendir$ docker-compose build
php uses an image, skipping
mysql uses an image, skipping
Building apache
Step 1/2 : FROM httpd:2.4
 ---> fb2f3851a971
Step 2/2 : COPY . ./htdocs
 ---> 3f3528537fe9

Successfully built 3f3528537fe9
Successfully tagged app-aragog-www_apache:latest

And update the running ones:

MacBook-Pro-di-Aerendir:app-aragog-www Aerendir$ docker-compose up -d
Recreating app-aragog-www_apache_1 ... 
Recreating app-aragog-www_apache_1 ... done
Starting app-aragog-www_php_1 ... done

Now enter the container and check if the files are in it:

MacBook-Pro-di-Aerendir:app-aragog-www Aerendir$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
8b10afca67dd app-aragog-www_apache "httpd-foreground" 8 seconds ago Up 4 seconds 8100->80/tcp app-aragog-www_apache_1
c633398a7d65 mysql:5.7 "docker-entrypoint.s…" 15 hours ago Up 15 hours 3306/tcp app-aragog-www_mysql_1
MacBook-Pro-di-Aerendir:app-aragog-www Aerendir$ docker exec -it 8b10afca67dd bash
root@8b10afca67dd:/usr/local/apache2# ls
Docker.md README.md bin build cgi-bin composer.json composer.lock conf config docker docker-compose.yaml docs error htdocs icons include logs modules public src symfony.lock var vendor

Are they there? Very good! We have just moved all the Symfony’s files in the container!

Let’s try to open our Symfony app: go to http://127.0.0.1:8100…

The message “It works!”? What’s happening?

Ok, try this: http://127.0.0.1:8100/public… A list of files? ? And there is our index.php listed ?

Click on it…

What are you seeing? What, the content of the php file, not interpreted but instead served as a simple text file? ?

What is damn happening? ?

Making Apache interpret our PHP files

Technically, we have just created our stack and it works because we are able to make up and running our containers with Apache and MySQL (PHP currently doesn’t work, in fact you have never seen it when running docker ps!).

And it works because we can see the Symfony’s index.php content.

So, technically it is working, practically it isn’t as we are not able to serve the php files as interpreted webpages.

Why does this happen?

Let’s recap our current configuration:

version: '3.7'
services:
  # "php" was "language" in previous example
  php:
    image: php:7.2

  # Configure the database
  mysql:
    image: mysql:5.7
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-root}

  # Configure Apache
  apache:
    build:
      context: .
      dockerfile: docker/build/apache/Dockerfile
    ports:
      - "8100:80"

As you can see we are building three containers: one for php, one for mysql and one for apache.

Leaving apart for the moment the container for mysql, let’s examine deeper what we are doing with php and apache containers.

We already know that the php container doesn’t work and never starts nor is built: maybe it is time to understand why!

To build php and apache we are using two images:

PHP: php:7.2
Apache: httpd:2.4 (this is in the Dockerfile in docker/build/apache)

So, the question at this point is: what happens when we build these images?

The first thing we know is that the php container is skipped. When running docker-compose build, in fact, we can read a clear message:

php uses an image, skipping

The second thing we know is that the php container is never run: using docker ps, in fact, we have never seen it in the list:

MacBook-Pro-di-Aerendir:app-aragog-www Aerendir$ docker ps

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
d109c4b4c431 app-aragog-www_apache "httpd-foreground" 11 minutes ago Up 11 minutes 8100->80/tcp app-aragog-www_apache_1
844167c3154f mysql:5.7 "docker-entrypoint.s…" 12 minutes ago Up 12 minutes 3306/tcp app-aragog-www_mysql_1

So, in the end, we have this situation:

We have the Apache container running, but without PHP;
We have the PHP container NOT running, and we anyway don’t have PHP (but, as seen in the previous post about how to configure the Apache web server on Docker, if we try to check the version of PHP, we get it, so it seems it is working: very confusing!).

Why?

Let’s see what these images do.

Introducing Docker images (and containers)

A Docker image is nothing more than a Dockerfile with some instructions to build a container.

More precisely

An instance of an image is called a container. You have an image, which is a set of layers as you describe. If you start this image, you have a running container of this image. You can have many running containers of the same image.

You can read more about images and layers here.

The relvant part is this:

A Docker image is built up from a series of layers. Each layer represents an instruction in the image’s Dockerfile. Each layer except the very last one is read-only.

…

Each layer is only a set of differences from the layer before it. The layers are stacked on top of each other. When you create a new container, you add a new writable layer on top of the underlying layers. This layer is often called the “container layer”. All changes made to the running container, such as writing new files, modifying existing files, and deleting files, are written to this thin writable container layer. The diagram below shows a container based on the Ubuntu 15.04 image.

And this is the representation of what you’ve read until now:

But from where these images come?

From the Docker Hub!

So, to understand why our httpd image doesn’t interpret correctly PHP files and why our php image doesn’t run in a container, we can read the Dockerfiles of them.

And here they are:

httpd
php

Let’s start examining first the httpd page.

If your read it, there is a section called “How to use this image.” that states this:

This image only contains Apache httpd with the defaults from upstream. There is no PHP installed, but it should not be hard to extend. On the other hand, if you just want PHP with Apache httpd see the PHP image and look at the -apache tags.

It is clear, now?

Our service doesn’t interpret PHP files because using the image httpd simply we don’t have it!

So, bascially, we are using the wrong image ?.

Let’s use the right one!

Configuring an Apache web server with PHP (for real!)

So, we need to use the the php image: and this is what we were already using.

But reading the description from the httpd image, it is not sufficient to use the image php:7.2 like we did: we need to use an image tagged with -apache!

So, go to the php image page on Docker Hub.

If you read the page, there is a section called “With Apache” that says:

More commonly, you will probably want to run PHP in conjunction with Apache httpd. Conveniently, there’s a version of the PHP container that’s packaged with the Apache web server.

And the instructions say that we need to use the image php:7.2-apache: very very well ?

So, what happened was this:

We saw the version of PHP because effectively we were building a container with it, so Docker was able to find it and returned its version;
We were not able to serve an interpreted PHP file as we were using the Apache container that is not equipped with PHP.

So, what we need to do now is:

Change the image we use for PHP in docker-compose.yaml;
Use a Dockerfile to create the php service (to merge the current configuration of the Apache container in the PHP container);
Copy all files from our machine to the new PHP image (instead of copying them in the Apache container);
Remove the old service with the httpd image (as it will be substituted by the php:7.2-apache image);
Remove the folder docker/build/apache as not needed anymore.

This is the resulting docker-compose.yaml file:

version: '3.7'
services:
  # "php" was "language" in previous example
  php:
    build:
      context: .
      dockerfile: docker/build/php/Dockerfile
    ports:
      - "8100:80"

  # Configure the database
  mysql:
    image: mysql:5.7
    environment:
      - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-root}

  # Configure Apache <- THIS NODE AS TO BE ENTIRELY REMOVED
  # apache:
  # build:
  # context: .
  # dockerfile: docker/build/apache/Dockerfile
  # ports:
  # - "8100:80"

Concretely, we have renamed apache to php and changed the path to the Dockerfile: nothing really complex.

Now the last thing remained is renaming the folder docker/build/apache in docker/build/php.

We also need to change the folder in which we move the files of our app as per documentation:

COPY . /var/www/html/

Where src/ is the directory containing all your PHP code.

So, in the file docker/build/php/Dockerfile, we need to change the FROM instruction to use the image php:7.2-apache:

# FROM httpd:2.4 <- Change this to use the PHP image
FROM php:7.2-apache

# Copy Symfony files
# COPY . ./htdocs # <- Remove this
COPY . /var/www/html/ # <- Keep attention: we have removed the "." (dot) at the beginning of the destination path

Last thing to do is building the image:

MacBook-Pro-di-Aerendir:app-aragog-www Aerendir$ docker-compose build
Building php
Step 1/2 : FROM php:7.2-apache
7.2-apache: Pulling from library/php
be8881be8156: Already exists
69a25f7e4930: Already exists
65632e89c5f4: Already exists
cd75fa32da8f: Already exists
15bc7736db11: Pull complete
b2c40cef4807: Pull complete
f3507e55e5eb: Pull complete
e6006cdfa16b: Pull complete
a3ed406e3c88: Pull complete
745f1366071d: Pull complete
bdfcada64ad8: Pull complete
86f2b695cc77: Pull complete
5f634a03970a: Pull complete
a329a7ebde19: Pull complete
fb3d2649f534: Pull complete
Digest: sha256:8188b38abe8f3354862845481452cd3b538bc0648e3c5cdef4ef9ee9365fe2d3
Status: Downloaded newer image for php:7.2-apache
 ---> 5e5a59788e34
Step 2/2 : COPY . /var/www/html/
 ---> a0cfd25e4828

Successfully built a0cfd25e4828
Successfully tagged app-aragog-www_php:latest
mysql uses an image, skipping

As we have removed a service, we need to run first docker-compose down --remove-orphans and then docker-compose up -d.

Once containers are created again, we can go to http://127.0.0.1:8100/public and we will see the most beautiful page in the world ?:

We are now ready to start building our Symfony based app… or maybe we aren’t?

Mmm, no, we are not still ready ???

What do we need to do now? ?

Conclusions and next steps

Well, also if we are able to copy the Symfony’s files in the web server container (that now is not httpd anymore, but, instead, php:7.2-apache) and we are able to make the web server serve interpreted PHP files, we still have some other things to do:

Understand why we can see Symfony only pointing directly to the public folder (SPOILER: DocumentRoot?);
Checking our web server meets all the Symfony’s requirements;
Maybe, also refine our copying of files as it is currently very dirty and heavy.

But for the moment we have reached some great goals:

We know how to create a real Apache web server on Docker;
We know how to deal with images, to understand how to chose and use them (now, for example, you can use MongoDB if you like: go to search for it!);
We know how to copy files in the containers and in doing this, we have understood what are Dockerfiles (one of the foundation of Docker!), what is the Docker Context, how it works and how we can manipulate it (almost: there is at least one other thing you need to know to master it… Next post!).

Those are a lot of things and we are closer to our goal of fully using Symfony on Docker!

The next post will teach you how to solve the problems I mentioned.

In the meantime, remember to “Make. Ideas. Happen.”.

I wish you flocking users!

L'articolo Moving Symfony files in the Docker container proviene da ÐΞV Experiences by Serendipity HQ.

How to dockerize a Symfony application: Introduction and the web server

Adamo Crespi — Wed, 21 Nov 2018 12:54:29 +0000

Starting to use Docker is not an easy task. Starting to use Symfony on Docker is far more complex.

There are too many things to understand before being able to dockerize a Symfony application.

I studied Docker for more than one month before being able to dockerize our Symfony applications.

How did I reach the goal? By trial and error. A lot of trial and error. And a lot of reading. Really really a lot of reading.

But I can make your journey a lot easier and this series of posts is exactly this: the final summary of my journey trying to dockerize our Symfony applications here at Serendipity HQ.

Unless you have lived on Mars in the last months, you have heard of Docker for sure.

And if you are a curious person, maybe the idea of trying it came to your mind.

But starting to use Docker is not an easy task, at least it wasn’t easy for me.

I read a lot of articles about how to set up a Symfony app to use Docker and all that I found was a bunch of short articles with some code to copy and paste without any explanation about what each line did and why it was there.

So I switched to analyse other Symfony projects that already use Docker (you will find a list of them at the end of this post), but the problem was the opposite: very complex configurations, impossible to understand nor to modify without an understanding of what each line did.

And also searching for each instruction didn’t help as what I lacked was the general picture.

So I decided to start from scratch, solving a problem after problem each time one arose.

And this series of posts is the result of this journey: configuring a Symfony app to use Docker.

Before starting to dockerize your Symfony application, it is better to explain what precisely Docker is.

Docker is:

… An open platform for developers and sysadmins to build, ship, and run distributed applications, whether on laptops, data center VMs, or the cloud.

Its main concept is the “container”:

Docker containers wrap up software and its dependencies into a standardized unit for software development that includes everything it needs to run: code, runtime, system tools and libraries. This guarantees that your application will always run the same and makes collaboration as simple as sharing a container image.
Docker for Developers > Get started

The main advantages of using Docker are:

Onboard faster and stop wasting hours trying to set up development environments, spin up new instances and make copies of production code to run locally.

Enable polyglot development and use any language, stack or tools without worry of application conflicts.

Eliminate environment inconsistencies and the “works on my machine” problem by packaging the application, configs and dependencies into an isolated container.

Alleviate concern over application security Docker for Developers > Get started

So, now that we have a clear picture of what is Docker and which problems it solves, let’s start to go a bit deeper in the topic.

The first thing I needed to learn was Docker Compose, the builder of our entire stack.

Our stack uses a docker-compose.yaml file put in the root of the application.

But now I’m going too fast: Let’s try to do one step at time.

Doing this, at the end of this series, you will have:

A clear picture of how each building block fits into the bigger picture;
Which are those building blocks;
The tools you need to debug Docker and better understand what is going on;
A basic configuration from which to start building your next dockerized application;
The tools required to expand this basic configuration.

For this series of posts, we are going to build a Dockerized application based on Symfony 4.2.

But keep in mind that the Symfony version is really irrelevant: once you have understood the basics you will be able to dockerize any version of Symfony, present, past and future.

So, please, before starting, create a new Symfony project to dockerize: having a test application makes things easier as it permits you to focus on the basics, without having to deal with your custom code or custom configurations, external services, etc.

As already told, once you will have the basics, you can dockerize any Symfony project!

Then, install Docker searching on Google “How to install Docker“: Google is really better than me at identifying your OS and giving you the right steps to install Docker, so, follow him!?

We are ready!

Just one note before really starting: in this series of post I will make you follow the exact trial-and-arr approach I followed the first time.

This way you will really understand which are the problems you will face when dockerizing any app and you will deeply understand the logic behind a dockerized infrastructure.

When someone starts using Docker not all concepts are obvious and the approach someone may use or the logic (s)he will follow or the assumptions (s)he will have may be wrong.

For this reason, I prefer to make you able to make errors: so, in this series of posts we will make errors (very bad?) but we will also fix them learning really how Docker works and ” thinks”!?

Introducing `docker-compose`

Ok, now that you have a fresh Symfony app to dockerize and have installed Docker on your computer, let’s start understanding the first building block of our Docker infrastructure: Docker Compose.

Compose is a tool for defining and running multi-container Docker applications.

With Compose, you use a YAML file to configure your application’s services.

Then, with a single command, you create and start all the services from your configuration.

To learn more about all the features of Compose, see the list of features.
Overview of Docker Compose

Ok, I know, this can be a bit obscure at the beginning: after all, you may have no idea of what a container is concretely, what it does and of which parts is composed of.

Don’t worry: keep reading and you will understand everything!

For the moment, you only need to understand and remember that Docker Compose uses a docker-compose.yaml file put in the root folder of the project to know what services it has to create to make your app able to work.

The Compose file is a YAML file defining services, networks, and volumes for a Docker application.

…

There are several versions of the Compose file format – 1, 2, 2.x, and 3.x
Compose file versions and upgrading

Mmm, services, networks, volumes, file versions: these are exactly the kind of things that make difficult to understand how to use Docker at the beginning. ?

Again, don’t worry and keep reading: I will explain to you everything in details. More often than not, it is easier to do things first and understand them later, instead of trying to understand them first and then doing them later.

So, how you write your Compose file, depends on the version of Docker you are running.

To know which version of Docker you are running, use the docker version command:

Aerendir$ docker version
Client:
 Version: 18.06.0-ce
 API version: 1.38
 Go version: go1.10.3
 Git commit: 0ffa825
 Built: Wed Jul 18 19:05:26 2018
 OS/Arch: darwin/amd64
 Experimental: false

Server:
 Engine:
  Version: 18.06.0-ce
  API version: 1.38 (minimum version 1.12)
  Go version: go1.10.3
  Git commit: 0ffa825
  Built: Wed Jul 18 19:13:46 2018
  OS/Arch: linux/amd64
  Experimental: false

As you can see, I’m now running Docker version 18.06.0-ce.

This means that I can use the Compose file format version 3.7.

How do I know this?

Because Docker provides a matrix where, for each Docker version, there is the corresponding Compose version supported.

Very easy, isn’t it? ?

Creating our `docker-compose.yaml` file

For a Symfony application we basically need three things:

A web server (Apache or Nginx);
PHP;
A database (MySQL, PostgreSQL, MongoDB, Redis, etc.).

Last but not least, we also need an operating system: MacOS, Windows, *nix, etc.

The combination of those technologies is our “stack”.

The stack we will build will have these characteristics:

It has to be easily configurable;
It has to have the very basic requirements needed to run Symfony;
It has a high probability, so we can use it in other projects without Docker to start dockerize them.

So, our stack will be this:

Operating system: Ubuntu;
PHP Version: 7.2 (this is the best choice, simply);
Database: MySQL;
Web server: Apache.

I’m going to use Apache because if you develop PHP applications, it is 99% probable that you are familiar with it.

Nginx is less common and requires another series of posts to be understood?.

Let’s maintain things simple: use Apache!

The same for MySQL: for more complex applications you may need MongoDB, but for a basic configuration and, mostly, to understand the basics of Docker, it is better to use something simple and familiar as MySQL instead of MongoDB of which you may have never heard of (do you live on Mars??).

To get this stack up and running we have to do two things:

Create a docker-compose.yaml file in the root folder of our project;
Define the components of our stack.

So, let’s start with PHP creating a docker-compose.yaml file in the root folder of the project:

# docker-compose.yaml

version: '3.7'
services:
  language:
    image: php:7.2

Now we can run docker-compose up -d and we will have a service with PHP up and running:

Creating network "app-test-www_default" with the default driver
Creating app-test-www_language_1 ... done
Attaching to app-test-www_language_1
language_1 | Interactive shell
language_1 | 
app-test-www_language_1 exited with code 0

Docker will start downloading a lot of things and the process may take a bit of time: be patient!?

When the exit code is 0, then all gone well.

NOTE: the -d flag means “detached”: this makes the console available again after Docker Compose finishes. If you don’t pass it, you will need to open a new console window to continue to use the command line.

Things to note here

version: is the version of the compose file. As told above, the version you use in your compose file depends on the Docker version you are running. Check the compatibility matrix for more information.
services.language: “language” here is the name of the service we are creating. It is an arbitrary string we use to name the service. Change this to “php” if you like: I used “language” just to show you how does it work and that it can be changed and is not related in any way to the image you are using. You can call the service as you like, also my_astonishing_and_super_power_service (but I warmly suggest you to use really shorter names!?)
service.language.image: defines the image we want to use to build the service. Here we are using the image of PHP 7.2. Later in this series of posts, I will explain to you how to find the images you may need and how to understand how to use them: be patient!?

So, we now have PHP configured: we still need other components to make our Symfony app run on Docker.

We need:

The operating system;
The web server;
A database.

Let’s configure them:

version: '3.7'
services:
  # "php" was "language" in previous example
  php:
    image: php:7.2

  # Configure the database
  mysql:
    image: mysql:5.7

  # Configure Apache
  apache:
    image: httpd:2.4

Now, running again docker-compose up -d reports this (note that Docker may first download images, so you will see a lot of logging before the lines below):

Aerendir$ docker-compose up -d
Starting app-test-www_mysql_1 ... done
Starting app-test-www_php_1 ... done
Creating app-test-www_apache_1 ... done

At this point, we have all the services up and running.

To verify this, other than the previous messages with “done”, you can do this:

Aerendir$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
6d9e627c81f5 httpd:2.4 "httpd-foreground" 18 minutes ago Up 18 minutes 80/tcp app-test-www_apache_1

As you can see, currently I have only one container running and it is the Apache one.

If you are asking yourself, yes, I run this command 18 minutes later because in the meantime I gone out with Loki, my pet.

(Ok, he’s not a cat, but he is anyway a really pretty dog!?)

Another thing you should note: there is only one service, the Apache one: where are the others? There seems that MySQL and PHP aren’t running ?

But anyway there is a container running: Let’s go exploring it without taking care of the others (for the moment!?).

Exploring the Apache container

To explore the container running Apache, use the command docker exec -it [container_id] bash:

Aerendir$ docker exec -it 6d9e627c81f5 bash
root@6d9e627c81f5:/usr/local/apache2# ls
bin build cgi-bin conf error htdocs icons include logs modules

NOTE: the -i flag stands for “–interactive” and makes possible to interact with the console that we called with the -t flag, that allocates a pseudo-tty console.

So, instead of using docker exec -i -t ..., we condensed the command into docker exec -it ... . More info here and here.

As you can see, we have some folders: from here you can go around with the commands cd to change the directory (ex.: cd bin) and with the command ls to see what’s inside the current folder (ex.: ls).

root@6d9e627c81f5:/usr/local/apache2# cd bin 
root@6d9e627c81f5:/usr/local/apache2/bin# ls
ab apachectl apxs checkgid dbmmanage envvars envvars-std fcgistarter htcacheclean htdbm htdigest htpasswd httpd httxt2dbm logresolve rotatelogs suexe

To come back to your user, out of Docker, simply type exit:

root@6d9e627c81f5:/usr/local/apache2/bin# exit
exit
MacBook-Pro-di-Aerendir:app-test-www Aerendir$

Now that we know how to enter a container, it’s time to understand where are our MySQL and PHP: are they lost?

Debugging MySQL and PHP in Docker

We defined three services in our docker-compose.yaml file: Apache, MySQL and PHP.

But our container seems have lost MySQL and PHP: where are they gone?

If you go to some paragraphs above, you notice we used the command docker-compose up -d: the -dflag means “detached”, so we have again the console available.

Try to not use the -d flag to run the command (first, run docker-compose down!):

Aerendir$ docker-compose down
Stopping app-test-www_apache_1 ... done
Removing app-test-www_mysql_1 ... done
Removing app-test-www_php_1 ... done
Removing app-test-www_apache_1 ... done
Removing network app-test-www_default

And now we can up again to see what happens:

Aerendir$ docker-compose up
Creating network "app-test-www_default" with the default driver
Creating app-test-www_mysql_1 ... done
Creating app-test-www_apache_1 ... done
Creating app-test-www_php_1 ... done
Attaching to app-test-www_mysql_1, app-test-www_apache_1, app-test-www_php_1
mysql_1 | error: database is uninitialized and password option is not specified 
mysql_1 | You need to specify one of MYSQL_ROOT_PASSWORD, MYSQL_ALLOW_EMPTY_PASSWORD and MYSQL_RANDOM_ROOT_PASSWORD
php_1 | Interactive shell
php_1 | 
apache_1 | AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.96.3. Set the 'ServerName' directive globally to suppress this message
apache_1 | AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.96.3. Set the 'ServerName' directive globally to suppress this message
apache_1 | [Sun Aug 12 15:14:59.981146 2018] [mpm_event:notice] [pid 1:tid 140196935767936] AH00489: Apache/2.4.33 (Unix) configured -- resuming normal operations
apache_1 | [Sun Aug 12 15:14:59.981380 2018] [core:notice] [pid 1:tid 140196935767936] AH00094: Command line: 'httpd -D FOREGROUND'
app-test-www_mysql_1 exited with code 1
app-test-www_php_1 exited with code 0

As you can see, using the detached mode all seems going well; but removing the -d flag, some errors seem to arise:

mysql_1 | error: database is uninitialized and password option is not specified

apache_1 | AH00558: httpd: Could not reliably determine the server’s fully qualified domain name, using 192.168.96.3. Set the ‘ServerName’ directive globally to suppress this message

So we have two errors: one from Apache and one from MySQL: Let’s fix them!

Further configuring MySQL

MySQL returned this error:

mysql1 | error: database is uninitialized and password option is not specified

mysql_1 | You need to specify one of MYSQL_ROOT_PASSWORD, MYSQL_ALLOW_EMPTY_PASSWORD and MYSQL_RANDOM_ROOT_PASSWORD

So, it is obvious what we have to do to fix it: set the required environment variables!

So, add the node services.mysql.environment and set the variable MYSQL_ROOT_PASSWORD:

version: '3.7'
services:
    # "php" was "language" in previous example
    ...

    # Configure the database
    mysql:
        image: mysql:5.7
        # Add the MYSQL_ROOT_PASSWORD to the environment variables
        environment:
          - MYSQL_ROOT_PASSWORD=${MYSQL_ROOT_PASSWORD:-root}

    # Configure Apache
    ...

NOTE: We are using a default value of root if the env variable MYSQL_ROOT_PASSWORD is not set; if it is set, instead, we use its value. This syntax is of Bash and is the one to set default values.

It is useful as it permits us to use an .env file locally (read by the Symdony’s Dotenv Component) while using real environment variables on production.

Now, on your keyboard, press CTRL + C to kill the current Docker container, then run docker-compose up (without -d):

Aerendir$ docker-compose up
Recreating app-test-www_mysql_1 ... done
Starting app-test-www_php_1 ... done
Starting app-test-www_apache_1 ... done
Attaching to app-test-www_php_1, app-test-www_apache_1, app-test-www_mysql_1
php_1 | Interactive shell
php_1 | 
apache_1 | AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.96.3. Set the 'ServerName' directive globally to suppress this message
apache_1 | AH00558: httpd: Could not reliably determine the server's fully qualified domain name, using 192.168.96.3. Set the 'ServerName' directive globally to suppress this message
apache_1 | [Sun Aug 12 15:21:41.877927 2018] [mpm_event:notice] [pid 1:tid 139714071852928] AH00489: Apache/2.4.33 (Unix) configured -- resuming normal operations
apache_1 | [Sun Aug 12 15:21:41.878086 2018] [core:notice] [pid 1:tid 139714071852928] AH00094: Command line: 'httpd -D FOREGROUND'
mysql_1 | Initializing database
...

The console will continue to give you log messages that tell you what is happening for MySQL service.

It seems that the MySQL errors are gone: let’s verify the container started.

You are not in detached mode, so, open a new console window and use the docker ps command to see the currently active containers:

Aerendir$: docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
baa0652ed2a6 mysql:5.7 "docker-entrypoint.s…" 4 minutes ago Up 4 minutes 3306/tcp app-test-www_mysql_1
f1da511ff5b5 httpd:2.4 "httpd-foreground" 4 minutes ago Up 4 minutes 80/tcp app-test-www_apache_1

As you can see, we now have two services running in our container: we have fixed MySQL!?

The last one remaining is PHP.

Further configuring PHP

Where is PHP? Our container only shows Apache and MySQL and there is no trace of PHP.

Let’s start from the beginning…

On any machine you run php, to see if it is running, you can simply use the command php -v: this will return the version of PHP currently used (the below command is run on my machine, not on any Docker container!):

Aerendir$ php -v
PHP 7.2.2 (cli) (built: Feb 1 2018 13:23:34) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.2.2, Copyright (c) 1999-2018, by Zend Technologies
    with Xdebug v2.6.0, Copyright (c) 2002-2018, by Derick Rethans
    with blackfire v1.21.0~mac-x64-non_zts72, https://blackfire.io, by Blackfire

As you can see, on my machine I’m using version 7.2 of PHP and I also have OPcache, Xdebug and Blackfire.

But how to run this command in a Docker container?

Try this:

Aerendir$ docker run php -v
PHP 7.2.8 (cli) (built: Jul 21 2018 07:47:51) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.2.0, Copyright (c) 1998-2018 Zend Technologies

Olè! PHP is running!

The version is the 7.2.8, different than the version 7.2.2 that is run on my local machine, and there is no trace of OPcache nor of Xdebug or of Blackfire.

So, we have basically nothing to debug: we have all the three services up and running!

Yes, I know what you are asking: then, if PHP is running, why isn’t it shown in the list of running containers?

I will clarify this later in this series of posts, for the moment put apart this question.

SPOILER: We made an error configuring the PHP service!?…?

Now it is time to access the web server from the browser!

Accessing the web server from the browser

Before accessing our Symfony app from the browser, we have to first figure out how to simply access the web server from the browser.

Let’s run again the ps command:

Aerendir$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
baa0652ed2a6 mysql:5.7 "docker-entrypoint.s…" 19 minutes ago Up 19 minutes 3306/tcp app-test-www_mysql_1
f1da511ff5b5 httpd:2.4 "httpd-foreground" 19 minutes ago Up 19 minutes 80/tcp app-test-www_apache_1

As you can see, the httpd:2.4 image is exposing the port 80: this is the default port used by the Apache web server for TCP connections.

But this port is exposed on Docker, not on our computer.

In fact, if you try to access http://localhost:80 you will continue to see your local web server.

How can we access the container’s web server instead?

We need to bind the port 80 from the Docker container to one of the ports of our computer.

The Docker’s documentation about Container networking states this:

By default, when you create a container, it does not publish any of its ports to the outside world.

Then it suggests using the flag --publish, but this is not useful in our case: we need to understand how to bind Docker ports to our machine ports (our machine is the Docker’s host), but using the Compose file.

This means a simple thing: you cannot access the services from your browser as Docker doesn’t make them “public”: they exist, but can be accessed only by other Docker’s services.

If you want to access the services from the world outside Docker, then you need to “bind” them.

The “binding” is a way of saying to Docker that all the traffic to the port of your computer has to be forwarded to the port it internally assigns to the service.

So, anticipating what we will do, we will bind the port 80 of the Docker service to the port 8100 of our computer. This will tell Docker that all the traffic to the port 8100 on our computer has to be forwarded to the port 80 he has mapped internally.

Docker offers two ways to create the “binding”.

So, let’s read the documentation of Compose file: there we can find two useful keys: expose and ports:

Expose

Expose ports without publishing them to the host machine – they’ll only be accessible to linked services. Only the internal port can be specified.

…

Ports

Expose ports.

Not so useful as descriptions, aren’t they? Honestly, they are also a bit confusing ?

Let’s try to clarify their meaning.

Basically, both expose and ports makes possible for a Docker container to make some ports reachable.

The main differences between the two are these:

expose makes those ports reachable only by services/containers in the same Docker network, but not from outside of it;
ports, instead, makes the ports reachable from outside the network.

So, guess what, our best option is using ports.

The Docker’s documentation about ports clearly states a caveat:

Note: When mapping ports in the HOST:CONTAINER format, you may experience erroneous results when using a container port lower than 60, because YAML parses numbers in the format xx:yy as a base-60 value. For this reason, we recommend always explicitly specifying your port mappings as strings.

This is relevant because you can do something like this:

version: '3.7'
services:
  apache:
    ...
    ports:
      - 80

or you can do something like this:

version: '3.7'
services:
  apache:
    ...
    ports:
      - '8100:80'

In the first case, we can simply use an integer, but in the second case, as we are using the format HOST:CONTAINER, we need to set the port mapping as strings (using “‘” – an “apex”) and not as integers to avoid the before mentioned problems in parsing.

More, in the first case, the port 80 will be bound to a random port of the host machine (again, the host machine is our machine that is running Docker): this is not useful, as with a random port we cannot anyway access the Apache server running in the container! (each time we should first check which is the current port assigned – using docker ps – and this is not useful).

So, our port binding of Apache ports is something like this:

version: '3.7'
services:
  # "php" was "language" in previous example
  php:
    ...

  # Configure the database
  mysql:
    ...

  # Configure Apache
  apache:
    image: httpd:2.4
    ports:
      - "8100:80"

Run the command docker-compose up -d to apply the new configuration.

Now, accessing the URL http://127.0.0.1:8100 we will see the default Apache index.html page with the famous “It Works!” message:

But where is this file located?

Good, a bit of context.

Apache stores the files to be served in the so-called “document root“.

So we need to find this document root in our container.

As already shown above, run docker ps to get the id of the container we want to enter:

Aerendir$ docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
718b8afc0626 mysql:5.7 "docker-entrypoint.s…" 34 minutes ago Up 5 seconds 3306/tcp app-test-www_mysql_1
57aa966336a2 httpd:2.4 "httpd-foreground" 34 minutes ago Up 5 seconds 8100->80/tcp app-test-www_apache_1

Then, we enter the container running this:

Aerendir$ docker exec -it 57aa966336a2 bash

Now we can start exploring the contents of the container with ls:

root@57aa966336a2:/usr/local/apache2# ls
bin build cgi-bin conf error htdocs icons include logs modules

Hey, htdocs is one of the default names of the Apache’s document root folders!

root@57aa966336a2:/usr/local/apache2# cd htdocs
root@57aa966336a2:/usr/local/apache2/htdocs# ls
index.html

Found! The file index.html is the one that showed us the message “It works!”: this is our document root!

We spotted it!?

Conclusions and next steps

For this post I think we have covered a lot of things:

We learned:

How to configure services with Docker;
How to access them;
Some basic debugging techniques;
How to bind the ports of a Docker’s services to the ports of our computer;
How to access the web server from our browser.

In doing this we also learned some basic concepts of Docker like what is a container, what is a service, some basic information about the networking capabilities of Docker (the binding of ports is “networking”!).

But now we also have some questions:

Why the PHP service doesn’t appear in the list of services?
And why, also if it doesn’t appear in such list, anyway PHP seems to be running?

And most important: now that we have our web server up and running, how can we move our Symfony files into it to start developing our app using Docker?

Duh, a lot of questions! (and many more will arise!?).

Well, trust me, I will answer all of them in the next post of this series.

For the moment, I will answer the last question: how to move the Symfony’s files into the web server run by Docker.

Remember to “Make. Ideas. Happen.”.

In the meantime, I wish you flocking users!

Next post: Moving Symfony files in the Docker container.

L'articolo How to dockerize a Symfony application: Introduction and the web server proviene da ÐΞV Experiences by Serendipity HQ.

How to set up the workflow to develop a Genestrap-based WordPress theme

Adamo Crespi — Mon, 19 Nov 2018 18:32:28 +0000

The main advantage of having a Genestrap-based WordPress theme is the ability to use the tools and the techniques you already use to develop your web apps.

Using a framework like Genesis to develop a WordPress theme takes the management of WordPress to a whole new level of complexity (and fun, too!).

So, once you have created the first version of your Genestrap-based WordPress theme, let’s see how to prepare your environment to manage the updates you will do developing your custom theme.

Setting up `git`

If you open a console window and write git remote: you will see this:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git remote
origin

Then, you can write this:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git remote show origin
* remote origin
Fetch URL: https://github.com/Aerendir/Genestrap.git
Push URL: https://github.com/Aerendir/Genestrap.git
HEAD branch: master
Remote branch:
master tracked
Local branch configured for 'git pull':
master merges with remote master
Local ref configured for 'git push':
master pushes to master (local out of date)

As you can see, the current repository is set to point to the main Genestrap repository on GitHub.

So, what we need to do, is to point it to one of our custom repositories.

Create a private `git` repository

The first thing you need to do is to create a private git repository in which store your developing Genestrap-based WordPress theme.

I use BitBucket that offers free private repository (with some limitations on the number of users, but this is not a problem for a project like this).

Obviously, you can use what you like: BitBucket, GitHub, GitLab or anything else.

I called my repo web-serendipityhq-wp_theme and below you can see my empty repository.

Now that we have a new empty repository to fill, let’s fill it!

Pushing your Genestrap-based WordPress theme to your private `git` repository

To push your Genestrap-based WordPress theme to your just created git repository, you need to change the origin remote from the current pointing to Genestrap’s GitHub repository to a new one pointing to your just created repo.

Doing this is really simple with a command like git remote remove and git remote add (run git remote --help for the full list of options).

So, start by removing the current remote:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git remote rm origin

NOTE: rm is the short version of remove and you can use both as you like.

Now, add a new remote pointing to your just created repo: we will call it origin, like the one we have just removed:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git remote add origin https://Aerendir@bitbucket.org/Aerendir/web-serendipityhq-wp_theme.git

Let’s see what happened:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git remote
origin
MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git remote show origin
Password for 'https://Aerendir@bitbucket.org': 
* remote origin
  Fetch URL: https://Aerendir@bitbucket.org/Aerendir/web-serendipityhq-wp_theme.git
  Push URL: https://Aerendir@bitbucket.org/Aerendir/web-serendipityhq-wp_theme.git
  HEAD branch: (unknown)

NOTE: Calling those commands you may be asked for your password: obviously, provide it in the console to access the remote repo. You should set your credentials via SSH to avoid providing the password any time.

We have a new origin remote that now points to our repository.

The thing to note is that now we have an unknown HEAD branch: this is because our repository is still empty: we need to push our Genestrap-based theme to it!

First push of the Genestrap-based WordPress theme to the `git` repository

You need a single, simple command:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git push -u origin master
Password for 'https://Aerendir@bitbucket.org': 
Counting objects: 200, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (79/79), done.
Writing objects: 100% (200/200), 100.19 KiB | 0 bytes/s, done.
Total 200 (delta 113), reused 200 (delta 113)
remote: Resolving deltas: 100% (113/113), done.
To https://Aerendir@bitbucket.org/Aerendir/web-serendipityhq-wp_theme.git
 * [new branch] master -> master
Branch master set up to track remote branch master from origin.

With this command we have told git to:

Push our code on the remote repository;
Set the upstream (the local repository now tracks the branch master);
The previous two points are executed on the remote called origin;
Sets the name of the branch to master.

If you refresh the page of your repository in the browser, you now see all the files of your Genestrap-based WordPress theme:

Verify again the remote:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git remote show origin
Password for 'https://Aerendir@bitbucket.org': 
* remote origin
  Fetch URL: https://Aerendir@bitbucket.org/Aerendir/web-serendipityhq-wp_theme.git
  Push URL: https://Aerendir@bitbucket.org/Aerendir/web-serendipityhq-wp_theme.git
  HEAD branch: master
  Remote branch:
    master tracked
  Local branch configured for 'git pull':
    master merges with remote master
  Local ref configured for 'git push':
    master pushes to master (up to date)

Perfect: now our remote points to our private git repository and correctly tracks our master branch!

Updating your custom Genestrap-based WordPress theme

Now that we have git set up, we can also start to customize the theme.

Each time you make a modification, commit and push and your remote repository will save the editings.

But those editings are not reflected on your WordPress site.

So, here we need to clarify a thing: what does it mean “Updating a Genestrap-based theme”?

Well, “updating” means three different things with Genestrap:

Updating the private remote repository (with a push);
Updating the built Genestrap-based theme on your WordPress site;
Updating the source code of your Genestrap-based theme FROM the main Genestrap public repository.

Each one of those “update” requires different actions, so let’s see them in details one by one.

Updating the private remote repository of your Genestrap-based WordPress theme

This is really easy as we already have set up our local repository to push to our remote one.

So, each time you made an editing, simply commit and then push: done.

Nothing complex here.

Updating the built Genestrap-based theme on your WordPress site

When you want to update the theme on your Genestrap-based theme WordPress site, you need to build it again.

So, run gulp build and the theme will be built again.

Then upload the files in the folder src/build/theme to the folder of your theme on your hosting and you are done.

But there are some caveats here.

If you inspect the code of any of the pages of your WordPress site, in fact, you will notice that css and js are imported using a query string parameter, something like this:

<link rel="stylesheet" id="genesis-with-bootstrap-css" href="https://io.serendipityhq.com/wp-content/themes/theme/style.css?ver=0.1" type="text/css" media="all">

This means that if you edit a css or a js file it will not be rendered on the page, also if you upload it to your hosting via FTP: this is because the query string parameter is used by the browser to cache the file and load the page faster on next load.

And this means that you will not see the changes until the browser invalidates the cache (or you empty it).

So, when editing front-end files, you need to also modify the version of your theme to force the browser to download (and cache again) the new, updated version.

To change the version number, you need to edit the files style.css and config.php:

The version in style.css will be used in the backend to show the current version of the theme;
The version in config.php will be used to create the query string parameter for caching purposes.

So, whenever you edit a css or an scss file, you have to update both version numbers to see your changes on your live WordPress site.

Updating the source code of your Genestrap-based theme FROM the main Genestrap public repository

From time to time we make some changes to the code of Genestrap to adapt to new versions of Genesis or Bootstrap or to add some functionalities.

In general, we try to keep Genestrap as thin as possible: this way we can maintain the freedom of both Bootstrap and Genesis.

However, sometimes, as told, some changes are required.

Those changes can be made to two kinds of files:

*.dist.* files;
Internal files.

In the first case, you have no other options than updating manually your code, as this kind of file is meant to be customized by any one of us using Genestrap.

When modifications happen to internal files, instead, it is useful to know how to update your theme to the new version.

The flow is really simple and involves the use of git.

In the first part of this post, we removed the remote origin that pointed to the public repository of Genestrap on GitHub.

But that remote is required to update our local files (and then, also the ones in our remote repository).

So, we need to add again the remote public repository of Genestrap, but this time, we’ll call the remote exactly genestrap (just to be original!):

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git remote add genestrap https://github.com/Aerendir/Genestrap.git

So, we now have two remotes: origin and genestrap.

We can check them, too:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git remote
genestrap
origin
MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git remote show genestrap
* remote genestrap
Fetch URL: https://github.com/Aerendir/Genestrap.git
Push URL: https://github.com/Aerendir/Genestrap.git
HEAD branch: master
Remote branch:
master new (next fetch will store in remotes/genestrap)
Local ref configured for 'git push':
master pushes to master (local out of date)

Now, when you want to save your modifications to your private repository, you use git push origin master.

To update your theme to the last version of Genestrap, instead, you need to follow these steps:

checkout the last version of Genestrap from GitHub;
merge the checked out branch in your master branch;
Edit the version of your theme if required, commit and push;
push the updated master branch to your remote repository;
gulp build the theme again (maybe, changing the version number if required) and upload it to your WordPress live site.

1. `checkout` the last version of Genestrap from GitHub

This means running a command like this:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git checkout -b genestrap-master genestrap/master
Branch genestrap-master set up to track remote branch master from genestrap.
Switched to a new branch 'genestrap-master'

Now your active branch is genestrap-master that is the branch master on the GitHub public repository of Genestrap: at this moment you have the last version of Genestrap.

Now, you need to update the master branch of your local repository.

2. `merge` the checked out branch in your master branch

To merge the branch genestrap-master to master, you need to:

Make the master branch the active one;
Merge genestrap-master in master.

In console commands, the procedure is this:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git checkout master
Switched to branch 'master'
Your branch is up-to-date with 'origin/master'.
MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git merge genestrap-master
Updating d493923..c7534fe
Fast-forward
gulpfile.js | 76 +++++++++++++++++++++++++++++++++++++++++++---------------------------------
1 file changed, 43 insertions(+), 33 deletions(-)

If you run the above command again, git will tell you that there is nothing to update:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ git merge genestrap-master
Already up-to-date.

3. Edit the version of your theme

As told before, you have to update the version number of your theme, so the cached files are ignored and the new ones are used by browsers.

You need to commit also these changes.

4. and 5. `push` and update

You already know how to do this: you need to push to your remote private git repository the update master branch (that now contains the modifications from branch genestrap-master), then rebuild the theme with gulp build.

Then you can update the theme on your WordPress live site.

Conclusions and next steps

You now have a clear understanding of how to use Genestrap to build a custom theme based on Genesis and Bootstrap for your WordPress site.

Remember to “Make. Ideas. Happen.”.

In the meantime, I wish you flocking users!

L'articolo How to set up the workflow to develop a Genestrap-based WordPress theme proviene da ÐΞV Experiences by Serendipity HQ.

How to create your custom WordPress theme with Genestrap

Adamo Crespi — Mon, 19 Nov 2018 17:52:37 +0000

Building a WordPress theme from scratch can be a really heavy task.

But buying a stock theme may not always be an option.

The result is that you need to develop your theme from scratch.

And if you would like to use Bootstrap, the task may become very complex very fast.

Genestrap is the perfect solution: easy, fast and flexible.

And you can develop seriously, without being a mere WordPress “point-and-clicker”!

For most of the websites based on WordPress that I manage, I go to one of the stock theme websites, buy a nice theme and install and configure it.

This is a good way of making a website up and running in a very fast way.

However, when I had the necessity to build the marketing sites of TrustBack.Me (marketing, support, ecc.) I decided to use WordPress instead of reinventing the wheel and develop all from scratch with Symfony.

But buying a stock theme was not an option: making one of them match perfectly the look and feel of TrustBack.me’s app site would be a nightmare, if not impossible at all.

On TrustBack.Me I used Bootstrap on top of Twig and Symfony, but buying a Bootstrap based stock template would not have given me the flexibility of which Bootstrap is capable of.

I needed a different solution.

So I came upon Genesis framework home page: hey, this seemed the right tool for the job!

I could have added Bootstrap on top of it, use all the tools that make Bootstrap flexible, I could have continued to feel a developer, and, in the end, have a WordPress site perfectly matched with the app site of TrustBack.Me!

Something about Genesis

The Genesis Framework was the foundation on top of which was built the main theme of CopyBlogger, one of the best places on the Internet to learn about blogging and digital marketing.

Now it is developed by StudioPress, that is in turn managed by WP Engine.

StudioPress describes the Genesis Framework this way:

The Genesis Framework empowers you to quickly and easily build incredible websites with WordPress. Whether you’re a novice or advanced developer, Genesis provides the secure and search-engine-optimized foundation that takes WordPress to places you never thought it could go.

We chose the Genesis Framework as it makes us able to develop our custom WordPress theme, using the tools all the developers like us love (Git, Webpack, Gulp, Yarn or NPM, etc.), maintaining our WordPress very very fast (try to test Serendipity HQ!) and allowing us to continue to use our development workflows.

To get a full list of the features of Genesis, you can read this.

Something about Bootstrap

Bootstrap is a CSS framework originally open sourced by Twitter.

It is the foundation of a lot of websites on the Internet.

Bootstrap offers out of the box a lot of useful tools to develop the CSS of a website, offering the most used CSS elements.

The framework is highly customizable and permits to change practically everything on each installation (also if a lot of people simply download the framework as is, without any customization, making their website practically the same of many others).

We use Bootstrap on TrustBack.me as it permits us to speed up the development of the app and permits us to reach a high level of customization, too.

Why we built Genestrap

Genestrap, as the name implies, is the fusion of Genesis Framework for WordPress with the Bootstrap CSS Framework.

We use Genestrap on a lot of our projects, like TrustBack.Me (for the marketing site), here at Serendipity HQ (works also with multisite that we use for Serendipity HQ), on eCommerceRS.NET.

It is our foundation for any WordPress site we build internally or for clients when the budget allows us to use it.

We wanted to use Bootstrap on top of Genesis and as the bootstrapping work is always the same, we decided to ship it as a package to use as the starting point.

You can find the main repository of Genestrap on GitHub.

How to start developing your new WordPress theme based on Genstrap (Bootstrap + Genestrap)

We will use PHPStorm, but if you feel more comfortable, feel free to use the command line alone: we prefer PHPStorm because it will give us a lot of useful features that a command line alone (nor a simple, yet advanced and fully configurable editor like Atom) simply cannot offer.

STEP 1: Clone locally the Genestrap repository

From PHPStorm:

Click on “Check out from Version Control” and select “Git” (or VCS > Check out from Version Control > Git): the “Clone Repository” window opens;
Now set the URL of the Genestrap repository: https://github.com/Aerendir/Genestrap.git;
Set the directory in which to clone the repo (we created a folder named [ProjectName] > [GenestrapTheme]: SerendipityHQ.com > SHQ-GenestrapTheme;
Click on the “Clone” blue button and PHPStorm will start to clone the repository.

When the repository is cloned, PHPStorm will ask you to open the new project: click on “Yes” blue button.

STEP 2: Create the `src/style.css` file

Copy the file src/style.dist.css in a new file called src/style.css.

This is the main entry point of the theme: the Theme Stylesheet.

It will be used by WordPress to get some useful information about your theme and to show a miniature in the list of themes available for your site once you have uploaded your new Genestrap-based theme.

It is a really simple file:

/*
   Theme Name: Genestrap
   Theme URI: http://www.serendipityhq.com/
   Description: This is Genestrap, Genesis with Bootstrap
   Author: Aerendir
   Author URI: http://aerendir.me

   Version: 0.1

   Tags: Genesis, Bootstrap

   Template: genesis
   Template Version: 2.3.0

   License: MIT

   Text Domain: shq-genesis-bootstrap
*/

Customize it with your details and the details of your project.

This is how I customized our Genestrap theme for eCommerceRS.NET:

/*
    Theme Name: eCommerceRS
    Theme URI: https://ecommercers.net/
    Description: This is the Genestrap based theme of eCommerceRS.NET.
    Author: Aerendir
    Author URI: http://aerendir.me

    Version: 0.1

    Tags: Genesis, Bootstrap

    Template: genesis
    Template Version: 2.3.0

    License: MIT

    Text Domain: shq-genestrap
*/

STEP 3: Create the `config.php`

Copy the file src/config.dist.php in a new file called src/config.php.

This file will set some base constants used by WordPress when showing you the theme in the list of themes.

It is a really simple file:

<?php

/*
 * All the configuration required to configure the theme
 */

return [
   'CHILD_THEME_NAME' => 'Genestrap',
   'CHILD_THEME_URL' => 'https://www.serendipityhq.com',
   'CHILD_THEME_VERSION' => '0.1'
];

This is how I customized the file for eCommerceRS.NET:

<?php

/*
 * All the configuration required to configure the theme
 */

return [
    'CHILD_THEME_NAME' => 'eCommerceRS',
    'CHILD_THEME_URL' => 'https://ecommercers.net',
    'CHILD_THEME_VERSION' => '0.1'
];

STEP 4: Create the `functions.php`

Copy the file src/functions.dist.php in a new file called src/functions.php.

This file sets some defaults of Genestrap.

STEP 5: Create the `src/scss/style.scss` file

Copy the file src/scss/style.dist.scss in a new file called src/scss/style.scss.

This is the file that really defines the customizations you will make to make the Genestrap theme really your Genestrap customized theme.

It basically includes Bootstrap and defines some other custom styles used by Genestrap.

More about this in a bit.

STEP 6: Create the `src/scss/bootstrap.scss` file

Copy the file src/scss/bootstrap.dist.scss in a new file called src/scss/bootstrap.scss.

This file defines the Bootstrap features that will be imported in your theme: this way if you don’t need some features of Bootstrap, you can simply exclude them and your resulting css will be lighter.

More about this in a bit.

STEP 7: Install Composer packages

Simply run composer install to install all the PHP dependencies:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ composer install
Loading composer repositories with package information
Installing dependencies (including require-dev) from lock file
Package operations: 5 installs, 0 updates, 0 removals
- Installing composer/installers (v1.5.0): Downloading (100%)
- Installing squizlabs/php_codesniffer (3.2.3): Downloading (100%)
- Installing wp-coding-standards/wpcs (0.14.1): Downloading (100%)
- Installing stevegrunwell/wp-enforcer (v0.5.0): Downloading (100%)
- Installing wp-bootstrap/wp-bootstrap-navwalker (v4.0.2): Downloading (100%)
wp-coding-standards/wpcs suggests installing dealerdirect/phpcodesniffer-composer-installer (^0.4.3)
Generating autoload files

This step is required as, among the others, the dependency iswp-bootstrap/wp-bootstrap-navwalker installed: it is required to make the menus work well and is copied in the theme’s package when you build it (we will see in a bit how to do this).

STEP 8: Install Javascript dependencies

You can use both NPM or Yarn to install the Javascript dependencies: we prefer Yarn, so our example will use it.

Simply run the command yarn install to install the Javascript dependencies:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ yarn install
yarn install v1.7.0
info No lockfile found.
(node:22378) [DEP0005] DeprecationWarning: Buffer() is deprecated due to security and usability issues. Please use the Buffer.alloc(), Buffer.allocUnsafe(), or Buffer.from() methods instead.
[1/4] ? Resolving packages...
warning gulp-sass > gulp-util@3.0.8: gulp-util is deprecated - replace it, following the guidelines at https://medium.com/gulpjs/gulp-util-ca3b1f9f9ac5
warning popper > rijs.sync > xrs > uws@9.148.0: stop using this version
[2/4] ? Fetching packages...
[3/4] ? Linking dependencies...
warning " > bootstrap@4.0.0" has unmet peer dependency "popper.js@^1.12.9".
[4/4] ? Building fresh packages...
success Saved lockfile.
warning Your current version of Yarn is out of date. The latest version is "1.9.4", while you're on "1.7.0".
info To upgrade, run the following command:
$ brew upgrade yarn
 Done in 44.88s.

Yes, I need to update Yarn on my machine. ?

Anyway, you now have all the required Javascript dependencies installed: they are required to build the theme being able to upload it to your WordPress website.

STEP 9: Building the theme

Before starting customizing the theme, let’s finish our round up to understand the complete flow: later I will explain you how to customize it, too.

So, the next step is to build our theme.

The building process will create a folder ready to be compressed and uploaded where you like: directly in a WordPress site or on a server to make the theme downloadable by all WordPress sites that use it.

To build the theme, simply run gulp build:

MacBook-Pro-di-Aerendir:GenestrapTheme Aerendir$ gulp build
[17:59:08] Using gulpfile ~/Documents/JooServer/_Projects/SerendipityHQ/SerendipityHQ.com/GenestrapTheme/gulpfile.js
[17:59:08] Starting 'build'...
[17:59:08] Starting 'clean-build'...
[17:59:08] Finished 'clean-build' after 3.5 ms
[17:59:08] Starting 'build-css'...
[17:59:08] Finished 'build-css' after 18 ms
[17:59:08] Starting 'move-bootstrap-bundle'...
[17:59:08] Finished 'move-bootstrap-bundle' after 3.13 ms
[17:59:08] Starting 'move-php-files'...
[17:59:08] Finished 'move-php-files' after 7.67 ms
[17:59:08] Starting 'move-bootstrap-walker'...
[17:59:08] Finished 'move-bootstrap-walker' after 1.42 ms
[17:59:08] Finished 'build' after 39 ms

As you can see from the stopwatching, it requires a blink of an eye to complete.

When building is done, check the folder build: here you will find the built theme.

STEP 10: Compress, upload and activate your Genestrap based theme

At this point your theme is ready to be compressed: create an archive, then upload it to your WordPress website.

So, go to Appearance > Themes > Add theme > Upload file and upload:

The Genesis Framework;
Then, the just created archive.

Your list of themes will appear similar to this:

In the image, the theme is called “TrustBack.Me” as we developed it to manage its marketing site: in a bit, you will be able to customize its name, trust me!

Now, the only thing left to do is to activate the theme and go to the home page of your WordPress site to check it.

The result will be similar to this (obviously, with different posts!):

Conclusions and next steps

Creating a working theme with Genestrap is really an easy task: nothing more complex of creating a WordPress theme from scratch, but with the more heavy work already done for you.

Now it is time to do the next step: Implement a development workflow, so you can save your customized Genestrap-based WordPress theme and collaborate with others to improve it.

Remember to “Make. Ideas. Happen.”.

I wish you flocking users, see you soon!

L'articolo How to create your custom WordPress theme with Genestrap proviene da ÐΞV Experiences by Serendipity HQ.

How to track unique users with behavioural analytics tools

Adamo Crespi — Sat, 17 Nov 2018 10:42:02 +0000

The identification of users is the second pillar of behavioural tracking, together with the understanding of events.

Identifying users precisely makes you able to track the complete journey of any user to understand why exactly (s)he signed up or goes away, making you able to uncover hidden causes that you can improve or use to improve your KPIs and your bottom line.

But the identification of users is not so simple as it may seem at a first look.

What does it mean “Identify customers”

This is something you simply cannot do with Google Analytics as it explicitly prohibits you to do this kind of profiling as it has deep GDPR implications (of which we will speak later in another post).

This is the core functionality of each behavioural analytics tool: identify customers and their journeys using your app.

This is logically the next step in your journey ( ) to instrumenting tracking on your SAAS application.

Identifying users is a complex thing: many scenarios may happen in the real world, and to some of them maybe you didn’t think too much.

Amplitude sums up the troubles very well:

Tracking unique users is a complex process because users can log in and out of your product, browse anonymously, and use multiple devices.

It also offers a good starting list of possible scenarios:

No User ID is assigned;
The user ID is assigned after anonymous events;
Same User ID on multiple Device IDs, with no anonymous events;
Multiple User IDs on the same Device ID;
Same User ID on multiple Device IDs, with anonymous events.

As told, this is only a starting list and Amplitude explains them in detail in the linked page.

In general, the track happens for users and for devices.

Woopra explains, too, the challenges we will face in understanding who did what on our website or in our native app:

The Woopra system needs to be able to tell which person profile performed an incoming tracked action (or property update.) The problem is that people in Woopra can exist in a number of different levels of being identified. They could be a first-time anonymous visitor to a website, or a long time paying customer.

Sometimes a person will make a few visits to your site anonymously over a year leading up to the time they decide to sign up for your newsletter, giving you their email. Sometimes this can even mean that what was previously considered to be two different people in Woopra, is now known to be a single person–perhaps originally from two devices–requiring a merge of the two profiles.

[…]

Another issue is that if you want to track anonymous behavior, and even attribute it to known people in the future as they identify themselves to you–a key value proposition in the Woopra system–then using a single id value per person become more complex.

Similarly, if you want to track behavior across channels–another key value proposition of Woopra–then it is basically impossible to maintain the database ID for the profile between your website, and, say, your email marketing automation service.

[…]

The Goal

Woopra needs to be able to take whatever information is available about a person performing the action in an incoming track request, and use it to determine with the highest accuracy possible, which other actions this person has performed, and thus to which profile the incoming actions belong.

If a user is coming anonymously to your website, all you have is a cookie, which is conceptually, a device ID pointing to that browser on that machine. It will be the same next time the person visits your site from that machine and that browser, but if they visit from a different browser, or on their phone, for instance, you will have a new cookie. So Woopra needs to be able to use multiple cookies to eventually refer to one person, assuming that one day you find out who that person is and can associate all their devices with them.

Similarly, you may have an incoming “Email Sent” event from your email marketing tool that is not from a browser and has no cookie. This event has an email address–another major identifier. Woopra needs to eventually (when the person signs in with that email address on their browser with that cookie they had in the past) be able to consider the actions performed by cookie 1, cookie 2, and email 1 all to belong to the same profile.

So, the edge combinations are

Same user on different devices on one side;
Different users on the same device on the other side.

A third factor to take into consideration in SAAS products is the domain: in fact, very probably, you will have at least two different domains, one for the application itself, one for the documentation, the marketing pages, etc.

So, the three factors to consider are:

Device identification;
Domain navigated;
User identification.

Wanting to be extremely synthetic, the real meaning of “user identification”, after all, is associating a user session with a single person and only one, or a sequence of sessions, or more sequences of sessions – oh damn, how complex is it? ? -, also cross devices ? and cross domains. ?

Why do you need to identify users one-by-one

As it is so complex to identify users one-by-one, it is helpful to understand the “Why”: why are we lavish all these energies in something so complex?

The answer is simple: knowledge!

The Customer is the foundation of every business on the Earth and knowing them better than themselves is fundamental to grow your business.

Knowing them with such degree of details opens up possibilities that you do not even imagine:

You can understand exactly why each of your customers didn’t upgrade to a premium feature;
You can understand which have in common the customers who upgraded to a premium feature;
You can understand which features are most used by which customers and what they have in common;
You can understand why a customer downgraded (maybe (s)he didn’t found a helpful article in your knowledge base after having navigated it for a lot of time?);
You can send custom messages to them (MixPanel has a built-in tool for this).

Obviously, those are simple examples: the limitations are tied only to the detail degree of your implementation, your imagination and the current necessities of your application.

Using the words of Amplify, identifying users concretely means this:

Products that have some kind of login system can track users even if they switch devices. Though assigning User IDs is optional, we recommend that products with a login system or a UUID (unique user identifier) system assign a User ID.

With a User ID, Amplitude can match events across multiple devices under the same user (same User ID). Furthermore, a User ID does not need to be assigned immediately. A user’s event data will be merged on the backend so that all anonymous events up to the point of User ID assignment will be connected to the assigned User ID (assuming the Device ID is consistent).

Basic concepts of Device Identification

All the tools that track users use the same identical approach.

There are basically two main categories of devices on which you track users:

A browser
A native application.

In Tracking Unique Users, Amplitude explains in details how the device identification happens.

This is what they say about:

Device ID: We will pull the IDFV or generate a random alphanumeric string that ends with the letter ‘R’ for Device ID and is stored locally in the browser’s cookie or mobile device. However, there is a flag that you can toggle to use the Identifier for Advertiser (IDFA for iOS) and the Advertising Identifier (AdID for Android) as the Device ID.

iOS: If the Device ID is set to the IDFA, then it will persist across installs. Note: iOS users have the option of resetting their IDFA on their devices at any time. As of iOS 10, if a user limits ad tracking, then this would send an IDFA of all zeros. Amplitude will instead set Device ID as IDFV or a randomly generated string.

Android: If the Device ID is the AdID, then it will persist across installs. Note: Android users have the option of resetting the AdID on their devices at any time.

Web: The Device ID will be set to a randomly generated UUID by default. It will persist unless a user clears their browser cookies and/or is browsing in private mode.

So, let’s recap.

IDFV: IDentifier For Vendor. This is a unique identifier available only on Apple devices. Used for tracking in-app. This can be changed to a new one by the user if they want to, but only if they actively change it. This remains the same if the user, for example, resets the device.

IDFA: IDentifier For Advertiser. This is a unique identifier available only on Apple devices. Now it is called “Advertiser IDentifier”, but this is practically irrelevant as the behaviour stays the same. Used for tracking in-app. This ID changes if the user resets the phone. It is equal to a series of zeroes if the user opts out from the tracking. If received as all zeroes, the tracking software generates a random string.

AdID: Advertiser IDentifier. This is a unique identifier available only on Android devices. Used for tracking in-app. If the user opts out from the tracking, this ID is a simple random string that changes every time, making the tracking impossible.

So, in conclusion:

the tracking in web browsers is done through a random string set in cookies;
the tracking in native apps is done using the identifiers provided by the vendors of the device (or a random string if the identifier is not available).

IDFV, IDFA, AdID or a random string stored in a cookie are used to uniquely identify any single device from which your site is being visited or your app is being used.

Those identifiers can change over time if the user decides to change them or if (s)he doesn’t want to be tracked anymore by anyone.

This means that if they are anonymous (you haven’t identified them) you cannot track them if they don’t want you to do (technically you continue to track, but you have a lot of unique visitors also if concretely you have only one visitor).

But, once they identify themselves on your system, you can start tracking them again using the User ID you set and associate with their tracking profile the previous anonymous events they fired (More about the User ID – and other ways of identifying users – in a bit).

Coming to names:

Amplitude call this ID “Device ID“;
MixPanel call this ID “Distinct ID” (read also this);
Woopra calls this with various names, depending on the context.

Basic concepts of cross-domain User identification

Identifying unique users across your domains is crucial as this permits you to follow the journey of your users between all your properties, making you able to understand if a user, for example, becomes a customer after (s)he read a piece on your support site.

It is common for a SAAS product to have more than one domain or more than one subdomain.

At TrustBack.Me, for example, we have the main app on www.trustback.me while other subdomains like aiuto.trustback.me or ciao.trustback.me contain the documentation or the marketing site.

The same we do also here at SerendipityHQ.com: io.serendipityhq.com contains posts about our R&D, www.serendipityhq.com contains our marketing site and there are other sub-domains that contain other kinds of information.

Woopra explains very well the importance of cross-domain tracking:

The importance of cross-domain tracking is to not only store your data in a single project, but to also maintain a visitors session when they explore unique domains. The events of previously identified visitors should be monitored and associated with the same customer profile, regardless of the domain they visit. This is the key to providing a comprehensive customer journey.

When a visitor visits an alternate domain, the cookies associated with that visitor are inaccessible, which causes split profiles in your Woopra project. Without the cookie to associate to the visitor, Woopra has no way of knowing that the visitor on the new domain is the same as the visitor from the previous domain. Woopra would require the visitor to again identify themselves by signing up to an email list or logging in to the platform.

This is the problem that cross-domain tracking solves. When configured correctly, Woopra will attach the cookie value to the url of all specified domains, allowing it to associate that cookie with the visitor from your previous domain.

The main concepts are those:

Use only one project to track all domains of interest;
Pass the User ID from one domain to the other using the query string.

Really simple.

Here is the relevant information about cross-domain tracking with each of the three tools:

Understanding when a User is identified

The first thing you should understand when thinking of Users’ identification is to find the exact moment in your app that permits you to be sure the User is a specific person.

The starting point is always your app as, after all, is your app that identifies Users and so is its responsibility to tell the tracking tools when it finds the person behind the data collected.

At a first look, you may think that your identification happens when you have an email.

But this is not always the case.

If you think deeper at the email, you realize that this is a variable ID: practically every application, in fact, permits to change the email address.

This means that you cannot rely on it to track a user across sessions, domains and devices, as, if (s)he changes the email in your app, then you lose the link between subsequent events (s)he performs in your app and all the previous ones, making the reporting difficult in the best cases, impossible in the worst ones.

So, you should not rely on email for identification.

Instead, you should use your unique ID: when you create a new user in your app, you always assign it a new unique ID: this is the identifier you have to use.

If your app doesn’t have a unique ID assigned to the users (for whatever reason), then the things get a bit more complex and sometimes you have to simply deal with this fact and take it into consideration when reporting and analyzing data.

Amplitude offers two important recommendations about setting User IDs:

Do not set User ID if there isn’t one. For example, setting a User ID to the string ‘None’ to multiple users will group all events under that ‘None’ User ID together (e.g. any user with the ‘None’ User ID is assumed to be the same single user). You can always set the User ID later, and Amplitude has a built-in logic that will merge the anonymous events to the later identified user (see Example 2).

Do not assign a User ID that might change. If someone’s email can change within your app, then it is not a good idea to set it as a User ID as Amplitude will mark the person as a new user if they change their email.

This told, let’s see which is the default user journey that makes him/her pass from the status of “anonymous visitor” to the status of “identified user”.

The easiest user’s journey: no cross-domain nor cross-device tracking

Let’s start simple and consider the simplest journey that any user follows on any app or website:

Lands on one of the pages of the website or opens the app;
Signs up;
Logs in;
Uses the website or the app;
Logs out.

This is the default journey of any user on any app or website.

Nothing complex to understand here: we are considering the use of a single device and the navigation of a single domain/app. No cross-device or cross-domain tracking.

We will start with this scenario analyzing how it is handled in each of the three tools we are considering.

We will call this the “Default User’s journey”.

Once we will have understood how to handle this Default User’s Journey, then we will go one step further and will analyze how to take into consideration cross-domain and cross-device tracking.

What do we want to track in the Default User’s Journey

Before making our hands dirty, let’s better understand what we are trying to accomplish.

We want to create a user profile on our tracking tool once the Visitor signs up (step 2);
We want to link with this user profile all the events (s)he fired while (s)he was anonymous (step 1);
We want to track all subsequent events, linking them to the User profile we created (step 3 and 4);

MixPanel has a great illustration that explains exactly what we are trying to accomplish.

We will leave apart the last step: login. In fact, this requires to explain some more things and for the moment we want to leave things easy.

We will address the login problem later.

Let’s start!

Note: All the three tools offer a wide variety of SDKs to make you able to integrate your app into whatever language it is written in.

In those examples, we will use the Javascript SDKs as the examples take into consideration the implementation in a web app.

Anyway, the examples can be adapted to be used in any other language.

How to track the Default User’s Journey in Amplitude

In the Javascript Documentation of the Amplitude’s SDK, there is a section named Setting Custom User IDs.

The process to identify a User is really simple: use the setUserId('USER_ID') method:

amplitude.getInstance().setUserId('USER_ID');

How to track the Default User’s Journey in MixPanel

MixPanel has an overview video you can watch to understand how generally the identification of visitors works on their platform.

MixPanel has two main methods for identifying visitors:

mixpanel.identify();
mixpanel.alias().

For the moment we will ignore the second method mixpanel.alias(): we will use it later in this post, when we will understand how works in MixPanel the merging of users and visitors.

So, for the moment, we will focus only on the first method: mixpanel.identify(), also if the example from MixPanel uses the second one, too.

How to track the Default User’s Journey in Woopra

In the Javascript Documentation of Woopra, they say we need to call the method identify().

But there are some things to consider:

In order to identify a customer, you need to send their email address to Woopra as a custom visitor property. Calling identify() does not send anything to Woopra, in order to send the properties and identify the visitor, you need to call track() after you identify(), or if you do not wish to track an event, you may call woopra.push() to send a call strictly with identifying information.

Woopra, too, as Mixpanel did, suggests us to use the email as the identifier: we continue to prefer to use our internal UUID and send the email attached as a User property (more on this debate later).

How to concretely track the users: code example

Now that we know how to identify a user with each of the three tools, let’s make our hands dirty and identify users.

But… there is a thing to consider first: how is your login system structured?

Maybe you have not thought at this too much, but if you do, you can discover that there are three main flows for the login:

With an intermediate logged in page (where you say something like “Hello [User], you are now logged in. We are redirecting you to [page requested | you user profile]”);
Without an intermediate logged in page BUT WITH a refresh;
Without an intermediate logged in page AND WITHOUT a refresh (Single Page Applications – SPA).

Do you understand what I’m saying?

The first flow, shows you a form (on a dedicated login page or in a form on any page): once you have provided your credentials, the system first redirects you to another page where it shows you a “You are now logged in message” and then, maybe, it redirects you to the page you were browsing or to your profile.

The second flow shows you a form for the login: if it is on the page you are browsing (maybe in the top-right part of the page), it refreshes the page and logs you in; if you log in from a dedicated page, it usually redirects you to your profile or main dashboard.

The third flow has any of the steps of the previous flows: whether you use a form on the browsed page or a dedicated page, no refresh happens as all is handled in front-end by a JS library like React or Angular that simply make the call to the login API you implemented and then, once the login is confirmed, update the DOM, all without any refresh nor redirect.

Why are we speaking about this?

Because each flow has its own implementation details: the only constant is the fact that you have to call the method to identify the user.

So, starting from the first flow, this is the easiest to track: simply call the method identify() in the page that confirms the login.

Jumping to the third flow, this case is easy, too: before you update the DOM you need to call the methods to identify the user.

When exactly you call those methods depends on your concrete implementation and framework of choice: unfortunately, I cannot help you more than this.

The most tedious approach to track is the second: no intermediate page and a refresh.

This is the most problematic as you practically have no concrete page where to call the the methods to identify the visitor: after the click on the “Login” button, the user is redirected to another page, but usually this doesn’t render anything and simply send a header to the browser to redirect it to the page from which the “login” button was clicked.

There are many possible ways to identify the user that logs in following this flow:

Call the methods to identify the users on all pages by simply checking if the user is logged in or not: if (s)he is, then identify him/her;
Call the identify() method using the server-side SDKs: the controller that logs in the user, other than set the cookie, calls the APIs of the tracking software, too;
Use a query string parameter to mark a page visited just after a login;
Use the session to store a parameter to read on successive page load: if present, then identify the user.

The first approach is horrible and I have no intention of talking about it further.

The second one is possible, but it requires another dependency in your server-side code base, probably a break of the SOLID principle (unless you use events, but this means other unit tests, more code to maintain, handling possible failures in communicating with the tracking APIs, possible delays waiting a response and the list can continue).

The third can be easy to implement, but if the user refreshes the page, it will be identified again: nothing so bad, but anyway a duplication.

TrustBack.Me is built on top of the Symfony framework and it has a convenient feature called flash messages:

You can also store special messages, called “flash” messages, on the user’s session. By design, flash messages are meant to be used exactly once: they vanish from the session automatically as soon as you retrieve them. This feature makes “flash” messages particularly great for storing user notifications.

This really seems the best approach:

It is clean;
Doesn’t have refresh issues of doubled identifications;
Automatically handled by Symfony;
No stress.

Let’s use it!

Step 1: Create the flash message after the user logs in

To create the flash message we need to create a subscriber that listens for the event security.interactive_login: during the login, in fact, Symfony fires some events that we can listen for.

We need to use this as

When a provider authenticates the user, a security.authentication.success event is dispatched. But beware – this event will fire, for example, on every request if you have session-based authentication. See security.interactive_login below if you need to do something when a user actually logs in.

So, this is our subscriber:


// src/Subscriber/LoginAnalyticsSubscriber.php

<?php

declare(strict_types=1);

/*
 * This file is part of the Trust Back Me Www.
 *
 * Copyright Adamo Aerendir Crespi 2012-2019.
 */

namespace App\Subscriber;

use Symfony\Component\EventDispatcher\EventSubscriberInterface;
use Symfony\Component\HttpFoundation\Session\Session;
use Symfony\Component\Security\Http\Event\InteractiveLoginEvent;
use Symfony\Component\Security\Http\SecurityEvents;

/**
 * Subscribes to the event `security.interactive_login` and
 * sets a flash message that marks the login.
 *
 * This way, in Twig, we can check for the existence of this
 * flash message and identify the user in analytics tools.
 */
class LoginAnalyticsSubscriber implements EventSubscriberInterface
{
    /**
     * {@inheritdoc}
     */
    public static function getSubscribedEvents():array
    {
        return [
            SecurityEvents::INTERACTIVE_LOGIN => 'onInteractiveLogin',
        ];
    }

    /**
     * Sets the message in the flash bag.
     *
     * @param InteractiveLoginEvent $event
     */
    public function onInteractiveLogin(InteractiveLoginEvent $event):void
    {
        /** @var Session $session */
        $session = $event->getRequest()->getSession();
        $session->getFlashBag()->add('identify', 'identify');
    }
}

Step 2: Read the flash message in the Twig template

In our template we use this:

<!DOCTYPE html>
<html lang="{{ locale }}">
    <head>
        ...

        {% if is_granted('IS_AUTHENTICATED_FULLY') or is_granted('IS_AUTHENTICATED_REMEMBERED') %}
            {% for label, messages in app.flashes(['identify']) %}
                {% for message in messages %}
                    <script type="text/javascript">
                        console.log('identifying the user.')
                        amplitude.getInstance().setUserId('{{ app.user.id }}');
                        mixpanel.identify('{{ app.user.id }}');
                        woopra.identify({id: '{{ app.user.id }}'});
            amplitude.getInstance().logEvent('Login');
                        mixpanel.track('Login');
                        woopra.track('Login');
                    </script>
                {% endfor %}
            {% endfor %}
        {% endif %}
    </head>
    <body class="site">
        ...

As you can see, after having set the User ID I also immediately log a “Login” event.

This is for two reasons:

It’s a good thing to track the logins as this gives you the measure of the engagement of users with your app;
You can see Users profiles only after having sent at least one event after the ID setting (more on this in a bit).

Note that I’ve inserted a console.log() message to have a feedback in the Javascript Console: I will obviously remove it in production, once I’ve checked the user identification works as expected.

Now that we have our identification in place, let’s go to see what happens in Amplitude, MixPanel and Woopra.

What happens in Amplitude when a user is identified

In Amplitude, Click on the “New” blue button

Then click on “User look-up”

Et voilà, here there is the list of all your users

If you click on the ID of any user, you will see his/her details

As you can see, all the previous events I triggered in the app, before I logged in, are now merged with my profile.

This means that, once you identify a User, you can see what (s)he did on your app, so rebuilding his/her full journey.

You can find more information about the User activity tracking in the dedicated page in the Amplitude documentation.

What happens in Woopra when a user is identified

In Woopra:

Click on “People” in the top horizontal menu;
In the report configuration, click on “New Column” pink button;
Select the column “Visitor Schema > id”;
Apply the changes (not shown in the image);
Click on the “Run” green button in the upper right corner of the page.

You will get the list of visitors that interacted with your website: click on the ID:

And those are the details of the user we have just identified:

Woopra is able to merge all previous anonymous events with the user profile once (s)he is identified. This way you can build the full user journey, no matter where it started and no matter how many time it was first an anonymous user that was then identified.

What happens in MixPanel when a user is identified

In MixPanel only one small thing happens: the random generated distinc_id is now changed to your custom distinct_id. Nothing more, nothing less: past events are not linked with the new distinct_id nor a user profile is created.

To link anonymous events with the identified user (and so, with his/her new distinct_id) you need to call mixpanel.push().

To create a user profile, you need to call mixpanel.people.set().

Nothing happens automatically in MixPanel.

So, we need to understand how to link the anonymous events fired by the user before (s)he is identified with the events (s)he fires when (s)he is then identified.

Building the full default user’s journey in MixPanel

As mentioned, when we call mixpanel.identify() the only thing that MixPanel does is to change the distinct_id from the randomly generated one to the custom one we set.

This call doesn’t link the past anonymous events (tied to the randomly generated distinct_id) with the new custom distinct_id making impossible to get a clear picture of the user journey in the reports.

More, MixPanel doesn’t even create a User profile where we can read all the User’s Properties we set nor where we can see all the events (s)he performed.

To solve this problem we need to call mixpanel.alias().

They use a fictional user to explain how to use mixpanel.alias() in conjunction with mixpanel.identify():

Sally comes to your website for the first time. Mixpanel assigns Sally a randomly generated ID, which is known as a Mixpanel distinct_id.

Mixpanel assigns Sally the distinct_id “12345”. Now all her actions are tied to that distinct_id.

After clicking through a few pages, she successfully signs up for an account.

The signup confirmation page calls the mixpanel.alias() method and passes Sally’s email address as an argument. For example, mixpanel.alias(“sally@gmail.com").

The mixpanel.alias(“sally@gmail.com") method doesn’t change her Mixpanel distinct_id. It adds the ID “sally@gmail.com” to a Mixpanel lookup table and maps it to the original Mixpanel distinct_id “12345”.

Now Mixpanel calls the mixpanel.identify("sally@gmail.com") method and passes the ID sally@gmail.com to all subsequent pages and logins whenever Mixpanel sees data for “sally@gmail.com”.

Mixpanel remaps her original distinct_id of “12345”. So all actions Sally takes – whether on your site, in your app, or anonymously before she signed up for her account – maps to her.

Just a note: MixPanel, as you can see, uses the email address: don’t follow their example and use the unique ID you assign on your system to avoid breaks in your data if you make possible for your users to change their email address!

If you want to have the email at hand when consulting reports, add it as a property of the user object.

So, summing up:

The visitor lands on your website or opens your app: (s)he is anonymous and has a randomly generated distinct_id (“12345”). (s)he navigates your website or app and fires some events that are tied to the distinct_id “12345”.
The visitor signs up: now (s)he is identifiable and you need to assign her (she’s Sally!) a custom distinct_id, usually the ID your system assigned her. As in this moment MixPanel still recognizes her by the randomly generated distinct_id it assigned her (“12345”), you need to tell MixPanel that from now on you want to recognize her also using your custom distinct_id that is SALLY_CUSTOM_USER_ID. So you call the method mixpanel.alias('SALLY_CUSTOM_USER_ID'). Now MixPanel knows that if it finds in the events it collects the distinct_id “12345” or either the distinct_id SALLY_CUSTOM_USER_ID, it has to link them together considering them as fired by the same person. This is done internally using a cross-table that links the two distinct_ids
Now Sally logs in: you call mixpanel.identify('SALLY_CUSTOM_USER_ID');
Sally continues to navigate your website or your app and continues to fire events: those events are tied to the distinct_id SALLY_CUSTOM_USER_ID but MixPanel knows that previous events tied to the distinct_id “12345” were fired by the same person that is now identified by the distinct_id SALLY_CUSTOM_USER_ID.

So, basically, what the other two tools do automatically, in MixPanel has to be done by ourselves making the implementation a bit more complex.

Before going to make our hands dirty with the code, we need to know another important thing.

MixPanel warns about one thing: call mixpanel.alias() only once during the life of a visitor.

They say this:

An alias can only point to one Mixpanel distinct_id.

If you’ve already mapped “sally@gmail.com” to Mixpanel distinct_id “12345”, any attempt to map sally@gmail.com to Mixpanel distinct_id 67890 fails.

This prescription is in practice a big limitation as it makes practically impossible a full tracking of a cross-device journey.

But as here we are considering the Default User Journey, I will not explain you more, postponing the question to the post about cross-device tracking.

One last thing: MixPanel requires that the call to the mixpanel.alias() method is made BEFORE using the new custom distinct_id.

Coming back to our code, we can do a simple thing: add the call to the mixpanel.alias() method directly before the call to the mixpanel.identify() call: in fact, if we call the mixpanel.alias() method more than once, nothing happens: no errors, no link between distinct_id.

So, also if technically, as developers, we always prefer to have all in order in our code, in this case, we can do an exception and rely on the behavior of MixPanel that doesn’t trigger any error in case of multiple calls to the mixpanel.alias() method: it simply does nothing when receives consecutive calls.

So, we need to only modify our Twig template and our code becomes like this:

{% if is_granted('IS_AUTHENTICATED_FULLY') or is_granted('IS_AUTHENTICATED_REMEMBERED') %}
    {% for label, messages in app.flashes(['identify']) %}
        {% for message in messages %}
            <script type="text/javascript">
                console.log('identifying the user.')
                amplitude.getInstance().setUserId('{{ app.user.id }}');

                // Add this call to the `alias()` method
                mixpanel.alias('{{ app.user.id }}');

                mixpanel.identify('{{ app.user.id }}');
                woopra.identify({id: '{{ app.user.id }}'});
                amplitude.getInstance().logEvent('Login');
                mixpanel.track('Login');
                woopra.track('Login');
            </script>
        {% endfor %}
    {% endfor %}
{% endif %}

Now we need to do one last thing: creating the User profile on MixPanel.

MixPanel says that:

A Mixpanel People profile reflects the most recent information about a user.

Mixpanel builds profiles by connecting information about a user to a distinct_id, which creates a location that collects current information about a user.

People profiles enable you to:

Analyze how users navigate an application.

Store and update additional information about your users.

Use the Messages and Campaign feature.

Mixpanel People properties contain properties that describe a user and the activity feed that displays events the user performed.

You can use the Explore report to aggregate and organize a collection of People profiles.

So, to get in MixPanel the same functionalities we get automatically with Amplitude and Woopra, we need to manually create the user profile.

Doing this is as simple as adding one more call to the mixpanel.pople.set() method, again, editing only our Twig template:

{% if is_granted('IS_AUTHENTICATED_FULLY') or is_granted('IS_AUTHENTICATED_REMEMBERED') %}
    {% for label, messages in app.flashes(['identify']) %}
        {% for message in messages %}
            <script type="text/javascript">
                console.log('identifying the user.')
                amplitude.getInstance().setUserId('{{ app.user.id }}');
                mixpanel.alias('{{ app.user.id }}');
                mixpanel.identify('{{ app.user.id }}');

                // Add this call to the `people.set()` method
                mixpanel.people.set({ id: '{{ app.user.id }}'' })

                woopra.identify({id: '{{ app.user.id }}'});
                amplitude.getInstance().logEvent('Login');
                mixpanel.track('Login');
                woopra.track('Login');
            </script>
        {% endfor %}
    {% endfor %}
{% endif %}

And the User’s profile is now created in MixPanel, too:

As you can see, also the anonymous event I fired EmailSoftReclaimRequest is now tied with my profile (plus some other test events ?).

Handling the logout

So you now know who is your visitor (now we can also call him/her a User!) and you can tie the events (s)he fires to his/her profile you created in your tracking software.

One last thing remains to do: handling his/her log out.

With this paragraph, we start introducing the next relevant topic: handling non-default journeys of your users.

Are you asking yourself why should you need to handle the logout?

Because one of the non-default journeys of your users is the use of the same device by different users: if my wife and I use the same device to navigate your site, you will tie our events to one profile while we actually are two different people with completely different behaviours.

This promiscuity in registering events will lead you to get wrong data, then wrong reports and, in the end, wrong conclusions.

And, more, when my wife logs out and then I log in, you will have only one profile that in the best case will get updated once with my wife’s data and then with my own data, having it changing continuously.

So, you need to handle the logout.

How to handle the log out with Amplitude

To handle the logout in Amplitude you need to do two things:

Set the User ID to null;
Regenerate the device_id.

The code required to do this is the following:

amplitude.getInstance().setUserId(null); // not string 'null'
amplitude.getInstance().regenerateDeviceId();

Amplitude warns that

if you choose to do this, then you will not be able to see that the two users were using the same browser/device.

But this is not a big issue as we will anyway have two users using the same kind of browser, the same OS, etc. and this the important thing while understanding precisely that it is exactly the same device is not important (not always important, at least).

How to handle the log out with MixPanel

In MixPanel the handling of logout is simple, too, but it has some other consequences that are typical of MixPanel:

you are resetting the Distinct Id and removing all existing super properties.

While resetting users is useful if you have many users on the same device, it does have some undesirable effects. First, all events for logged out users will appear anonymous, meaning the unique user counts for these events will not be correct. In addition, because you remove super properties, you will need to again register these for each user on login.

Ultimately the tradeoff for the above drawbacks is that each profile is one unique user within your implementation. We typically only recommend implementing the above if you anticipate this scenario happening as the norm – if multiple users on the same device is not common, implementing logic on logout to handle this scenario may be more trouble than it is worth.

What do I think about this? You have to evaluate the tradeoff (yes, the exact same thing MixPanel told us ?).

In our app TrustBack.Me, we decided to implement the logout handling as we don’t know in advance if more people will use the same device but we evaluated that handling proper user tracking is more important than the effort required (that is, incidentally, not so much anyway).

So, in this MixPanel is easier to use as it requires only one call:

mixpanel.reset()

For JavaScript, calling the reset method will clear the Distinct Id and all super properties, as well as generate a new Distinct Id for the user.

How to handle the log out with Woopra

I was not able to find information about how to handle the logout of users with Woopra.

So I contacted their support that in less than 24 hours told me to use the method woopra.reset() and also to set to an empty object the user’s properties:

woopra.reset();
woopra.visitorData = {};

Concrete implementation of logout handling

For the login, we used Symfony events to mark the login.

Unfortunately, Symfony doesn’t fire corresponding events for logout.

But it permits to do two things:

Configure a page to which redirect the user after a successful logout;
Use a logout success handler that is simply a service implementing the LogoutSuccessHandlerInterface.

When first implementing the logout handling, we wanted to use the same approach we used for login, setting another flash message to mark the logout:

namespace App\Subscriber;

/**
 * Subscribed to the event `security.interactive_login` and sets a flash message that markes the login.
 *
 * This way, in Twig, we can check for the existence of this flash message and identify the user in analytics tools.
 */
class AnalyticsSubscriber implements EventSubscriberInterface, LogoutSuccessHandlerInterface
{
    /**
     * {@inheritdoc}
     */
    public static function getSubscribedEvents():array
    {
        ...
    }

    /**
     * Sets the message in the flash bag.
     *
     * @param InteractiveLoginEvent $event
     */
    public function onInteractiveLogin(InteractiveLoginEvent $event):void
    {
        ...
    }

    /**
     * @param Request $request
     *
     * @return RedirectResponse
     */
    public function onLogoutSuccess( Request $request ):RedirectResponse
    {
        /** @var Session $session */
        $session = $request->getSession();
        $session->getFlashBag()->add('identify', 'logout');

        return new RedirectResponse($request->getSchemeAndHttpHost());
    }
}

As you can see we returned a RedirectResponse object as the interface obliges to return a ResponseInterface object. So we thought to redirect the User to the homepage after the redirect.

Unfortunately, the redirect resets also the flash messages and so it is impossible to mark the login.

So we have to go with the first approach.

Basically, you need to create a logout.html.twig template and implement in it the code to reset the tracking.

So the full Symfony’s configuration requires those steps:

Create a SecurityController and inside it a method bye() that will be the route that will render the logout.html.twig template:

<?php

declare(strict_types=1);

/*
 * This file is part of the Trust Back Me Www.
 *
 * Copyright Adamo Aerendir Crespi 2012-2017.
 *
 * This code is to consider private and non disclosable to anyone for whatever reason.
 * Every right on this code is reserved.
 *
 * @author Adamo Aerendir Crespi <hello@aerendir.me>
 * @copyright Copyright (C) 2012 - 2017 Aerendir. All rights reserved.
 * @license SECRETED. No distribution, no copy, no derivative, no divulgation or any other activity or action that
 * could disclose this text.
 */

namespace App\Controller;

use FOS\UserBundle\Controller\SecurityController as FOSSecurityController;
use Sensio\Bundle\FrameworkExtraBundle\Configuration\Route;
use Symfony\Component\HttpFoundation\RedirectResponse;
use Symfony\Component\HttpFoundation\Request;
use Symfony\Component\HttpFoundation\Response;

/**
 * {@inheritdoc}
 */
class SecurityController extends FOSSecurityController
{
    /**
     * Shows the logout page.
     *
     * This page is required to reset the tracking cookies.
     *
     * @Route("/bye")
     *
     * @return Response
     */
    public function logout(): Response
    {
        return $this->render('App/security/logout.html.twig');
    }
}

As you can see, I render the template directly in the controller without using the @Template annotation: this is a Symfony’s best practice.

Configure the security in the security.yaml file.

# config/packages/security.yaml

security:
    ...
    firewalls:
        ...
        main:
            ...
            logout:
                target: /bye
            ...

Finally, create the logout.html.twig template

{# templates/App/security/logout.html.twig #}

{% extends 'App/base.html.twig' %}

{% block title %}Logout{% endblock %}
{% block metaDescription %}Mostra che sei un commerciante affidabile. Ti serve solo un indirizzo e-mail.{% endblock %}

{% block body %}
    <div class="wrapper bg-light-gray">
        <div class="container">
            <div class="row">
                Logout
            </div>
        </div>
    </div>
    <script type="text/javascript">
        console.log('logging out the user.');
        amplitude.getInstance().logEvent('Logout', null, function() {
            amplitude.getInstance().setUserId(null);
            amplitude.getInstance().regenerateDeviceId();
        });
        mixpanel.track('Logout', null, function() {
            mixpanel.reset();
        });
        woopra.track('Logout', null, function() {
            woopra.reset();
            woopra.visitorData = {};
        });
    </script>
{% endblock %}

Note: there is a call to console.log(): you can remove it once you have seen how does the logout handling work.

As you can see, I first log the Logout event, then use a callback to reset the identifiers: this is to be sure the Logout event is associated with the logging out user. Without using a callback, there is the risk of logging the Logout event after the reset, linking the event to the new anonymous randomly generated ID assigned after the reset.

What happens in Amplitude when you handle the logout of users

Click on the “New” blue button and then select “User Look-Up”: you will see the list of events fired on your website.

As you can see from the screenshot, the user is assigned a random User ID 71984668067.

Then, once (s)he logs in, the ID is changed to 1 that is the ID (s)he has in TrustBack.Me database users table.

When (s)he logs out, the Logout event is registered with the ID 1, then (s)he is assigned a new random generated ID 71984682059.

This way, when someone else logs in using the same device, (s)he will have his/her own ID and we can track all activities very precisely.

What happens in MixPanel when you handle the logout of users

If you click on the LiveView in MixPanel, you will see the last events tracked.

As you can see, there is a user that did the login, viewed some pages and then logged out. After the logout, MixPanel correctly tracks another page view, assigning the event to a different user.

What happens in Woopra when you handle the logout of users

Woopra registers the two users: one is identified (has the custom ID 2) while the other is anonymous.

The anonymous user is actually me after the logout, so Woopra successfully resets the ID.

In Woopra I’m observing a strange behaviour: I’m expecting the Logout event being tied to the identified user.

Instead, Woopra seems to first reset the user and then log the Logout event, also if the resetting is done in a callback (and so, the code should be executed only after the tracking of the Logout event is completed – and tied to still identified user).

As we will not use Woopra in our applications here at Serendipity HQ, I will not going deeper in this behavior: if you will use Woopra, keep in mind this and, maybe, contact their support: they are very fast and exaustive!

The two images show what I’m saying.

In the image below, I expect to see also the Logout event, but it isn’t there.

Instead, it is tied with another anonymous user, the one created after the identified user does the logout.

Conclusions

So we have completed the tracking of the default user’s journey:

Anonymous navigation;
Registration;
Login;
Navigation as a logged-in user;
Logout.

The logout opens the first big issue we will face with tracking: handling multiple users on the same device.

This is only one of the possible issues we have to handle.

There are many others: check them out in the next post and learn how to deal with them and how to solve them.

Remember to “Make. Ideas. Happen.”.

I wish you flocking users, see you soon!

L'articolo How to track unique users with behavioural analytics tools proviene da ÐΞV Experiences by Serendipity HQ.

How to track events with behavioural analytics tools

Adamo Crespi — Fri, 16 Nov 2018 18:49:59 +0000

The easiest way to start using concretely a behavioural analytics tool is to start tracking some events.

This way you will see them registered in the behavioural analytics tool of your choice, and you can start better understanding how they work.

As told, in theory is all simple.

But the problems arise when you have to concretely decide which events to track.

How to decide which events to track

In your first attempt at identifying events, maybe you will be influenced by the “Pageview centric vision”.

This concept will be more clear in a bit.

For the moment, let’s start by understanding what is an Event in the tracking context.

Then I’ll provide you with a concrete example of a “Pageview centric vision” and how to extract events from it.

The Amplitude’s Data Taxonomy Playbook explains very well what an event is and also provides a lot of helpful considerations about them.

So, what is an event? An event is any distinct action a user can perform in your product (e.g. start game, add to cart) or any activity associated with a user (e.g. in-app notifications, push notifications). When deciding what events to track, it is helpful to think of event categories first.

Start first by determining what your product’s critical flow is. For a food delivery app it would be order placed, for a healthcare app it would be sessions started/booked. When you have your critical flow you can go through and determine what types of events you’ll need in order to best understand that critical event and the flows surrounding the event (it is not enough just to know how many people placed an order or booked a session, but also what factors were causing them to enter that flow.)

Identifying broad categories of your events will help you conceptualize the major components of your product. It also will allow you to better organize your events in Amplitude’s UI. Make sure that each category tracked will help towards accomplishing one of your business objectives. Sticking with the business objectives we defined in the above pre-work section, event categories could be:

Onboarding

Checkout

Once you have determined your overarching categories, start to drill down into the individual events in these groupings. Begin with the top priority events that align with your business goals.

You do not need to track every event in your product as doing so will increase noise in your project and make your taxonomy difficult to understand. We recommend instrumenting no more than 50 events in your first pass. Having fewer events enables you to think about and focus on what is truly important to track in your product. However, you do want to detail out every event in a critical path (e.g. onboarding flows, purchase funnels). Without carefully instrumenting every step, you will not be able to accurately see and analyze drop-offs.

When deciding on events to track, keep in mind the questions you want to answer and the hypotheses you want to test. While we would not expect you to know 100% of the questions and hypotheses you’ll be testing down the road, try to be as diligent as possible when selecting actions to record in Amplitude.

Common questions to consider are:

Are my users successfully completing onboarding?

How many of my users convert into paying users?

How many times are my users performing Event A vs. Event B?

What are my most engaged users doing?

Which groups of my users are engaging with feature C?

So, there are five main steps to take to figure out what events to track:

Determine the business goals;
Determine the product’s critical flows;
Identify broad categories of events;
Formulate the questions you want an answer to from the tracked events;
Identify the single events in each category.

The definition of business goals (first point) is an entire topic on its own and cannot be covered here.

However, as a starting point, you can think about those goals for your business:

Increase onboarding conversion;
Increase retention of paying users (customers);
Increase checkout funnel conversions.

In defining the goals, you should think about the impact that their improvement has on the bottom line.

Events serve only one purpose: understanding the User Journey

This concept is explained at its best once again by Amplitude:

As you start creating your event taxonomy one important thing to think about is the end user journey as they go through your product. Your taxonomy should allow you to split your analysis into 3 different levels

Event counters – Being able to track DAU/MAU, Total purchases etc.

Funnels and conversions – Retention rates, Funnel conversion, Power curve chart

Behavioural analysis – Impacts of performing actions on your other metrics

If we think of a new user we can map their journey from new to ultimately to a power user. The key here is to understand the factors that is causing a user to transition between these states and having clear events tracked for these key areas will allow you to better analyze patterns that help/hinder users from transitioning between these states.

How to concretely determine the events to track filtering them from all the possible actions a User can perform

Now we know in theory how to decide which events to track.

When coming to concrete things, however, we face a concrete problem.

Let’s explain this problem with an example.

When deciding what to track on TrustBack.Me, we decided to start simple and start tracking the Email Soft Reclaim.

The Email Soft Reclaim Journey permits any user to provide their email address to see if TrustBack.Me has any orders associated with it.

Practically, it permits to check the orders associated with any email without requiring a registration (so, no password required!).

When we tried to slice the journey, we found there are those actions:

The user lands on the starting page of the Email Soft Reclaim Journey;
The user fills the form with his/her email;
The user clicks on the button to submit the form;
The form redirects to a confirmation page that explains that an email was sent to the provided address;
TrustBack.Me sends an email with the access link to the provided email address;
The user clicks on the access link;
The user lands on the access page and sees the orders associated with the email (if any);
The user continues taking other actions (see the details of an order, a feedback (s)he released and so on).

Now, given this sequence of actions, which are the relevant events that they fire and that we needed to track?

So, at this point, we started to formulate the questions to which we want an answer from the tracked events.

The questions we formulated were these:

How many Email Soft Reclaim requests are submitted in a defined period of time?
How many Users, after having submitted the request, then effectively see the list of orders?
What do they do next?

Three simple questions to which we can answer properly tracking our events.

So, the possible events we can track are those:

View of starting page of Soft Email Request;
Click on the submit button;
View of confirmation page;
Click on the access link;
View of the access page with the list of orders.

Five actions and each one of them can be considered an event that can be tracked.

Now, let’s consider the usefulness of each of those events (spoiler: I’m starting to show you what does it mean “Pageview centric vision”).

1. View of starting page of Soft Email Request.

Do we really need to track this? Maybe we don’t. At least not in this first tracking instrumentation.

In fact, tracking this will for sure tells us how many people viewed the page and, combining this information with the last event (view of access page with the list of orders), permits us to understand the ratio between the requests and the accesses.

But this is not a relevant information, also if at a first look it seems it is.

In fact, this is not currently a business goal nor is relevant to know the drop off between the starting page views and the concrete access: the access to the initial page, in fact, can happen for two reasons:

For curiosity, just to see what the page contains;
For necessity, because the user really wants to see the list of his/her orders.

We need to track the performance only for the second group of people, not the first.

So, as we have other actions after this, we can use them as the starting point of our tracking.

We decided that this event is not relevant and we don’t track it.

2. Click on the submit button

This event is more close to what we want to understand from our tracking: how many people start the Email Soft Reclaim (and then, how many of the started Soft Reclaims lead to a concrete access to the list of orders).

Maybe we should track this.

Before deciding, let’s understand better the next action.

3. View of confirmation page

This action, if tracked as an event, basically offers the same information of the previous action. In fact, it doesn’t show an error if we don’t know the email (for privacy reasons: telling if we have or not the email may disclose privacy information). Tracking errors is a good thing to do, but, as told, this is not the case.

So, we need to track only one between the “Click on the submit button” action and this: tracking both is a duplication that only causes messiness in the data collected.

NOTE: If this page had also shown an error in case the email was not present in our database, then, this would have been an event to track, but not mandatory.

4. Click on the access link

This is not really an event: more probably here we should track the UTM params, setting them in a way that makes us able to recognize a user as coming from an Email Soft Reclaim access link.

But this is a topic we will cover later in another post.

5. View of the access page with the list of orders

This is the last action in the journey, so we need to track it for sure as we need it to find the end number of people who arrive at the end.

So, how many events should we track and what are them?

So, in the end, we need to track those events:

Click on the submit button;
View of the access page with the list of orders.

All other steps are not relevant: we only need to add also the UTM query parameters to the access links in the emails we send (again, this topic will be explained deeply in another blog post).

Event properties

Now that we have our events defined, we need to understand which information we want to record each time one of them happen.

That information is stored in the properties of the event.

The properties provide additional information that can then be used to better analyze data.

Woopra documentation explains very well the usefulness of events’ properties:

[…] wouldn’t it be helpful to know a little more about each action the customers take? In practice, when a visitor performs the action “pageview,” users can find out what page was viewed, how long the customers there, what device was used, and so on.

This is where action properties come in to play. Think of action properties as specific attributes that provide additional details for each event you’re tracking.

For example, users may want to track every time a customer makes a payment, one could send the “Payment” action to Woopra, along with associated properties such as:

payment amount

user plan or package

payment option (e.g. credit card, Paypal, ACH), and so on.

Mixpanel offers another useful point of view about Events’ properties:

Event Properties are bits of extra information that you send along with your Events which describe the details of that action. They are usually specific to the Event they’re describing and don’t apply universally to other Events. Leveraging Event Properties allows you to conduct deeper analysis to better understand user behavior for a specific action.

While Event Properties generally describe the Event itself, they can also describe a user’s actions in relation to that specific Event. For example, for a Video Played Event, you may want to include a Property that lets you know if the user firing that Event finished watching the entire video or played only part of the video.

Other examples of Events and Event Properties you might send with that Event:

Event: Page View; Property : Name of page viewed.

Event : Video Played; Properties : Length, Artist, Genre.

Event : Made Purchase; Properties : Item(s) purchased, Total, Did the user have a coupon or not.

Event : Leveled Up; Properties : Level, Time to level up, Hours user played before this level up.

The difference between the Event Properties and the User Properties

In defining the properties of the events you want to track, you need to take into consideration also the difference between the properties of the events and the properties of the users you track.

Amplitude explains this difference very well:

The most general explanation of the difference between the two is that user properties are attached to users and reflect the current state of the user at the time of the event while event properties are attached to events and reflect the state of the event that was triggered.

Event and user properties can be sent along with every event to Amplitude. All of these values will be tracked with the specific event they were sent with and the user properties sent will also update the user’s user properties with the most recent values. Another way to think about user properties and event properties is that user properties apply to the user for all events going forward until they are changed, and event properties are applied only to events at the time they are sent.

So, there is no complexity in understanding what Event Properties are.

The complexities, as usual, arise when we have to decide concretely which properties to associate with our events.

At this point, it is possible that a lot of possible properties are coming to our mind.

But we are missing another important piece of information.

Tracking the relation between journeys using the Event Properties

Introducing you to behavioural analytics tools, I told you to think also about the relation between journeys:

Practically, each journey can lead to another or to more than one another journey.

How do we track these relations?

We need some links between them and those links are simply some IDs, exactly as what we do when creating the relations between our entities in the database.

Once more, Amplitude offers a great explanation of this concept:

[…] So, for this e-commerce example, we recommend having two custom event property values, ‘Property ID’ (‘Product ID’ I think they mean, editor’s note) and ‘Cart ID’. Then, use two funnels to track if a user has viewed and purchased the same product. Two examples are outlined below.

Adding Product to Cart Funnel

Step 1: ‘Product – Page Viewed’

‘Property ID’ = ‘3345’

Step 2: ‘Product – Quantity Selected’

‘Property ID’ = ‘3345’

‘Quantity’ = ‘1’

Step 3: ‘Product – Added to Cart’

‘Property ID’ = ‘3345’

‘Quantity’ = ‘1’

‘Cart ID’ = ‘1299’

Purchasing Cart Funnel

Step 1: ‘Cart – Viewed’

‘Cart ID’ = ‘1299’

Step 2: ‘Cart – Start Checkout’

‘Cart ID’ = ‘1299’

Step 3: ‘Checkout – Address Added’

‘Cart ID’ = ‘1299’

Step 4: ‘Checkout – Shipping Method Selected’

‘Cart ID’ = ‘1299’

Step 5: ‘Checkout – Payment Added’

‘Cart ID’ = ‘1299’

Step 6: ‘Checkout – Complete’

‘Cart ID’ = ‘1299’

So, using two IDs, PropertyID (again, I think this should be ProductID, instead) and CartID, we can understand if we can really convert a User from a product page to the checkout.

The concept of “Super Property”

The final concept we need is the one of Super Property.

The concept of Super Property is really of MixPanel, that has in place a system to make it easier to track them, but it can be helpful with every other tool you use, also if you have to track them manually.

So, as it is of MixPanel, let its documentation tell us about it:

Super Properties are a type of Event Property that you can set once to automatically attach to every Event you’re tracking. They are Event Properties that apply to all Events without having to manually include them in each mixpanel.track() call.

Because Super Properties send values associated with Events, they are useful for seeing Properties over time. For example, you could set a Super Property with all of your Events that indicates whether they were done by a free or a paid user. Then, if a user changes from free to paid, you can see over time which Events led up to that conversion and which specific Events they did as each type of user.

Other examples of Super Properties you might send with every Event:

Organization that a user belongs to or rolls up into

Account age or signup date

Referring user ID or invited date

Because Super Properties are stored in the user’s cookie or in local storage, you’ll want to be relatively discerning about what you set as a Super Property so that site or app performance is not compromised.

Apart from the technical and privacy considerations, the concept is really useful if applied to our tracking of events and we should consider it.

Deciding which properties to add to tracked events

So, now you should have a more clear vision of Events and of their properties.

But, as told, the things become to get messy when we try to apply concretely the abstract concepts.

So, let’s continue with our example of TrustBack.Me and its Email Soft Reclaim.

As told, we want to track two events:

Click on the submit button;
View of the access page with the list of orders.

Let’s start with the first: Click on the submit button.

Defining the properties of the Click on the submit button event

Event name: EmailSoftReclaimRequest

Properties:

email

The email is the minimum required to track the event.

But going deeper, we can find other interesting properties.

For example, we may like to know which kind of user requested to access the email: is a merchant or a simple customer? Or is an anonymous user?

We cannot decide now which other properties we want to track as we still don’t know anything about how to track users other than events: this is a key distinction and we need to understand it well to have a complete tracking setup.

Understanding how to track users is really more complex than what it might appear at a first look.

I will explain the tracking of users in detail in another post.

For the moment, let’s focus on the tracking of events.

This is the code we used to track the event in all the three tools, Amplitude, Mixanel and Woopra:

...
<div class="submit">
    <script type="text/javascript">
        const eventName = 'EmailSoftReclaimRequest';
        const eventProps = {
            "email": document.getElementById("email_reclaim_email").value,
        };
        const track = function() {
            // Mixpanel
            mixpanel.track(eventName, eventProps);
            // END Mixpanel
            // Woopra
            woopra.track(eventName, eventProps);
            // END Woopra
            // Amplitude
            amplitude.getInstance().logEvent(eventName, eventProps);
            // END Amplitude
        };
    </script>
    <button id="Start" class="btn btn-lg btn-block btn-secure" onclick="track();">{% trans from 'email_reclaim' %}email.reclaim.soft.form.submit.label{% endtrans %}</button>
</div>
...

Note that I created a function called track() that is invoked when the button is clicked: this way we can send the event to all the three tracking systems (yes: currently we are using them together to understand which best fits our needs).

Defining the properties of the “View of the access page with the list of orders”

Event name: EmailSoftReclaimAccessed

Properties:

email

Also for this event, the email is the minimum required to make the event useful.

This is the code we use to track the event in all the three tools, Amplitude, Mixanel and Woopra:

...
<script type="text/javascript">
    const eventName = 'EmailSoftReclaimAccessed';
    const eventProps = {
        'email': '{{ reclaim.email.email.toString }}',
    };
    // Mixpanel
    mixpanel.track(eventName, eventProps);
    // END Mixpanel
    // Woopra
    woopra.track(eventName, eventProps);
    // END Woopra
    // Amplitude
    amplitude.getInstance().logEvent(eventName, eventProps);
    // END Amplitude
</script>
...

Note that here I don’t use a function as this code is outside of a form and isn’t called on the click of any button: the code is simply executed when the page is loaded.

Now that we have our two events tracked, it is time to build the reports.

Building the funnel/journey reports from the two events

As told before, we want to answer the following questions:

How many Email Soft Reclaim requests are submitted in a defined period of time?
How many Users, after having submitted the request, then effectively see the list of orders?
What do they do next?

We are not able to answer the third question right now as we don’t have instrumented the events. But the first two questions can find an answer right now.

We only need to build the reports.

Building the Email Soft Reclaim Report in Amplitude

This is the list of the tracked events in Amplitude:

As you can see, we have both EmailSoftReclaimRequest and EmailSoftReclaimAccessed.

What we need to do now is to put them on the same report.

So, start by clicking on the “New” button on the upper left corner of the Amplitude dashboard:

In the new screen, select Chart > Funnel

In the new screen you can configure the funnel selecting the events you want to include in the analysis:

Amplitude will do the rest as you can see:

If you like the result, click on the “Save” button in the upper right corner of the page to save the report.

For more information about funnel reporting in Amplitude click here.

Building the Email Soft Reclaim Report in MixPanel

This is the list of the tracked events in MixPanel:

As you can see, in this case, too, we have both EmailSoftReclaimRequest and EmailSoftReclaimAccessed.

We now need to build a funnel report also in MixPanel

So, start by clicking on the “Funnel” tab on the horizontal menu and then click on “Build a funnel” button:

Now you can configure the funnel adding the relevant events:

Click on the “Save” button and MixPanel will show you the funnel and the dropoffs:

For more information about funnels in MixPanel click here.

Building the Email Soft Reclaim Report in Woopra

When you go to Woopra dashboard of your project, it doesn’t show any list of events: in Woopra, in fact, you don’t have a list of events. To see the details of each event, you need to go to the People report, then click on a visitor and you can see the list of all events fired by that person. Then, clicking on one event, you can see its details.

So, instead of showing a list of events, Woopra shows them all aggregated on the same chart:

Woopra calls a funnel report “Journey”, yes, the term we used early: we took it really from Woopra!

So, to create a funnel report in Woopra, you need to select the “Analyze” item in the horizontal top menu, then click “Journeys” in the “Report types” section of the left menu and in the page that loads, click finally on the “New report” button in the upper right corner:

In the modal window that appears, select the first option “Journeys”:

We are now ready to configure our funnel (Journey).

Add the relevant events to the journey and for each of them set also their name (yes: you need to write it manually -.-‘ – the price of flexibility?):

Then click on the green button “Run” in the upper right corner of the page and the journey is built:

As you can see, when I started to track events in Woopra, I never accessed the page that lists the orders associated with the email I provided.

If you like to know more about journeys (funnels) in Woopra click here.

Advanced funnels

A final note about Amplitude’s Funnels and Woopra’s Journeys: as I know that we as humans are visual people, I don’t want to leave you with the funnel I built that is really simplistic and ugly to see.

So, here there are more complex funnels built by the Amplitude’s and Woopra’s teams that better shows their functionalities:

Much prettier

NOTE: I wasn’t able to find a more advanced example from MixPanel. When I’ll find it, I will update the post.

Best Practices for events tracking

Finally, here there are some best practices from Amplitude about events:

Only instrument properties you think you will use. Having too many properties makes analysis more difficult.

Before you include sensitive information (email, addresses, etc.) consider if it goes against the privacy policies for your product.

Enable the UTM and referrer user properties to better understand where you users come from on the web.

Instrument key properties on every event in a critical path.

Others may be:

Be consistent with names of events: don’t mix “CamelCased” and “snake_cased” names;

Woopra, instead, gives us some ideas about what events to track in a saas application.

Conclusions

Our journey in implementing a behavioural analytics tool to track our SAAS product is still at the beginning.

A lot of things still remain to understand and to implement.

But now we at least had seen some basic reports and started to grasp the surface of behavioural analytics.

I’d like to point out some important things:

When you start implementing the tracking, focus only on a few properties and events: don’t over-track as this will make the implementation longer and the main mantra in the startup world is “Think big. Start small. Scale fast”. To follow it, you need a few events and a few properties;
Keep particular attention to the PII you collect in your events: in the examples I provided I used the email. The email is a PII and its use in the tracking of events has really important consequences for the purposes of privacy compliance (in other words, compliance with the GDPR). Be careful!

Remember to “Make. Ideas. Happen.”.

I wish you flocking users, see you soon!

L'articolo How to track events with behavioural analytics tools proviene da ÐΞV Experiences by Serendipity HQ.

How to start using behavioural analytics tools to track your SAAS product

Adamo Crespi — Fri, 16 Nov 2018 18:01:33 +0000

When it comes to tracking the numbers of your SAAS product, Google Analytics is not sufficient anymore. This is the hard truth, believe it or not.

To go one step further, you need to properly implement a behavioural tracking analytics tool and think about your SAAS product differently.

Integrating behavioural tracking thoughts into the ones about your SAAS product can be tricky. Really tricky.

If in theory, the things are really simple, when you arrive at the point of concretely define what to track you may become really confused.

I’m going to explain to you exactly what changes in your thoughts about your SAAS application when you think at behavioural tracking and how to plan and implement it properly.

The differences between Google Analytics and behavioural analytics tools

Google Analytics is really a great product.

With it, you can measure a lot of things and get a lot of actionable data.

However, Google Analytics is not the best tool to track the numbers of your SAAS product (or of your e-commerce) as it has some limitations.

This doesn’t mean that you have to remove it completely.

This only means that you need to pair it with another tool, with different features.

Yes, I’m speaking about behavioural analytics apps.

Here are some of the main differences between Google Analytics and the behavioural analytics tools you can use.

Retroactivity

When configuring Google Analytics, you need to already have in mind the numbers you need.

If at a certain point you need to measure something new or extract some other numbers, this can get really complex and the worst thing is that they may not be retroactive: this means that if you need to measure a goal you didn’t set early on, you simply have unusable data.

Also if concretely the data are in the Google Analytics servers, you cannot interpret them under a new light because Google Analytics doesn’t allow you to do it.

If you are running an e-commerce company, this may not be a big problem: after all, the metrics of an e-commerce are all really similar and well known in advance: in this case, you can properly configure Google Analytics beforehand and have all the data you need from the start.

But if you have to measure a SAAS application, then the things become really more heterogeneous and the metrics are not always predictable in advance.

Flexibility

Really linked to the previous point is this second one.

Google Analytics doesn’t make your life easy when you need to extract and link data in new ways at which you didn’t think the first time you configured the tracking.

In some cases, get additional data in an understandable format requires you to create new views or do tricky “hacks” to get the data you need or extract the evidence you need.

A behavioural tracking app removes this complexity making you able to elaborate and combine data the way you want, the way you need!

Point of view

This is maybe the best difference between Google Analytics and behavioural tracking tools.

Google Analytics is focused on page views while the behavioural tracking tools are focused on customers and the things they do WITH your application and IN your application.

Google Analytics measures pages first, and then, incidentally, users and their behaviours.

Behavioural analytics tools, instead, measure first customers, and eventually, if you like, also page views.

This is reflected also in the data you can get from them: while in Google Analytics, for example, you can get the keywords searched, this data is never present in a behavioural analytics tool (almost never as they anyway can track UTM params).

This kind of tools answers other, more relevant questions:

How do my customers use my product?
Why do they do one thing instead of another?
What my best customers did before to become customers?

Google Analytics can answer to these questions, too, but only incidentally and anyway always partially, and not at the level that that kind of software provides.

This is an important switch in perspective: the main mantra in the startup world (and in the e-commerce world, too) is “Customers first”.

Is for that reason that methodologies like “customer development” become so popular (and useful!).

To put it at its full power, you need to measure properly the metrics that matter.

The issues of Google Analytics that harm you

You cannot collect PII.

PII stands for Personal Identifiable Information.

Google explicitly prohibits to send personal data of visitors to Google Analytics. Collecting those information goes against the Terms of Use of Google Analytics.

The main purpose of a behavioural analytics tool, instead, is exactly the opposite: identify the visitor as a person, so it is possible to link his/her journey as an anonymous visitor to his/her journey as a registered user.

This opens a new world of analytics possibilities as you are able to really understand what your users want and do.

Going deeper into the differences with Google Analytics: the solved challenges

So, to be more clear, let’s better understand which are the challenges that behavioural analytics tools solve.

Woopra explains them very well:

1) Non-linear Customer Journeys and Challenges in Data Democratization

The modern customer journey is highly complex and non-linear in nature. To understand the customer experience across channels, teams and tools, you must expand beyond the silos of strictly marketing or product related touchpoints.

Today’s businesses need to derive insights from across the organization, leveraging marketing, paid and unpaid advertising, customer success, product, support, sales and other groups within the company.

Most businesses find it challenging to democratize data across the organization and obtain a 360-degree view of their customers to drive end-to-end business objectives.

2) Limited Understanding of Product Engagement

It’s crucial to know how prospective users engage with your product or service in order to identify leads at the right time and on the right channel. This is especially true for SAAS companies that offer a ‘freemium’ model when acquiring new customers. Often, they spend an unreasonable amount of time and resources on lead acquisition by sending repeated communications to prospects on an action such as a free trial sign up form.

Unfortunately, given the vast array of options available out there, such actions are no more telling of lead’s propensity to purchase. What’s worse is that unwanted emails or calls can do more harm than good by deterring them from your product or service altogether.

For a company focused on the customer experience, actions such as repeated use of certain features, downloading how-to tutorials and viewing software setup videos are much more indicative of interested prospects. However, there are few tools that track behavior across a product or service while enabling teams to engage with their users in real-time.

3) Functional Data Siloes

With an estimated 66 different SaaS applications that are in use per enterprise – primarily for web and social traffic based analytics activity – companies are unable to properly leverage the data flowing in their organization to generate meaningful and timely insights.

Moreover, the market is crowded with solutions promising results such as higher conversion rates that are generating further confusion about what solution to implement, at what cost, and at what time. Such a conundrum gives rise to data siloes that often fail to render timely insights to business users. Most importantly, companies often fail to realize how difficult it is to bring all the customer-centric data together in order to find a single source of truth.

Given the fragmentation of investments in tools across all sizes of enterprises, a unifying customer journey analytics solution integrates with all the relevant applications to provide a singular view of each customer and better analyze their interaction and needs across all channels. Companies need to anticipate their customers’ behaviors and shift how they think about and engage throughout the decision journey.

4) Inability to Monetize from Data

Although every organization is now aware that data is their gold mine, few have managed to establish a sound data strategy that will allow them to consistently monetize from it. Most companies have trouble simplifying their data architectures to build a single layer that augments all the relevant customer related information in one central location. Connecting with leads and channelizing efforts to truly convert engagement to revenue is that piece of the puzzle that unfortunately many are still trying to solve.

Which are the alternatives to Google Analytics

There a lot of tools on the Internet that can help you better and more deeply analyze your SAAS application.

The most popular are three (in simple alphabetical order):

They differ for the price, but the features are quite the same: track events, track customers and combine data in useful reports.

Think at tracking when you think at your product

Tracking is not something to think about later, when your product is built and ready to be shipped (really early! Does the acronym MVP sound familiar?).

During the development of your product, you need to think at the permissions of your users: each set of permissions usually links with a journey (a funnel) and with an actor.

You have to merge the tracking part with the development part: still during development, when you design your internal flows, you have to highlight the KPIs: how many visitors became users? How many users became customers? You are understanding the song, don’t you?

In doing this, you need to also consider privacy.

This is called Privacy by Design and is a fundamental part of your product: the GDPR requires you to have precise documents that explain what, why, how and for how much time you treat personal data.

This job is useful also for your documentation: you need to explain to your customers how to solve their problems using your products.

All this without speaking about marketing (not only content marketing and blogging).

So, thinking at tracking already during the design phases of your SAAS product reverberates its beneficial effects on at least five areas of your startup:

Flow design and usage paths;
Measurement;
Privacy;
Documentation;
Marketing.

This is not a win-win, this is a jackpot!?

Preparing for tracking

So, you are already doing the hard parts of the tracking instrumentation.

Below there is a step by step flow you can follow: it will turn useful also for developers and your product team: They will love you for this!

List your actors

Any person on your app can do a lot of things.

Now, some user can follow some journeys, some others can follow some other journeys.

When a user can follow a particular journey, we call him/her an Actor.

Practically, an actor is a User who is making a journey to do something.

For example, on TrustBack.Me the journeys that a User who is a Merchant can follow are different than the journeys that a User who is a Customer can follow. Incidentally, a User who is a Merchant can be also a Customer.

So, you should list your Actors.

Highlight the journeys (flows, paths, funnels)

Now that you have a clear picture of your Actors, you can go to the next step: identify the journeys they can make.

In your application, any user can do various things to accomplish a task.

This is commonly called a flow (the devs call it that), or a funnel (more salesy term) or path or a “journey”: yes, this last term is the one we prefer.

The term “flow” is aseptic, so logical.

The term “funnel” has implicitly a connotation of sell: you have sale funnel, the lead capturing funnel and so on.

The term “journey” perfectly highlights the experience your users experience using your product.

After all, the UX is the first thing you should take care of in building your SAAS application!

So, coming back to our main point, some journeys can be followed by some users, while some other journeys can be followed by some other users.

For example, in Facebook, a User can post a status, upload a photo, comment on a status or a photo, leave a Like, ecc.: all those things are a journey (or a flow from the development perspective).

So, the first thing you should do is to highlight the possible journeys your users can experience in your application.

Amplitude has a good Kickoff video (from about minute 3) and a concrete example that explain exactly how to identify the main journeys users can take in your app.

To map your journeys you can use a tool like Draw.io or Realtime Board.

2. Identify key events in journeys

Each journey fires some events: those are the events you are going to track and measure to see how they relate and which frictions exist in them.

Once you have your journeys highlighted, you need to understand which users can follow which journeys.

When a User follows a journey (s)he is not a user anymore: (s)he is an actor as (s)he acting like a particular kind of user.

For example, in Facebook, a Merchant who is promoting his/her page, act differently from a User who posts a status.

And also if the posting of a status can be done by a Merchant on his/her page or by a user on his/her own profile, those are two completely different journeys.

So, the next step is to give a name as an actor to each user who is following a journey.

For example, on TrustBack.Me, among the others, a User can be a Merchant or a Customer (someone who bought something on the store of a Merchant).

Highlight the relations between journeys

Journeys are not standalone sequences of events.

Practically, each journey can lead to another or to more than one another journey.

For example, on TrustBack.Me, the Reclaim Domain journey leads to the Getting Started journey which goal is to lead the Merchant to configure the connection between TrustBack.Me and his/her e-commerce store.

We also do our best efforts to make a buyer who releases a feedback (release a feedback journey) become a Consumer, following the Hard Email Reclaim journey: the two are linked. And once the Buyer becomes a Customer, we do our best to understand if we can make (s)he transition to the Merchant Actor figure following the Domain reclaim procedure (if applicable).

Highlight the transitions between actors

Continuing to navigate your journeys, any user takes the clothes of different actors.

So, you need to understand the possible transitions between those actors figures.

For example, with TrustBack.Me our interest is to make it happen that the actor Visitor transitions to the actor Merchant.

Or that a simple Buyer transitions to the actor Consumer.

When analyzing your data, you can track the progress toward your app’s main goals.

Conclusions

This was only a birds eye in your journey to implement a behavioural analytics tool to track your application.

There are many other things you need to consider, write and do before you are ready to fully implement and understand how to use a tracking tool like this.

Anyway, you now should have a better understanding of the differences between Google Analytics and behavioural analytics tools: Google Analytics focuses on Page Views; behavioural analytics tools focus on people and the actions they take in your app (events).

Remember to “Make. Ideas. Happen.”.

I wish you flocking users, see you soon!

L'articolo How to start using behavioural analytics tools to track your SAAS product proviene da ÐΞV Experiences by Serendipity HQ.

How to manage static images with Symfony’s Webpack Encore

Adamo Crespi — Fri, 06 Apr 2018 09:02:57 +0000

Any web project has basically 4 kind of places where images reside:

In the CSS;
In a font file (think at icons);
In the cloud (think at AWS S3);
In the project itself as static files.

Symfony’s Webpack Encore is great to manage the first two places as it is able to manage the images referenced in the CSSes and to manage fonts (as they are handled as CSSes are).

The third group of images, the ones in the cloud, have not to be managed by Webpack but by another bundle like LiipImagineBundle.

So we are left only with the fourth group: the static images.

Which is the problem with static images

When you want to use an image in your Twig templates (or any other asset) you are required to use the asset() function.

Something like this:

{# templates/stylesheets.html.twig #}
{% block stylesheets %}
     <link href="{{ asset('build/app_global.css') }}" rel="stylesheet" />
{% endblock %}

As you can see, the path references the build folder: its full path is public/build and it is created by Webpack Encore when running node_modules/.bin/encore dev (or simply yarn dev if you have the script set in your package.json).

This command does some simple things:

Reads the webpack.config.js file;
Combines all css and scss into one file and puts it in the public/build folder;
Finds all the images referenced in the css and copies them in the folder public/build/path/to/image;
Eventually minifies the generated css (if you configured the minification);
Combines all js files into one file;
Eventually minifies the generated js (if you configured the minification);

Then you can include the js and css files using the asset() function as shown above.

But running node_modules/.bin/encore dev doesn’t copy the images referenced in the Twig templates through the asset() function.

For instance, if you have a code like this, the images will not be copied in the build/images/privacy folder:

<div class="privacy text-center">
    <span ...>
        <img src="{{ asset('build/images/privacy/noSpam.png') }}" ... />
    </span>
</div>

And this is a problem as we practically cannot reference them through the asset() function!

What the Symfony Best Practices tell about managing static images with Webpack Encore

Nothing! The best practices don’t mention at all the static images, nor they have a section dedicated to them.

Better, they tell this:

Web assets are things like CSS, JavaScript and image files that make the frontend of your site look and work great.

So, they reference to the Symfony’s Webpack Encore documentation.

But this documentation doesn’t mention at all the handling of images: only the handling of CSS and JavaScript.

So, basically, in this moment there is no best practice in place to manage static images nor a documentation that explains how to manage them.

But static images are a big part of a web project as not all images have to be referenced in CSS nor have to be in a font or in the cloud.

We need a solution to manage static images with Symfony’s Webpack Encore!

So, here it is: lets see it!

How to manage static image files with Symfony’s Webpack Encore

Fortunately the Symfony’s community is very large and active, so, sifting both StackOverflow and the Symfony’s issues on GitHub it is possible to reconstruct a solution to manage static images with Webpack.

The two main discussion about this problem are these:

In these discussion there is the full solution to manage static images with Symfony’s Webpack Encore.

Lets see them one by one.

The first (ugly and messy) approach: putting all images directly in the `templates/images` folder

The simplest way to make the asset() function is to simply put the static images directly in the public/build/images folder: this way the function is able to find them and reference them.

But this approach, although very simple, has some major drawbacks:

You have your assets in two different places: in the assets folder and directly in the public folder (and also in the templates folder for twig templates: three different places!)
You cannot use versioning as Webpack Encore is not aware of these images and so cannot put them in the manifest.json file (more about soon).

So, if you don’t use versioning and you are ok using a messy approach, go this way and put your static images directly in the public/build/images folder.

I don’t like messy things and I want to use versioning, so I want a more clear and organized way.

The second (tiring) approach: requiring images through `require`

The basic concept here is that to make Webpack aware of images.

So, basically, we have to require all the images using require in our JavaScript files: this way Webpack can move them to the public/build/images folder.

Something like this:

// assets/images.js
require('./images/privacy/noSpam.png');

Then you have to import this file in one of your already existent files so it can be processed by Webpack Encore:

// assets/js/_main.js
require('../../images');

This way you will have the file noSpam.png copied in public/build/images/noSpam.d47c971d.png .

As you can see it was versioned and it was also added to the manifest.json file:

{
  ...
  "build/images/noSpam.png": "/build/images/noSpam.d47c971d.png",
  ...
}

This approach was suggested by Ryan Weaver, the author of Webpack Encore, and is already a solution but its major drawback is that we have to create a file to require images and also require any image we want to use in our Twig templates: this means extra work and also an error prone one as we may forget to add the image, or, anyway, we have to explain to anyone who comes working on the project that (s)he has to add the images to the file, and we the file name changes we have to update it in the images.js file too, and if we decide to add 10 more images we have to require each one of them, and if…

Definitely, this is a solution, but not a good one: we need something better.

The third (ÐΞV’s way) approach: introducing `require.context`

What we want is to simply put all the images we need into a folder and then let Webpack Encore do the rest, without any further action on our side.

So we need a way to make Webpack Encore able to

Scan a folder recursively;
Find the image files;
Move them to the public/images folder;
Add them to the manifest.json file.

But, how can we achieve this?

The solution is require.context as suggested by Vincent Le Biannic (aka Lyrkan).

The code is this:

// assets/js/_main.js
const imagesContext = require.context('../images', true, /\.(png|jpg|jpeg|gif|ico|svg|webp)$/);
imagesContext.keys().forEach(imagesContext);

The require.context call creates a custom context in Webpack:

The first argument is the folder to scan;
The second argument indicates to scan also the subfolders;
The third argument is a regular expression used to match the file names we want to include in the context.

Then the imagesContext.keys().forEach cycle through the result and require each found element.

Really simple!

The result is that:

All found images are copied in the path public/build/images/ folder (in a flatten way, without taking care of the original subfolder: it is not included in the resulting path);
All fund images are also added to the manifest.json file.

This is the resulting manifest.json file (at least a small portion of it ?):

{
  ...
  "build/images/NoAdvertisers.png": "/build/images/NoAdvertisers.97d35611.png",
  "build/images/Unsubscribe.png": "/build/images/Unsubscribe.c63a3aac.png",
  "build/images/noSold.png": "/build/images/noSold.2b558443.png",
  "build/images/noSpam.png": "/build/images/noSpam.d47c971d.png",
  "build/images/onlyIntendedUse.png": "/build/images/onlyIntendedUse.ac6db9bf.png",
  "build/images/timeIndefinite.png": "/build/images/timeIndefinite.45f77ed9.png",
  ...
}

This solution is very close to what we need but it has a (maybe minor) drawback, too: the files are copied all into the public/images folder, without reflecting the full path: this maybe an issue if you have two or more images with the same name in different folders and you are not using the versioning (enabling the versioning, anyway, solves the problem).

But also if you can solve the “same name” problem, remains the fact that having the same folder structure maybe helpful to find on the fly the image in the assets folder: not fundamental, but certainly useful.

So, a bit of syntactic sugar to reflect the folder structure is this:

// webpack.config.js
const Encore = require('@symfony/webpack-encore');

Encore
 // directory where all compiled assets will be stored
 .setOutputPath('public/build/')

 // Relative to your project's document root dir
 .setPublicPath('/build')

 // empty the outputPath dir before each build
 .cleanupOutputBeforeBuild()

 // Here all other required configurations
 ...

 .configureFilenames({
 images: '[path][name].[hash:8].[ext]',
 })
;

Doing this will reflect in public/build the folder structure of assets producing something like public > build > assets > images > sub_folder > you_image.png.

As you can see it anyway include the folder asset but removing it requires too much work and the effort is not worth the benefit.

Conclusion

We have seen how to manage static images with Webpack in a convenient way.

We have explored three ways of doing it:

hand-putting the images directly in the public/build folder (messy);
Using require for each image we want to be moved in public/build and versioned (tired);
Using require.context and a little bit of syntactic sugar in webpack.config.js (the ÐΞV’s way).

Which one of these solutions will you choose? ?

Mmm, I think I have no doubts about! ? … ?

Remember to “Make. Ideas. Happen.”.

I wish you flocking users, see you soon!

L'articolo How to manage static images with Symfony’s Webpack Encore proviene da ÐΞV Experiences by Serendipity HQ.

How to use the Doctrine command line developing a Symfony Bundle

Adamo Crespi — Fri, 30 Sep 2016 16:14:04 +0000

Maybe you need to use the Doctrine’s command line while you are developing a Symfony Bundle (or anything else).

This will make you able, for example, to validate the schema or to convert an annotated schema to its XML counterpart.

Here is how to use Doctrine command line outside of a Symfony Application.

You have to simply create a cli-config.php file and put it into the root directory of your developing bundle.

<?php 

use Doctrine\ORM\Tools\Console\ConsoleRunner;
use Doctrine\ORM\Tools\Setup;
use Doctrine\ORM\EntityManager;

$paths = [getcwd() . "/Model"];
$isDevMode = true;

// the connection configuration
$dbParams = array( 'driver' => 'pdo_mysql',
    'user' => 'root',
    'password' => 'root',
    'dbname' => 'test_your_bundle',
);

// @see http://stackoverflow.com/a/19129147/1399706
$config = Setup::createAnnotationMetadataConfiguration($paths, $isDevMode, null, null, false);

$entityManager = EntityManager::create($dbParams, $config);

return ConsoleRunner::createHelperSet($entityManager);

Now you can use vendor/bin/doctrine your:command to perform the task you need.

Very simple!

Remember to “Make. Ideas. Happen.”.

I wish you flocking users, see you soon!

L'articolo How to use the Doctrine command line developing a Symfony Bundle proviene da ÐΞV Experiences by Serendipity HQ.

Magento 2: SSL certificate problem: unable to get local issuer certificate (cURL problem)

Adamo Crespi — Fri, 29 Apr 2016 07:07:18 +0000

The message “SSL certificate problem: unable to get local issuer certificate” shows up when trying to connect to Magento Connect or when, generally, you try to use cURL to connect to a remote web site.

This error happens because cURL cannot find a cacert.pem file from which take the trusted signatures.

There are some ways to set this file in cURL:

Pass the cacert.pem file path directly to cURL when making the call;
Set the path to the cacert.pem file in the php.ini.

Other options are to set the environment variable CURL_CA_BUNDLE or to put the cacert.pem file in a defined directory on your filesystem depending on your OS.

But, as we are working with digital certificates with PHP cURL, lets use PHP! :)

Pass the `cacert.pem` file path directly to `cURL` when making the call

To do this, simply pass the cacert.pem file path as parameter to pass to stream_context_create() function:

$contextOptions = [
    'ssl' = [
        'verify_peer' = true,
        'verify_peer_name' = true,
        'allow_self_signed' = false,
        'cafile' = 'path/to/you/cacert.pem',
        'ciphers' = 'HIGH',
        'disable_compression' = true,
        'capture_peer_cert' = true,
        'capture_peer_cert_chain' = true,
        'capture_session_meta' = true,
    ]
];

$context = stream_context_create($contextOptions);

How to set the path to the `cacert.pem` file path in the `php.ini`

The other more robust solution is to set the cacert.pem file path directly in the php.ini.

To do this, find the line curl.cainfo:

[curl]
; A default value for the CURLOPT_CAINFO option. This is required to be an
; absolute path.
;curl.cainfo =

[openssl]
; The location of a Certificate Authority (CA) file on the local filesystem
; to use when verifying the identity of SSL/TLS peers. Most users should
; not specify a value for this directive as PHP will attempt to use the
; OS-managed cert stores in its absence. If specified, this value may still
; be overridden on a per-stream basis via the "cafile" SSL stream context
; option.
;openssl.cafile=

; If openssl.cafile is not specified or if the CA file is not found, the
; directory pointed to by openssl.capath is searched for a suitable
; certificate. This value must be a correctly hashed certificate directory.
; Most users should not specify a value for this directive as PHP will
; attempt to use the OS-managed cert stores in its absence. If specified,
; this value may still be overridden on a per-stream basis via the "capath"
; SSL stream context option.
;openssl.capath=

To make cURL work with digital certificates is sufficient to simply set the curl.cainfo parameter:

[curl]
; A default value for the CURLOPT_CAINFO option. This is required to be an
; absolute path.
curl.cainfo = /usr/local/etc/openssl/certs/cacert.pem

Save the php.ini file and restart Apache. Try again and all should work well.

Magento: SSL certificate problem: unable to get local issuer certificate

Obviously, to solve the “SSL certificate problem: unable to get local issuer certificate” error in Magento when trying to connect to MagentoConnect the option we should choose is the second: set the cacert.pem file path directly in the php.ini.

Where to download a `cacert.pem` file

There isn’t an official cacert.pem, so we have to use the most accredited one, that is the one compiled by Mozilla and that can be downoaded from http://curl.haxx.se/ca/cacert.pem

If you like, here you’ll find other useful php.ini settings for local web development.

Remember to “Make. Ideas. Happen.”.

I wish you flocking users, see you soon!

L'articolo Magento 2: SSL certificate problem: unable to get local issuer certificate (cURL problem) proviene da ÐΞV Experiences by Serendipity HQ.

How to use LiipImagineBundle to manage thumbnails through Amazon S3

Adamo Crespi — Mon, 25 Jan 2016 16:45:35 +0000

Configure LiipImagineBundle to create thumbnails of images stored on Amazon S3 and save a cache version of them again on S3 may be really painful.

In this post I’ll try to guide you step by step in the configuration process, to have a full configured data flow to create thumbnails stored on Amazon S3 using LiipImagineBundle.

Which is our goal

Our goal is to configure the following data flow:

Get a source image file from AWS S3 (from the [bucket_name]/images folder);
Manipulate it in some ways with LiipImagineFilters (for example, the thumbnail filter);
Save the manipulated image again to AWS S3 (into the bucket_name/cache folder).

As the configuration of the full data flow could be difficult and prone to errors, we will use a simple configuration to reduce to minimum possibilities of issues.

So we will try to get an already uploaded image from the images folder in our bucket on S3 (hardcoding the path in the Twig template), resize it applying custom filters test_medium_thumb and test_small_thumb (that we will configure), then we will save again the resized image in the cache folder into same bucket on S3.

Before starting with the real configuration of the LiipImagineBundle we have to prepare our test bucket and our test route to simplify things.

We will, in fact, create a dedicate action in a controller to test in the tiniest way our filters. This will help us to test in a simpler manner the funcionalities of LiipImagineBundle.

Install the required bundles: `LiipImagineBundle` and `KnpGaufretteBundle`

To make all things work we need to install and activate, besides LiipImagineBundle, also KnpGaufretteBundle.

So, install and activate LiipImagineBundle and install and activate KnpGaufretteBundle.

Gaufrette is a PHP5 library that provides a filesystem abstraction layer.

Gaufrette is required to make LiipImagineBundle to get and save files from and to Amazon S3 buckets.

For the moment believe me and install the bundles: continuing in reading of this post you’ll better understand why we need also KnpGaufretteBundle.

Prepare our app for the test

Now that we have installed and activated our required bundles, lets prepare our app to test the data flow:

Create in your bucket the folder images;
Into this images folder upload a big image (I used this);
Create a cache folder in your bucket;
Create a DefaultController::TestThumbAction in your DefaultController (or in another controller: this action will be used to just render the test image and its thumbnail) that return the path of the just MANUALLY uploaded test image:

    /**
     * @Route("/test_thumb", name="thumb")
     * @Template()
     */
    public function testThumbAction()
    {
        $image = 'https://s3-'.
            $this->getParameter('amazon.s3.region').
            '.amazonaws.com/'.
            $this->getParameter('amazon.s3.bucket').
            '/images/symfony2.png';

        return [
            'image' => $image
        ];
    }

Create the template for the DefaultController::TestThumbnailAction:

<h1>Test resized with <code>test_small_thumb</code></h1>
{# NOTE THAT FOR THE MOMENT THE THUMBNAIL IS COMMENTED #}
{# <img src="{{ image | imagine_filter('test_small_thumb') }}" alt="" /> #}

<h1>Test resized with <code>test_medium_thumb</code></h1>
{# NOTE THAT FOR THE MOMENT THE THUMBNAIL IS COMMENTED #}
{# <img src="{{ image | imagine_filter('test_medium_thumb') }}" alt="" /> #}

<h1>Original Image</h1>
<img src="{{ image }}" alt="" />

Now go to http://127.0.0.1:8000/test_thumb and check that the test image is loaded correctly.

If the test image is not loaded, may be you have to set it as public on AWS S3 (select the image and in the Actions dropdown select Make Public).

Once you can see the test image in its original size, remove comments from the image with test_small_thumb filter applied (only from this for the moment!) and reload the page.

You’ll see this Twig_Error_Runtime exception

An exception has been thrown during the rendering of a template (“Could not find configuration for a filter: test_small_thumb”) in src/AppBundle/Resources/views/Default/testThumb.html.twig at line 2.

500 Internal Server Error – Twig_Error_Runtime

1 linked Exception: NonExistingFilterException »

Ok, we are now ready to start configuring LiipImagineBundle to create our thumbnails of images stored on Amazon S3 and save the cached thumbnail again on Amazon S3. Lets start!

Understand the dataflow of `LiipImagineBundle`

The dataflow followed by LiipImagineBundle is described in the documentation. I suggest you to read it to fully understand what we need to make the thumbnails being generated and saved to S3.

The short story is this:

STEP 1: LiipImagineBundle retrieve the original image from S3 using a DataLoader. Our DataLoader is the StreamLoader (that uses Gaufrette to work).
STEP 2: The source image is then processed with filters. The filters we will configure are test_medium_thumb and test_small_thumb.
STEP 3: The resulting image file is then saved (cached) to S3 using a CacheResolver. The CacheResolver we’ll use is the AwsS3Resolver.

STEP 1: Get the image source file from AWS S3 with Gaufrette

So, as told, the first step is to get the source image from the images folder in our bucket on AWS S3.

To do this, LiipImagnineBundle requires a DataLoader.

LiipImagnineBundle comes with some bundled DataLoaders. The default DataLoader is the FilesystemLoader as described also in the Configuration chapter of the documentation.

To load images from Amazon S3, we need another DataLoader, the StreamLoader:

The Liip\ImagineBundle\Binary\Loader\StreamLoader allows to read images from any stream (http, ftp, and others…) registered thus allowing you to serve your images from literally anywhere.
LiipImagineBundle Documentation, Stream Loader

The easier way to use this loader is configuring it to use Gaufrette.

So we have to first configure Gaufrette and then use it with the LiipImagineBundle‘StreamLoader.

Understand how Gaufrette works

At this point Gaufrette should be already installed as you should have already installed and activated KnpGaufretteBundle.

But, before continuing, just a little bit of theory to understand how Gaufrette works.

As told before,

Gaufrette is a PHP 5.3+ library providing a filesystem abstraction layer. This abstraction layer allows you to develop applications without needing to know where all their media files will be stored or how.

In other words, you interact with Gaufrette’s API that emulates actions you can take on a filesystem to manage files, so, when you decide to switch from the local filesystem to a cloud environment, you have to simply change some parameters in the configuration. Clean, simple and easy!

If you like, read some more details on the Knp Labs’ blog.

Anyway, to make your application “storage agnostic”, Gaufrette uses two main concepts: Adapters and Filesystems.

Your application will ever communicate with a Filesystem object. An Filesystem object uses a Adapter to communicate with the “storage” and perform the real actions on it.

Here the details about this simple process.

Introducing Gaufrette `StreamWrapper`

To make LiipImagineBundle StreamLoader work with images we need to understand another concept used by Gaufrette, the last one: StreamWrapper.

As we don’t simply want to interact with the filesystem, but we want also to manipulate images, we need a StreamWrapper that we’ll then use as DataLoader in LiipImagineBundle.

A StreamWrapper permits to register a new stream wrapper so we’ll can get the contents of the image and use them to create a new image.

Configuring `KnpGaufretteBundle` to work with AWS S3

So, after the theory, finally the practice! Lets configure KnpGaufretteBundle.

So, we need to configure those three things:

An Adapter: we’ll use the AwsS3Adapter
A Filesystem that uses the AwsS3Adapter: we’ll call it filesystem_aws_S3_images
A StreamWrapper (we don’t have to do anything more than writing a directive in our config.yml file to configure it)

Configure the AwsS3Adapter for `KnpGaufretteBundle`

The AwsS3Adapter needs the credentials to access the bucket on Amazon AWS S3, so, we first configure these credentials, then we’ll configure the S3Client (that uses the credentials):

shq.amazon.s3Credentials:
    class: Aws\Credentials\Credentials
    arguments: ["%amazon.s3.key%", "%amazon.s3.secret%"]

shq.amazon.s3:
    class: Aws\S3\S3Client
    arguments:
        - version: %amazon.s3.version%
          region: %amazon.s3.region%
          credentials: "@shq.amazon.s3Credentials"

Now, we can configure the AwsS3Adapter, the Filesystem and the StreamWrapper:

# config.yml
# KnpGaufretteBundle Configuration
knp_gaufrette:
    adapters:
        adapter_aws_s3_images:
            aws_s3:
                service_id: "shq.amazon.s3" # ! ! ! Without @ ! ! !
                bucket_name: "%amazon.s3.bucket%"
                options:
                    directory: 'images'
                    create: true
    filesystems:
        filesystem_aws_s3_images:
            adapter: adapter_aws_s3_images
    stream_wrapper: ~

As told in the documentation,

The stream_wrapper settings allow you to register filesystems with a specified domain and then use as a stream wrapper anywhere in your code like: gaufrette://domain/file.txt
GaufretteBundle Documentation, Stream Wrappers

Testing that `filesystem_aws_s3_images` works

Now that we have configured our Adapter and our Filesystem, we should test that it really works as expected.

Once configured, Filesystems are accessibile from the container, so your DefaultController::testThumbAction() write the following code:

    /**
     * @Route("/test_thumb", name="thumb")
     * @Template()
     */
    public function testThumbAction()
    {
        $filesystem = $this->get('knp_gaufrette.filesystem_map')->get('filesystem_aws_s3_images');
        $file = $filesystem->get('symfony2.png');

        /*
         * Note the use of the dump() function.
         * If you don't have the VarDumperComponent installed, use var_dump().
         * @see http://symfony.com/doc/current/components/var_dumper/introduction.html
         */
        dump($file);die;

        $image = 'https://s3-'.

        ...
    }

If the dump() shows you a Gaufrette\File object, then… CONGRATULATIONS! Gaufrette is correctly configured!

As we know that Gaufrette works, we can now use it with LiipImgineBundle!

Using Gaufrette as `DataLoader` for `LiipImagineBundle`

As told, we need a DataLoader to make LiipImagineBundle get the real content of the file so it can create a thumbnails.

Now that we have a working Gaufrette, we need to use it to configure the StreamLoader.

We have to alternatives to configure a DataLoader: using the factory and using a defined service.

We’ll use the factory method, so we have all configuration in one place (I find this more comfortable, but you can use the method you like).

So, to define the DataLoader using the factory method, put this in your config.yml file:

liip_imagine:
    loaders:
        loader_aws_s3_images:
            stream:
                # This refers to knp_gaufrette filesystems configuration
                wrapper: gaufrette://filesystem_aws_s3_images/

Perfect: we have done. Now we have to configure custom filters.

STEP 2: Define filters to resize images and create the thmbnails

Now that we can get images from AWS S3, it’s time to define our custom filters to manipulate them.

We will create two filters: test_medium_thumb and test_small_thumb.

To define filters we have to edit the config.yml file:

liip_imagine:
    loaders:
...
    filter_sets:
        test_medium_thumb:
            data_loader: loader_aws_s3_images
            # We don't yet have a cache resolver configured
            cache: cache_resolver_aws_s3
            quality: 75
            filters:
                thumbnail: { size: [400, 400], mode: outbound }
        test_small_thumb:
            data_loader: loader_aws_s3_images
            # We don't yet have a cache resolver configured
            cache: cache_resolver_aws_s3
            quality: 75
            filters:
                thumbnail: { size: [100, 100], mode: outbound }

Done! Really simple, doesn’t it?

But, as you can see, we have the directive cache in each of the two defined filters: these directives use the CacheResolver called cache_resolver_aws_s3.

So, as it may be yet clear, we have to define the cache_resolver_aws_s3. We are fastly moving toward the STEP 3!

STEP 3: Define the `CacheResolver` `AwsS3Resolver`

As we want to save our generated thumbnails to AWS S£, we need to use the CacheResolver AwsS3Resolver.

In this case, too, we can decide to define our resolver as a service or using the factory.

In this case, too, I’ll use the factory, but you can choose to use the method you like.

So, in the config.yml file:

liip_imagine:
    loaders:
        ...
    resolvers:
       cache_resolver_aws_s3:
          aws_s3:
              client_config:
                  credentials:
                      key: %amazon.s3.key%
                      secret: %amazon.s3.secret%
                  region: %amazon.s3.region%
                  version: %amazon.s3.version%
              get_options:
                  Scheme: 'https'
              put_options:
                  CacheControl: 'max-age=86400'
    filter_sets:
        ...

Test that all works well

And now that we have all configured, it’s time to test!

Edit `DefaultController::TestThumbAction`

First: change the code into the action:

    /**
     * @Route("/test_thumb", name="thumb")
     * @Template()
     */
    public function testThumbAction()
    {
        //$filesystem = $this->get('knp_gaufrette.filesystem_map')->get('filesystem_aws_s3_images');
        //$file = $filesystem->get('symfony2.png');

        /*
         * Note the use of the dump() function.
         * If you don't have the VarDumperComponent installed, use var_dump().
         * @see http://symfony.com/doc/current/components/var_dumper/introduction.html
         */
        //dump($file);die;

        /*$image = 'https://s3-'.
            $this->getParameter('amazon.s3.region').
            '.amazonaws.com/'.
            $this->getParameter('amazon.s3.bucket').
            '/images/symfony2.png';*/

        $image = 'symfony2.png';

        return [
            'image' => $image
        ];
    }

Activate filters in the template

Now activate the filters in the template removing comments:

<h1>Test resized with <code>test_small_thumb</code></h1>
<img src="{{ image | imagine_filter('test_small_thumb') }}" alt="" />

<h1>Test resized with <code>test_medium_thumb</code></h1>
<img src="{{ image | imagine_filter('test_medium_thumb') }}" alt="" />

<h1>Original Image</h1>
<img src="{{ image }}" alt="" />

Test!

Reload your page at http://127.0.0.1:8000/test_thumb and, if you’ve done well all the steps, you’ll see something like this:

Obviously, “Original image” is no longer visible as the path isn’t correct (we commented it in the controller!).

Done: now you have LiipImagineBundle up and running. You can now save your images to Amazon AWS S3, get them and resize to thumbnail sizes and save them again on AWS S3!

Some things to note: what happened behind the scenes

The first thing you should note is the URLs of the thumbnails.

The first time you load the page the resulting URLs are those:

<h1>Test resized with <code>test_small_thumb</code></h1>
<img src="http://127.0.0.1:8000/media/cache/resolve/test_small_thumb/symfony2.png" alt="" />

<h1>Test resized with <code>test_medium_thumb</code></h1>
<img src="http://127.0.0.1:8000/media/cache/resolve/test_medium_thumb/symfony2.png" alt="" />

<h1>Original Image</h1>
<img src="symfony2.png" alt="" />

As you can note, they are in this form: media/cache/resolve/[filter_name]/.

Try to reload the page.

The resulting URLs, this time, are those:

<h1>Test resized with <code>test_small_thumb</code></h1>
<img src="https://s3-eu-west-1.amazonaws.com/trustback-me-dev/test_small_thumb/symfony2.png" alt="" />

<h1>Test resized with <code>test_medium_thumb</code></h1>
<img src="https://s3-eu-west-1.amazonaws.com/trustback-me-dev/test_medium_thumb/symfony2.png" alt="" />

<h1>Original Image</h1>
<img src="symfony2.png" alt="" />

They refers to the public URL on AWS S3: https://s3-eu-west-1.amazonaws.com/[bucket_name]/[filter_name/.

In fact, the first time you load the images, if LiipImagineBundle cannot find a cached version, points to a controller defined in vendor/liip/imagine-bundle/Resources/config/routing.xml

The URL you see the first time points to the controller that applies the filters. Once the resulting image is cached, the next time LiipImagineBundle uses directly the URL of the bucket on AWS S3.

The second thing you should note, is that LiipImagineBundle automatically creates a folder for each filter in your bucket:

In practice, for each configured filter, the resulting images are saved into a dedicated folder.

Troubleshooting

And finally, some advices to debug in case of issues:

Keep attention to differences between UPPERCASE and lowercase letters: maybe you can write filesystem_aws_s3_images in one place and filesystem_aws_S3_images in another place. Spot the difference!
As the bundle is usually usedfrom inside a template, you’ll don’t get exceptions or errors. If something doesn’t work, go to the logs! There there are all the errors: find them looking for strings like “gaufrette” or “liip_imagine” or “liip/imagine-bundle” or “request.ERROR” or “php.DEBUG” and you’ll find all the info you need to understand what went wrong.

And finally, before to close this long post, remember that in the Issues section on GitHub there are a lot of useful information to better understand how to use the bundle.

Here are a couple of interesting issues:

Remember to “Make. Ideas. Happen.”.

I wish you flocking users, see you soon!

L'articolo How to use LiipImagineBundle to manage thumbnails through Amazon S3 proviene da ÐΞV Experiences by Serendipity HQ.

DEV Community: Adamo Crespi

How to debug ANY Symfony command simply passing `-x`

TL;DR

WARNING

The RunCommandInDebugModeEventListener listener

Moving Symfony files in the Docker container

Create the folder structure and the Dockerfile

The Docker Context: what is it, why we need to understand it and how to use it

Trying to copy only a single file test.txt

Moving Symfony’s files into the container

Making Apache interpret our PHP files

Introducing Docker images (and containers)

Configuring an Apache web server with PHP (for real!)

Conclusions and next steps

How to dockerize a Symfony application: Introduction and the web server

Introducing docker-compose

Creating our docker-compose.yaml file

Exploring the Apache container

Debugging MySQL and PHP in Docker

Further configuring MySQL

Further configuring PHP

Accessing the web server from the browser

Conclusions and next steps

How to set up the workflow to develop a Genestrap-based WordPress theme

Setting up git

Create a private git repository

Pushing your Genestrap-based WordPress theme to your private git repository

First push of the Genestrap-based WordPress theme to the git repository

Updating your custom Genestrap-based WordPress theme

Updating the private remote repository of your Genestrap-based WordPress theme

Updating the built Genestrap-based theme on your WordPress site

Updating the source code of your Genestrap-based theme FROM the main Genestrap public repository

1. checkout the last version of Genestrap from GitHub

2. merge the checked out branch in your master branch

3. Edit the version of your theme

4. and 5. push and update

Conclusions and next steps

How to create your custom WordPress theme with Genestrap

Something about Genesis

Something about Bootstrap

Why we built Genestrap

How to start developing your new WordPress theme based on Genstrap (Bootstrap + Genestrap)

STEP 1: Clone locally the Genestrap repository

STEP 2: Create the src/style.css file

STEP 3: Create the config.php

STEP 4: Create the functions.php

STEP 5: Create the src/scss/style.scss file

STEP 6: Create the src/scss/bootstrap.scss file

STEP 7: Install Composer packages

STEP 8: Install Javascript dependencies

STEP 9: Building the theme

STEP 10: Compress, upload and activate your Genestrap based theme

Conclusions and next steps

How to track unique users with behavioural analytics tools

What does it mean “Identify customers”

Why do you need to identify users one-by-one

Basic concepts of Device Identification

Basic concepts of cross-domain User identification

Understanding when a User is identified

The easiest user’s journey: no cross-domain nor cross-device tracking

What do we want to track in the Default User’s Journey

How to track the Default User’s Journey in Amplitude

How to track the Default User’s Journey in MixPanel

How to track the Default User’s Journey in Woopra

How to concretely track the users: code example

Step 1: Create the flash message after the user logs in

Step 2: Read the flash message in the Twig template

What happens in Amplitude when a user is identified

What happens in Woopra when a user is identified

What happens in MixPanel when a user is identified

Building the full default user’s journey in MixPanel

Handling the logout

How to handle the log out with Amplitude

How to handle the log out with MixPanel

How to handle the log out with Woopra

Concrete implementation of logout handling

What happens in Amplitude when you handle the logout of users

What happens in MixPanel when you handle the logout of users

What happens in Woopra when you handle the logout of users

Conclusions

The `RunCommandInDebugModeEventListener` listener

Create the folder structure and the `Dockerfile`

Trying to copy only a single file `test.txt`

Introducing `docker-compose`

Creating our `docker-compose.yaml` file

Setting up `git`

Create a private `git` repository

Pushing your Genestrap-based WordPress theme to your private `git` repository

First push of the Genestrap-based WordPress theme to the `git` repository

1. `checkout` the last version of Genestrap from GitHub

2. `merge` the checked out branch in your master branch

4. and 5. `push` and update

STEP 2: Create the `src/style.css` file

STEP 3: Create the `config.php`

STEP 4: Create the `functions.php`

STEP 5: Create the `src/scss/style.scss` file

STEP 6: Create the `src/scss/bootstrap.scss` file

The first (ugly and messy) approach: putting all images directly in the `templates/images` folder

The second (tiring) approach: requiring images through `require`

The third (ÐΞV’s way) approach: introducing `require.context`

Pass the `cacert.pem` file path directly to `cURL` when making the call

How to set the path to the `cacert.pem` file path in the `php.ini`

Where to download a `cacert.pem` file

Install the required bundles: `LiipImagineBundle` and `KnpGaufretteBundle`

Understand the dataflow of `LiipImagineBundle`

Introducing Gaufrette `StreamWrapper`

Configuring `KnpGaufretteBundle` to work with AWS S3

Configure the AwsS3Adapter for `KnpGaufretteBundle`

Testing that `filesystem_aws_s3_images` works

Using Gaufrette as `DataLoader` for `LiipImagineBundle`