DEV Community: Alejandro Riera

TIL: How to debug cron jobs

Alejandro Riera — Tue, 27 Feb 2018 11:05:28 +0000

Today I learned how to properly debug cron jobs, thanks to https://serverfault.com/a/85906.

They key is to recreate the correct environment in which your scripts get executed. I’m going to do this for root but you can do this with every user.

First: edit your crontab and add one line

crontab -e
* * * * * /usr/bin/env > /root/cron-env

Once it has had the chance to execute once you can remove it.

Second: let’s create a new script called run-as-cron that will run your scripts with the correct environment cron is running.

cd /root
touch run-as-cron
chmod +x run-as-cron
vim run-as-cron
#!/bin/bash
/usr/bin/env -i $(cat /root/cron-env) "$@"

Finally, debug your problematic script like this:

/root/run-as-cron /the/problematic/script --with arguments --and parameters

Django + webpack + Vue.js: setting up a new project that’s easy to develop and deploy (part 1)

Alejandro Riera — Tue, 26 Sep 2017 06:55:25 +0000

UPDATE: I recently created a working project at ariera/django-vue-template using the latest version of the vue-webpack template and bundling all the changes highlighted in this post in just this commit 8a5c552. You may also be interested in checking a vue project template that CharlesAracil created taking inspiration from this entry.

I recently started a new web project based on both Django and Vuejs. It took a good deal of reading half-related and/or outdated posts, tons of documentation, and a good deal of trial and error to get it right.

In this post we will cover the key points on setting up a nice and solid development environment that is consisten with production and hence easy to deploy. The specifics of how to deploy, final configuration touches and (automation) will be discussed in a future article.

This is the solution that I found, there may be better alternatives (which I’d be happy to read) and it is utimately written with the intention of helping others as well as my future-self :)

Continue reading at ariera.github.io

dev-toolbar a web developer’s best friend!

Alejandro Riera — Wed, 20 Sep 2017 16:05:28 +0000

The dev-toolbar is a simple concept we came up with many years ago. The idea is to provide developers with a toolbelt full of tricks (shortcuts, toggles, login-as…) hidden in the frontend. It is such an integral part of my workflow that it is usually the first thing I set up in any project.

In this entry we will see how to set it up in a Django web application. We will focus on the feature that everybody loves : the ability to login as any user in the database (great time saver). We call it impersonate , and this is how it looks like:

A simple select with all the users in your database. Click on one and you’ll be logged in as him/her.

Let’s get on with it!

1. Create a dev app

We will start by creating a new app that we will call dev. It will hold all of our impersonation logic, but is also a useful place to have to keep a style guide for developers, or some handy snippets to share accross the team, or maybe just to house a temporary experiment.

python manage.py startapp dev

and don’t forget to add it to your INSTALLED_APPS

# ./your_app/settings.py
INSTALLED_APPS = [
  'dev.apps.DevConfig',
  # ...
]

2. The view and urls

Here we will control if the current user is allowed to impersonate or not, and perform the actual impersonation. In this case we are allowing any superuser to login as any other user, but you may choose any other criteria, for example only in development environment, or only if DEBUG is True.

# ./dev/views.py
from django.shortcuts import redirect, render, get_object_or_404
from django.contrib.auth import get_user_model, login
from django.http import Http404

def impersonate(request, user_id):
  current_user = request.user
  if current_user.is_superuser:
    user = get_object_or_404(get_user_model(), pk=user_id)
    login(request, user)
    return redirect('/dev')
  else:
    raise Http404("Impersonate is only available for superusers")

# ./dev/urls.py
from django.conf.urls import url
from . import views

app_name = 'dev'
urlpatterns = [
    url(r'^$', views.index, name='index'),
    url(r'^impersonate/(?P<user_id>[0-9]+)/$', views.impersonate, name='impersonate'),
]

# ./your_app/urls.py
from django.conf.urls import include, url
from django.contrib import admin

urlpatterns = [
    url(r'^dev/', include('dev.urls')),
    url(r'^admin/', admin.site.urls),
]

3. Template tag

We want to reuse our toolbar across many templates so we will extract it into it’s own template tag, using Django’s inclusion-tags functionality.

For that you need to create this directory: ./dev/templatetags and an empty __init__.py file inside of it.

Inside our templatetags folder we will create the following file, which acts like the view of our new dev_toolbar templatetag. Notice 2 things:

First we are linking it to the template dev/toolbar.html that we will define later.
And second we use takes_context=True to be able to access to the current user. This will allow us to do some basic authorization (ie. only superusers are allowed to use this feature). This is redundant with the logic we defined before, but it’s good to have a safety net when you’re handling delicate data.

# ./dev/templatetags/dev_toolbar.py
from django.contrib.auth import get_user_model
from django import template

register = template.Library()

@register.inclusion_tag('dev/toolbar.html', takes_context=True)
def dev_toolbar(context):
  request = context['request']
  current_user = request.user
  if current_user.is_superuser:
    users = get_user_model().objects.all()
  else:
    users = []
  return {'users': users, 'current_user': current_user}

The template file contains its own css, javascript and html. Some simple positioning and styling and bit of css+js logic to make the bar more subtle.

We will render a select box with all the users in the database and a javascript will take care of calling the impersonate action whenever we choose another user other than ourselves. Two more handy links to /dev and /admin areas are included.

Last, but not least, I usually include a small close button, it comes in handy sometimes when doing CSS work, you just want to get rid of the bar.

<!-- ./dev/templates/dev/toolbar.html -->
{% if current_user.is_superuser %}
<style>
  #dev-toolbar{
    background-color: #ccc;
    position: fixed;
    bottom: 0;
    right: 0;
    padding: 5px;
  }
  #dev-toolbar.subtle {
    opacity: 0.2;
  }
</style>
<script>
  window.dev = {
    closeToolbar: function closeToolbar(){
      var toolbar = document.getElementById("dev-toolbar")
      toolbar.parentElement.removeChild(toolbar)
    },
    showToolbar: function showToolbar(){
      var toolbar = document.getElementById("dev-toolbar")
      toolbar.classList.remove("subtle");
    },
    impersonate: function impersonate() {
      var select = document.getElementById("dev-impersonate")
      window.location = select.value
    }
  }
</script>
<div id="dev-toolbar" class="subtle" onmouseover="dev.showToolbar()">
  <small><a href="/dev">dev</a></small>
  <small><a href="/admin">admin</a></small>

  {% if users %}
    <select id="dev-impersonate" onchange="dev.impersonate()">
      {% for user in users %}
         <option value="{% url 'dev:impersonate' user.id %}"
             {% if user == current_user %}selected="selected"{% endif %}>
             {{user.email}}
         </option>
      {% endfor %}
    </select>
  {% endif %}

  <button id="dev-close-toolbar" onclick="dev.closeToolbar()">X</button>
</div>
{% endif %}

4. Using it

Our feature is complete and ready to be used. Simply load it in any view and call it.

{% load dev_toolbar %}
{% dev_toolbar %}

5. Show it in the admin panel

We are going to override the default admin/index.html template and include our new dev_toolbar snippet. For that we need to create a generic ./templates directory at the root of the project and add it to TEMPLATES['DIRS'] in the settings of the project.

# ./your_app/settings.py
TEMPLATES = [
  {
    'DIRS': [os.path.join(BASE_DIR, 'templates')],
    # ...
  }
]

And now create a new file to override the default administration index tempalte.

<!-- ./templates/admin/index.html -->
    {% extends "admin/index.html" %}
    {% load dev_toolbar %}

    {% block sidebar %}
        {{block.super}}
        {% dev_toolbar %}
    {% endblock %}

References

This post was originally published on ariera.github.io

"How to use email as user identifier in Django 1.11"

Alejandro Riera — Wed, 20 Sep 2017 08:55:28 +0000

(This post was originally published on ariera.github.io)

I found several guides and tips on how to do this. They were all very inspiring and helpful but I had to figure some things on my own because none of them provided all I needed: a fully working admin panel that replicates all the functionality of the original one.

I won't stop to explain what every line is doing, in the end I'm just trying to save myself some trouble in the future, but I share it because it may be useful to someone else as well. At the moment of writing this I am using Django==1.11.5.

Before we start

Remember that you need to do this at the beginning of the life of your project, before running any migrations, so drop&create your DB if you are already that far. I would do this in my local machine with postgres:

pg_ctl -D /usr/local/var/postgres restart
dropdb my_app
createdb my_app

First create your new app

Since it this is gonna be very generic, project-wide app, I like to call it base. I may end up placing other basic models / views in here.

python manage.py startapp base

Configure your settings

./my_app/settings.py

# ./my_app/settings.py
AUTH_USER_MODEL = 'base.User'
INSTALLED_APPS = [
    'base.apps.BaseConfig',
    # ...
]

Configure your `models` and `admin`

This will be your models, highly inspired by the original Django source code.

./base/models.py

from django.db import models
from django.contrib.auth.models import AbstractBaseUser
from django.contrib.auth.models import BaseUserManager
from django.contrib.auth.models import PermissionsMixin
from django.utils.translation import ugettext_lazy as _

class MyUserManager(BaseUserManager):
    """
    A custom user manager to deal with emails as unique identifiers for auth
    instead of usernames. The default that's used is "UserManager"
    """
    def _create_user(self, email, password, **extra_fields):
        """
        Creates and saves a User with the given email and password.
        """
        if not email:
            raise ValueError('The Email must be set')
        email = self.normalize_email(email)
        user = self.model(email=email, **extra_fields)
        user.set_password(password)
        user.save()
        return user

    def create_superuser(self, email, password, **extra_fields):
        extra_fields.setdefault('is_staff', True)
        extra_fields.setdefault('is_superuser', True)
        extra_fields.setdefault('is_active', True)

        if extra_fields.get('is_staff') is not True:
            raise ValueError('Superuser must have is_staff=True.')
        if extra_fields.get('is_superuser') is not True:
            raise ValueError('Superuser must have is_superuser=True.')
        return self._create_user(email, password, **extra_fields)



class User(AbstractBaseUser, PermissionsMixin):
    email = models.EmailField(unique=True, null=True)
    is_staff = models.BooleanField(
        _('staff status'),
        default=False,
        help_text=_('Designates whether the user can log into this site.'),
    )
    is_active = models.BooleanField(
        _('active'),
        default=True,
        help_text=_(
            'Designates whether this user should be treated as active. '
            'Unselect this instead of deleting accounts.'
        ),
    )
    is_superuser = models.BooleanField(default=False, help_text='Designates that this user has all permissions without explicitly assigning them.', verbose_name='superuser status')
    created_at   = models.DateTimeField(auto_now_add=True)
    updated_at   = models.DateTimeField(auto_now=True)
    first_name   = models.CharField(blank=True, max_length=30, verbose_name='first name')
    last_name    = models.CharField(blank=True, max_length=30, verbose_name='last name')

    USERNAME_FIELD = 'email'
    objects = MyUserManager()

    def __str__(self):
        return self.email

    def get_full_name(self):
        return self.email

    def get_short_name(self):
        return self.email

And here is the admin file, also highly inspired by the original Django source code

./base/admin.py

from django import forms
from django.contrib import admin
from django.contrib.auth import password_validation
from django.contrib.auth.admin import UserAdmin as BaseUserAdmin
from django.contrib.auth.forms import ReadOnlyPasswordHashField
from django.contrib.auth.forms import UserChangeForm as BaseUserChangeForm
from django.utils.translation import gettext, gettext_lazy as _
from .models import User

class UserCreationForm(forms.ModelForm):
    """A form for creating new users. Includes all the required
    fields, plus a repeated password."""
    password1 = forms.CharField(
        label=_("Password"),
        strip=False,
        widget=forms.PasswordInput,
        help_text=password_validation.password_validators_help_text_html(),
    )
    password2 = forms.CharField(
        label=_("Password confirmation"),
        widget=forms.PasswordInput,
        strip=False,
        help_text=_("Enter the same password as before, for verification."),
    )


    class Meta:
        model = User
        fields = ('email', 'is_staff', 'is_active', 'first_name', 'last_name', )

    def clean_password2(self):
        # Check that the two password entries match
        password1 = self.cleaned_data.get("password1")
        password2 = self.cleaned_data.get("password2")
        if password1 and password2 and password1 != password2:
            raise forms.ValidationError("Passwords don't match")
        return password2

    def save(self, commit=True):
        # Save the provided password in hashed format
        user = super().save(commit=False)
        user.set_password(self.cleaned_data["password1"])
        if commit:
            user.save()
        return user

class UserChangeForm(BaseUserChangeForm):
    """
    Override as needed
    """

class UserAdmin(BaseUserAdmin):
    # The forms to add and change user instances
    form = UserChangeForm
    add_form = UserCreationForm

    # The fields to be used in displaying the User model.
    # These override the definitions on the base UserAdmin
    # that reference specific fields on auth.User.
    list_display = ('email', 'is_superuser', 'is_staff', 'is_active', 'first_name', 'last_name', 'created_at', 'updated_at', 'last_login')
    # list_filter = ('email',)

    readonly_fields=('created_at', 'updated_at',)
    fieldsets = (
        (None                , {'fields': ('email', 'password')}),
        (_('Personal info')  , {'fields': ('first_name', 'last_name',)}),
        (_('Permissions')    , {'fields': ('is_active', 'is_staff', 'is_superuser', 'groups', 'user_permissions')}),
        (_('Important dates'), {'fields': ('last_login', 'created_at', 'updated_at')}),
    )
    # add_fieldsets is not a standard ModelAdmin attribute. UserAdmin
    # overrides get_fieldsets to use this attribute when creating a user.
    add_fieldsets = (
        (None, {
            'classes': ('wide',),
            'fields': ('email', 'password1', 'password2')}
        ),
    )
    search_fields = ('email',)
    ordering = ('email',)
    # filter_horizontal = ()


admin.site.register(User, UserAdmin)

Apply changes

And lastly you need to generate your migrations and migrate

python manage.py makemigrations base
python manage.py migrate
python manage.py createsuperuser

References

Setting up GitLab’s Continuous Integration with Laravel, PostgreSQL and PHPUnit

Alejandro Riera — Thu, 09 Mar 2017 17:06:56 +0000

I recently set up a continuous integration process based on GitLab for a project based on Laravel and PostgreSQL. It took a while to get it right, so I am sharing here my final config.

Basic structure

The main GitLab CI configuration is held by a file at the root of your project that should be called .gitlab-ci.yml. From this file it is possible to call other shell scripts, but I have tried to hold up everything in just one place.

We will however manage a second file called .env.gitlab-ci that will hold all ENV variable configuration optios specific to GitLab CI.

Last but not least you will need a PHPUnit xml configuration file.

Step by step look into `.gitlab-ci.yml`

We’ll go line by line in the config file, but you can see the final result at the end of the article.

image: php:7.0

Here we select one of the available docker images with php already configured. You can find a complete list here: from https://hub.docker.com/r/\_/php/

services:
  - postgres:latest

From GitLab’s official documentation src:

The services keyword defines just another docker image that is run during your job and is linked to the docker image that the image keyword defines. This allows you to access the service image during build time.

It is very important to remark that if you add postgres as service to your application, the service container for PostgresSQL will be accessible under the hostname postgres. So, in order to access your database service you have to connect to the host named postgres instead of a socket or localhost. This will be important later when we configure our .env.gitlab-ci file.

variables:
  POSTGRES_DB: mydb-test
  POSTGRES_USER: runner
  POSTGRES_PASSWORD: ""

Here you may use all the values you want as long as they match with those of .env.gitlab-ci and your phpunit.xml.

before_script:

Again from the official documentation src

before_script is used to define the command that should be run before all jobs, including deploy jobs, but after the restoration of artifacts. This can be an array or a multi-line string.

Lets explore all these commands one by one.

- >
  set -xe
  && apt-get update -yqq
  && apt-get install -yqq
  git
  libicu-dev
  libpq-dev
  libzip-dev
  zlib1g-dev

Simply installing some dependencies

git is needed to later install composer
libicu-dev will be necessary for the internationalization php extension (intl)
libpq-dev are the header files for postgres
libzip-dev and zlib1g-dev are necessary for zip manipulation and needed to activate the corresponding php extension. Without them our composer install command will throw a lot of warnings and try to download each package twice slowing the whol process down for 2 or more minutes

- >
  docker-php-ext-install
  pdo_pgsql
  pgsql
  sockets
  intl
  zip

docker-php-ext-install is a handy script for installing php extensions provided by the official php docker image. I restricted myself to the bare minimun of extension necessary for my project to run. You may need more.

- curl -sS https://getcomposer.org/installer | php
- php composer.phar self-update
- php composer.phar install --no-progress --no-interaction

Install composer and use it to install all of our project dependencies.

- cp .env.gitlab-ci .env

Set up the correct ENV variables. We’ll look into this file in a moment.

- php artisan key:generate
- php artisan help config:clear
- php artisan route:clear
- php artisan migrate:refresh

Set application key, clear a couple of caches just in case and run the migrations.

test:app:
  script:
  - vendor/bin/phpunit --configuration phpunit-gitlabci.xml

And finally run our PHPUnit test suite using our phpunit-gitlabci.xml config file.

Small break

That was the most complicated file, the rest is a piece of cake, but you deserve a rest :P

A brief look into `.env.gitlab-ci`

The file itself is quite self explainatory, and you can see the final version at the end of the article. But I wanted to highlight the database configuration.

DB_CONNECTION=pgsql
DB_PORT=5432
DB_HOST=postgres
DB_DATABASE=mydb-test
DB_USERNAME=runner
DB_PASSWORD=

Notice how the values for DB_HOST, DB_DATABASE and DB_USERNAME are the same as those of inside of the variables key on the .gitlab-ci.yml configuration file.

A brief note on `phpunit-gitlabci.xml`

This file may be optional depending on your local configuration. If your normal phpunit.xml file doesn’t define any DB_DATABASE env variable you would not need this. But if it happens to define one, and has a different value from that that you chose for GitLab’s CI you have to remember to set it correctly:

<env name="DB_DATABASE" value="mydb-test"/>

Summary

GitLab’s continuous integration system can be a blessing once is properly configured, but it can get a bit confusing if you trail off of the usual path and all you hear is docker image configuration linking and you’ve never worked with it before. Hopefully you’ll find this small guide useful

.gitlab-ci.yml

image: php:7.0
services:
  - postgres:latest
variables:
  POSTGRES_DB: mydb-test
  POSTGRES_USER: runner
  POSTGRES_PASSWORD: ""

before_script:
- >
  set -xe
  && apt-get update -yqq
  && apt-get install -yqq
  git
  libicu-dev
  libpq-dev
  libzip-dev
  zlib1g-dev
- >
  docker-php-ext-install
  pdo_pgsql
  pgsql
  sockets
  intl
  zip
- curl -sS https://getcomposer.org/installer | php
- php composer.phar self-update
- php composer.phar install --no-progress --no-interaction
- cp .env.gitlab-ci .env
- php artisan key:generate
- php artisan help config:clear
- php artisan route:clear
- php artisan migrate:refresh

test:app:
  script:
  - vendor/bin/phpunit --configuration phpunit-gitlabci.xml

.env.gitlab-ci

APP_ENV=testing
APP_KEY=key
APP_DEBUG=true
APP_LOG_LEVEL=debug
APP_URL=http://localhost:8000

DB_CONNECTION=pgsql
DB_HOST=postgres
DB_PORT=5432
DB_DATABASE=mydb-test
DB_USERNAME=runner
DB_PASSWORD=

BROADCAST_DRIVER=log
CACHE_DRIVER=array
SESSION_DRIVER=array
QUEUE_DRIVER=sync
MAIL_DRIVER=log

This post was originally published on ariera.github.io

Using Postgre’s UUIDs in Laravel and Eloquent

Alejandro Riera — Wed, 01 Mar 2017 18:06:56 +0000

One of the things I love from postgres is the ability use UUIDs as primary keys. Tom Harrison Jr. wrote recently a summary worth reading about the pros and cons of this approach. For me the pros outweight the cons, but that’s something you may have to decide by your own and on a project basis.

However, if you decide to go for it and you are using Laravel you’ll be surprised to see that getting UUIDs to work is not as easy as the official documentation makes you think.

The following is a small cheatsheet for the things you have to take care of

Setting up the db

The first thing you need is to activate the UUID extension in postgres. Simply create a new migration that looks like this:

database/migrations/1900_01_01_000001_add_uuid_extension_to_postgresql.php

<?php
use Illuminate\Support\Facades\Schema;
use Illuminate\Database\Schema\Blueprint;
use Illuminate\Database\Migrations\Migration;
use Illuminate\Support\Facades\DB;

class AddUuidExtensionToPostgresql extends Migration
{

    public function up()
    {
        DB::statement('CREATE EXTENSION IF NOT EXISTS "uuid-ossp";');
    }

    public function down()
    {
        DB::statement('DROP EXTENSION IF EXISTS "uuid-ossp";');
    }
}

Migrations

Let’s say you want to have a users table. You probably want your ids to be automatically generated. Most of the tutorials I have found rely on beforeCreate callbacks on the model, at the applications level. I think it is best if the database takes care of this.

Unfortunately the nice Schema DSL doesn’t provide a way to do this, so we need to use plain SQL for it.

database/migrations/2017_03_01_000001_create_users.php

<?php
Schema::create('users', function (Blueprint $table) {
    $table->uuid('id');
    $table->primary('id');
    // ...
});
DB::statement('ALTER TABLE users ALTER COLUMN id SET DEFAULT uuid_generate_v4();');

Models

Now let’s create the corresponding User model.

App/User.php

<?php
namespace App;
use Illuminate\Database\Eloquent\Model;
class User extends Model
{
  protected $casts = [
    'id' => 'string'
  ];
  protected $primaryKey = "id";
}

Couple of caveats here. First you need to make sure that your primary key is typecasted as string , otherwise it will be typecasted to integer giving you all sorts of funny errors.

Second: in this example I am naming my primary key id, which is Eloquent’s default name. So overriding the $primaryKey in this particular example is not very interesting, but many people opt for naming their pks uuid, so pay attention to that if that’s your case.

Lastly, many comments online and tutorials indicate that the public attribute $incrementing should be set to false, arguing that UUIDs don’t need to be autoincremented. As much sense as that might make it managed to broke my applications. Disabling the incrementing property somehow nullifies the id attribute in newly created records , as the following code highlights.

App/User.php

<?php
class User extends Model
{
  public $incrementing = false;
  // ...
}

$user = new User;
$user->save();
$user->id; /* => null */

Summary

Having UUIDs as primary keys for your records can be tricky to do right on Laravel for the first time. Hopefully this tips will help you avoid the beginners gotchas that I fell on :D

This post was originally published on ariera.github.io

DEV Community: Alejandro Riera

TIL: How to debug cron jobs

Django + webpack + Vue.js: setting up a new project that’s easy to develop and deploy (part 1)

Continue reading at ariera.github.io

dev-toolbar a web developer’s best friend!

1. Create a dev app

2. The view and urls

3. Template tag

4. Using it

5. Show it in the admin panel

References

"How to use email as user identifier in Django 1.11"

Before we start

First create your new app

Configure your settings

Configure your models and admin

Apply changes

References

Setting up GitLab’s Continuous Integration with Laravel, PostgreSQL and PHPUnit

Basic structure

Step by step look into .gitlab-ci.yml

Small break

A brief look into .env.gitlab-ci

A brief note on phpunit-gitlabci.xml

Summary

Using Postgre’s UUIDs in Laravel and Eloquent

Setting up the db

Migrations

Models

Summary

Configure your `models` and `admin`

Step by step look into `.gitlab-ci.yml`

A brief look into `.env.gitlab-ci`

A brief note on `phpunit-gitlabci.xml`