DEV Community: David Jiménez

Reaching consensus and empowering discussions in your technical team

David Jiménez — Mon, 07 Feb 2022 07:00:02 +0000

Technical discussions can (and will) prove challenging and sometimes blocking. Surveying the team is enriching though, and helps with the spread of knowledge, awareness of relevant topics and speeds up decision making.

Planning Poker can serve as a tool to poll the team for certain topics, and get to know their opinions quickly while empowering discussions.

Here’s a screenshot of an example of what such a “survey poker” session could be:

Several topics could be polled with this method. The cards used for such sessions are pretty self-explanatory:

Yay: “I agree with this topic”
Nay: “I disagree with this topic”
RFI: “Request for Information. I need more information about this topic to make an informed decision“
STC: “Show me The Code“. A variant of “RFI”, you’d like to see code to make an informed decision.
POC: “Proof of Concept”. I like the topic, let’s put it to the test in our project or in a sample one to make a better informed decision.
?: “I have no strong feelings one way or the other/ I don’t care“
Pass: Equal to ?

Upon revealing, team members can debate on why they chose a certain card and bring those points/concerns to the rest of the team. Afterwards team members can change their vote, should they will.

This techniques works even on big teams. Since then you can pass the results to the management team to help them on their decision making.

Happy voting!

Trivia

This technique is homonymous to the real world referendums.

How the Colemak layout distribution and a proper keyboard doubled my coding performance

David Jiménez — Sat, 24 Jul 2021 07:27:28 +0000

Introduction

As a developer, I stay long hours in front of a monitor and a keyboard. It's only natural that I want to optimize my workspace and the tools I use. One of those tools is the keyboard. And after experiencing some shoulder and back pain over time, and after seeing how optimizing my usage of the keyboard reduced dramatically this pain trust me: you REALLY want to optimize your usage of the keyboard. And not only that: my typing speed and performance as a developer more than doubled from how I worked before.

The classical QWERTY keyboard layout, as well as the staggered, row distribution of the keys date from the XIX century. It was designed to make people type faster with the old typewriters, which had physical limitations. These limitations do not apply anymore with modern computers, yet the layout is still the most mainstream layout that we can find. And not because it's good, but because it's just been with us for so long.

In this article I want to present you the Colemak layout and several recommendations about your keyboard and your typing style. Everything I comment in this article lead me to the coder typer I am today. It doesn't mean that the same will work for you, but there is a good chance it will as well.

The Colemak layout

I'm not gonna go deep into the details of the layout (there are better pages suited for it, like DreymaR's Big Bag of Kbd tricks), but basically this layout offers several serious advantages over QWERTY:

Backspace on the Bloq Mayus key for easy reach of such a common key
The most frequent letters of English A, R, S, T, and N, E, I, O are placed in the home row, easy to reach. Other most commonly used keys are also placed in a location which is easy to reach by the fingers, while less used keys stay the same.

When the letters you press the most are at a closer reach distance your fingers move less, and let you stay in a calm, comfortable position more often, which, even if you don't realize it over time, helps tremendously against most common illnesses derived from long ours in front of a keyboard: carpal tunnel syndrome, back and shoulder pain, wrist pain, fatigue, repetitive strain injury (RSI)...

Some claim that Colemak is not the most efficient layout if you switch language. But as a developer the language you will encounter most of the time regardless of your mother languange is English, so that's not a real issue.

Install instructions

The default Colemak that is shipped with the system is good, but the mod DH variation is way, way better in terms of speed (and they only change a couple of keys). Follow the install instructions available on this link, and reach the "DOWNLOAD AND INSTALL" section: https://forum.colemak.com/topic/1438-dreymars-big-bag-of-keyboard-tricks-linuxxkb-files-included/

Tools of the trade

While switching to Colemak will definitely improve your typing performance, this improvement is crippled and slowed down without the right tools for the job. So besides changing the layout, you need as well to:

Master touchtyping
Have an appropriate keyboard

Touchtyping

An ELI5 definition I'd give:

Touchtyping is a technique you learn in which you leverage all the fingers of your two hands to type while, at the same time, you do not look at the keyboard with your eyes to search for a certain key.

(If you wish a more formal, and more comprehensive explanation, go check the Wikipedia page)

And it's a relatively easy skill that you can acquire in one week of intense practice. I gotta say that it's a skill that is a lot easier to acquire and consistently keep if you use an efficient keyboard layout (ahem... colemak) and a good keyboard as I mentioned previously.

On my learning journey I stumbled across a couple of sites that I can't recommend enough. I have tried many sites, and none of them compare to these two:

Keybr.com: https://www.keybr.com/
Typing.io: https://typing.io/ (suited for developers)

Start with KeyBr and, when you feel good and have acquired some decent speed, go to typing.io.

Your keyboard

Get yourself a columnar mechanical keyboard. Mainstream staggered keyboard which are found everywhere date from the age when the typewriter was the only piece of technology to write with a keyboard. Due to the physical limitations presented by the typebars clashing among each other that those old machines had. Again, these limitations do not apply to modern computers.

I use an Iris split keyboard like the one shown in the next image. It's extremely comfortable and improves my back posture due to being able to separate the shoulders (note that the key caps are blank, no need to have printed letters on the keys when you touchtype and the memory muscle triggers always).

My keyboard:

The next keyboard I have on my purchase list is the Preonic, which has the same key distribution, but it's more compact and ideal for travel and moving around the house.

In all seriousness, if you are a coder your keyboard is your main tool and you want to make sure you have the best tool available to offer the best code you can. Having a mechanical keyboard really improves not only your speed, but lowers the amount of mistakes you make when typing. In fact typing these blog posts is a pleasure with these keyboards.

Conclusions and final tips

Changing keyboard layouts as well as keyboard type is a massive change on your workflow; you will start slow, and making a lot of mistakes while your muscle memory builds up. But once you pick up speed, the results are impressive to say the least; It's such a quality of life improvement that I just can't describe with words precisely. I can tell that my programming speed has doubled and, if I spend less time thinking on how to type, I can spend much more time thinking to actually solve the problem at hand.

And as a bonus, writing in the right way becomes a pleasant, almost relaxing experience.

My final tips for someone wanting to get his feet wet would be to do, in order:

Install the Colemak keyboard distributions on your computer. If you are using a regular, staggered keyboard, use the Curl/Ergo mod variant, since it will put your fingers in a more natural position.
Start practicing and be patient. It takes about almost week of intense training with Keybr.com.
Once you are convinced that Colemak is for you, get a columnar or orthogonal mechanical keyboard, but don't pay more than 200$ for it if possible. You can get a second-hand one from reddit at the MechMarket subreddit. Make your own research though, but I recommend a Preonic, an Iris or a Lily58.

References

DreymaR's Big Bag of Kbd tricks
Colemak official site
Dvorak vs Colemak(Always good to read people that challenge the Colemak layout and compares it with another popular alternative distribution, Dvorak)

Getting started on profiling with python

David Jiménez — Fri, 23 Jul 2021 18:32:32 +0000

Motivation

Some time ago I contributed to OpenRA. OpenRA is an opensource Real-Time strategy game, and one of the most critical operations in this type of games is pathfinding; it needs to work, and it needs to run fast with lots of units moving at the same time each frame. Therefore, it needs to perform well to offer a nice gaming experience to the player.

Precise performance measurement is an activity present in almost every serious project, not necessarily gaming. For a software engineer the optimization part of algorithms and code in general is where this job gets a good name. And it's a task which important to get comfortable with it. In development there is a saying when implementing a feature: make it work, and then make it better.

My main daily driver at work is the python language. In this article I will show you some tools I use and some practical simple usages to get you started quickly on the topic of profiling. However, never miss the opportunity to access the official documentation of the tools exposed to fine tune the profiling to your own necessities!

The example I'm going to use for all the demonstrations

I'm gonna use as base the same example over all the profiling executions I'm going to show in this article. In short, this
is a unit test from my project jumper-py, and it computes, given a map, the shortest path between two nodes:

    def test_astar_intercardinal_search(self):
        map = [
            [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
            [0, 0, 0, 0, 0, 0, 0, 0, 1, 0],
            [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
            [0, 0, 0, 1, 0, 0, 0, 0, 0, 0],
            [0, 0, 1, 0, 0, 0, 0, 0, 2, 0],
            [0, 0, 1, 1, 1, 0, 0, 2, 0, 0],
            [0, 0, 0, 1, 1, 0, 2, 0, 0, 2],
            [0, 0, 0, 0, 1, 0, 0, 0, 0, 2],
            [0, 0, 0, 0, 0, 0, 0, 0, 0, 0],
            [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
        ]

        walkable = lambda v: v != 2

        grid = Grid(map)
        finder = Pathfinder(grid, astar.search, walkable, heuristic=cardinal_intercardinal)
        finder.annotate_grid()

        startx, starty = 0, 0
        endx, endy = 8, 8
        agent_size = 2

        X = 'x'
        marked_map = [
            [X, 0, 0, 0, 0, 0, 0, 0, 0, 0],
            [0, X, 0, 0, 0, 0, 0, 0, 1, 0],
            [0, 0, X, 0, 0, 0, 0, 0, 0, 0],
            [0, 0, 0, X, 0, 0, 0, 0, 0, 0],
            [0, 0, 1, 0, X, 0, 0, 0, 2, 0],
            [0, 0, 1, 1, X, 0, 0, 2, 0, 0],
            [0, 0, 0, 1, 1, X, 2, 0, 0, 2],
            [0, 0, 0, 0, 1, 0, X, X, 0, 2],
            [0, 0, 0, 0, 0, 0, 0, 0, X, 0],
            [0, 0, 0, 0, 0, 0, 0, 0, 0, 0]
        ]

        path = finder.get_path(startx, starty, endx, endy, agent_size)

        self._assert_the_resulting_path_nodes_should_be_like(marked_map, path)

A quick and dirty time measure

Sometimes less is better. A simple time measurement of a function might be more than enough for your needs.

The context manager:

from time import perf_counter, process_time

class timewith:
    def __init__(self, name="operation", cpu_time=False):
        self.name = name
        self.cpu_time = cpu_time
        self.time_func = perf_counter if not cpu_time else process_time
        self.start = self.time_func()

    @property
    def elapsed(self):
        return self.time_func() - self.start

    def checkpoint(self, name=""):
        time = 1000.0 * (self.time_func() - self.start)
        print(f"{self.name} {name} and took {time} ms".strip())

    def __enter__(self):
        return self

    def __exit__(self, type, value, traceback):
        self.checkpoint("finished")

Sample usage:

        [...]

        with timewith():  # The usage of the context manager
            path = finder.get_path(startx, starty, endx, endy, agent_size)

        [...]

Once you execute the test, it will output something like this:

operation finished and took 0.45445101568475366 ms

Quite simple, but as a measurement it can get the job done.

cProfile and Yappi decorators

Yappi and cProfile are what they call "deterministic profilers". The main difference between the two is that yappi supports multithreading execution and CPU measurement besides "Clock wall" measurement. These profilers are very detailed and provide a lot of information of the profiled code. Very useful.

The context manager:

import cProfile
from contextlib import contextmanager
from time import perf_counter, process_time

import yappi

@contextmanager
def profile(filename='results', engine='yappi', clock='wall', output_type='pstat'):
    if engine == 'yappi':
        filename = f'profiling/results/{filename}.prof'
        yappi.set_clock_type(clock)
        try:
            yappi.start(builtins=True, profile_threads=False)
            yield
        finally:
            yappi.stop()
            stats = yappi.get_func_stats()
            stats.print_all()
            stats.save(filename, type=output_type)
    else:
        profile = cProfile.Profile()
        try:
            profile.enable()
            yield
        finally:
            profile.disable()
            profile.print_stats()
            profile.dump_stats(f'profiling/results/{filename}.prof')

Adjust the variables to your own needs. Check the documentation for the respective projects.

Sample usage:

        [...]

        with profile():  # The usage of the context manager
            path = finder.get_path(startx, starty, endx, endy, agent_size)

        [...]

Afterwards, you can visualize the profiling results with a tool like kcachegrind or snakeviz.

Display of the results with kcachegrind:

pyprof2calltree -i profiling/results/results.prof -k

Display of the results with snakeviz:

snakeviz profiling/results/results.prof

Snakeviz is very visual, displaying each function in a different size in proportion to the time spent running it. Even then, I miss the display of more information in the icicle graph like the total number of calls each function performs to another function, like Kcachegrind does. You can refer for these values in the table that Snakeviz prints below the graphic though, but it would had been nice to include them. Maybe a function takes longer to execute not because it's slow per se, but because it's been called too many times unnecessarily, and that thing is not made explicit on the graph. A pity.

This is why I usually use both tools when optimizing code. Snakeviz for having a quick, general overview of the code execution and, if I need more information and call details, I switch to Kcachegrind. I still haven't found the "profiling visualizer to rule them all".

Worth mentioning too, you can use gprof2dot and xdot to have a quick glance at the call tree of your profile function:

gprof2dot -f pstats profiling/results/results.prof -o profiling/results/results.dot
xdot profiling/results/results.dot

PyInstrument

PyInstrument, compared to cProfile or Yappi, it's a lot easier to use and requires much less configuration.

With some small tweaking, we can take the code example from their documentation and write a decorator that we can easily place wherever we need to run this profiling.

The context manager:

from pyinstrument import Profiler


@contextmanager
def instrument(filename='results'):
    profiler = Profiler()
    directory = Path("profiling/results/")
    try:
        profiler.start()
        yield
    finally:
        profiler.stop()
        directory.mkdir(parents=True, exist_ok=True)
        path = os.path.join(directory, f'{filename}.html')
        with open(path, 'w') as fs:
            fs.write(profiler.output_html())

Sample usage (test from my project jumper-py):

        [...]

        with instrument():  # The usage of the context manager
            path = finder.get_path(startx, starty, endx, endy, agent_size)

        [...]

Screenshot of the result:

xdg-open profiling/results/results.html

As you can appreciate, it quickly gives you information on how much time it has spent on each function so you can quickly focus your attention on what is important.

Bear in mind though, pyInstrument does not replace other types of profiling, since it has a lower level of detail than cProfile or Yappi. But it's a really good starting point for your profiling.

Conclusions

Profiling is a crucial skill for any developer worth its salt. Maybe it's not something that you will use every day unless your business has high performance requirements, but it's something you can often rely on when "inexplicably" your application runs slower and you don't know why. And, eventually, you will cross paths which such problems, either in the form of algorithm optimization (like pathfinding algorithms) or in web development, helping you finding out why you're making so many database calls and give you hints on which ORM calls could be optimized.

References

https://uwpce-pythoncert.github.io/Py300/profiling.html#today-s-topics
https://pythonspeed.com/articles/beyond-cprofile/
https://blog.blackfire.io/profiling-101-for-python-developers-existing-profilers-3-6.html

Techniques to declare settings in a third party Django library

David Jiménez — Fri, 23 Jul 2021 17:59:07 +0000

Following up my article How to set up and run migrations in third-party Django packages, another question I faced when developing third-party Django libraries was: how can/should I implement the settings for the application in a decoupled and reusable manner, while allowing users of my library configure the settings?

The problem of using `django.conf.settings` directly in a third-party library

In Django, whenever you need to access a setting, unless you wrap them in a class, you have to import the settings module from django.conf.

For example:

from django.conf import settings
a = 10
b = a * settings.MULTIPLICATION_FACTOR

Can I you use this import inside your third party library? Sure, you could. However this poses three problems:

You need to document that the developer using your library MUST specify the MULTIPLICATION_FACTOR. Otherwise, codes like the previous example will crash.
You cannot define a default value for the variables of your library.
And again, unless you document your library very well, the settings your library uses are scattered everywhere, making it difficult to identify what I can customize.

A proposed better solution: the wrapper class

This is a solution I found when looking at the source code of the well known package Django Corsheaders. It is very, very simple. Just define a wrapper over the django.conf.settings, and specify a property per each setting that your library allows to customize.

Example:

from django.conf import settings

class Settings:
    @property
    def MULTIPLICATION_FACTOR(self):
        return getattr(settings, "MULTIPLICATION_FACTOR", 2)

settings = Settings()

Then we would modify the first code example like this:

from your_django_module.conf import settings
a = 10
b = a * settings.MULTIPLICATION_FACTOR

From that point forward, every time I would need a setting in my third-party library I would use that alternative import.

The Settings class and the MULTIPLICATION_FACTOR property are quite straightforward to explain: it checks if the user has defined an override for the settings called "MULTIPLICATION_FACTOR". If he/she did, take that value, otherwise, default to 2.

This simple trick decouples you furthermore from the Django infrastructure, which promotes cleaner and more reusable code.

Alternative technique

I'm not going to go in-depth with it, but I've seen a different technique used to achieve the same result in the SimpleJWT project that can be found in this file in the source code. Basically the technique relies on the settings_changed signal to establish default values for settings and update those settings in the context of the package. I'm not sure if there is a massive improvement on this approach over the previous one (maybe caching?), but if there is, feel free to comment.

How to set up and run migrations in third-party Django packages

David Jiménez — Thu, 22 Jul 2021 17:02:10 +0000

Creating a simple python package is a quite easy endeavor: create an empty project, create a module, put content inside, package it and done, you have a fully fledged package. However, creating a package that needs to integrate with Django is not that easy; you need to make sure that the models, migrations, default settings, etc... are in the correct place in order to flawlessly integrate without much pain.

This article will show you a way to run the Django migrations for your third party packages, decoupled from any external dependency. If any of you knows a simpler solution than the one exposed here, I will appreciate it so much!

DISCLAIMER: This guide assumes that some concepts and terms related to Django like migrations and settings are known by the reader.

Pre-requisites

Django is a dependency of any third party django package, so you'll have to declare it as such (with poetry, pipenv...)

The solution

You placed your Django Models in a models.py, but since you're implementing this third party library outside of the context of your monolith, there is no manage.py to be found. But you want to run your migrations!

Then you'll need:

Create a python package folder called migrations inside your django package. Once you successfully run the migrations, they will be placed here.
Create a scripts folder on the root of the project.
- Inside this folder, create a makemigrations.py file. Put this content inside it. With that you'll be able to run the makemigrations script:

import os
import sys

if __name__ == "__main__":
    os.environ.setdefault(
        "DJANGO_SETTINGS_MODULE", "scripts.mock_settings"
    )
    from django.core.management import execute_from_command_line

    args = sys.argv + ["makemigrations"]
    execute_from_command_line(args)

Add a mock_settings.py too. This file is necessary for the makemigrations command since any Django execution (and makemigrations is) requires a settings file. However, since we only need it to generate the migrations we can have a sample and dumb settings file just to do this operation. Put this content in the file:

import os

SECRET_KEY = 'Not a secret key at all, actually'
DEBUG = True

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": "postgres",
        "USER": "postgres",
        "PASSWORD": "example",
        "HOST": "localhost",
        "PORT": "60000",
    }
}

INSTALLED_APPS = [
    # Django apps necessary to run the admin site
    'django.contrib.auth',
    'django.contrib.admin',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.staticfiles',
    'django.contrib.messages',

    # And this app
    'your_django_lib',
]

STATIC_URL = '/static/'
MIDDLEWARE = [
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
]

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'OPTIONS': {
            'context_processors': [
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        }
    },
]

NOTE: Change this mock settings depending on your necessities. Instead of having to configure a real postgres database you could just change de DATABASE settings and configure a sqlite database. However sqlite does not support the likes of the JSONField, and migrations with JSONFields won't work.

If you need to spin up a real database just for the sake of migrations, you could use docker-compose. Use the following minimal setup:

version: '3.6'

services:
  postgres:
    image: postgres:9.5-alpine
    volumes:
      - postgres_data:/var/lib/postgresql/data
    ports:
      - 60000:5432
    expose:
      - "5432"
    restart: always
    environment:
      POSTGRES_PASSWORD: example

volumes:
  postgres_data:

In the end, the minimal structure for this to work would look something like this:

your-django-lib/
├─ your_django_lib/
│  ├─ migrations/
│  │  ├─ __init__.py
│  ├─ __init__.py
│  ├─ models.py
├─ scripts/
│  ├─ __init__.py
│  ├─ mock_settings.py
│  ├─ makemigrations.py
├─ .gitignore
├─ docker-compose.yml

Now you can run the makemigrations script. Depending on your virtual environment you'll have to execute the command in a different manner. With poetry for example you'd run it like this:

poetry run python scripts/makemigrations.py

Congratulations! It's been tough, but now you can run your migrations decoupled from any infrastructure or external Django dependencies! Now you should have your model migrations in the migrations folder!

Ease your development hurdles with these [M]ake recipes

David Jiménez — Wed, 21 Jul 2021 07:07:21 +0000

Introduction
Some Recipes/Tasks for your backend projects
1. Build the project
2. Clean up the workspace
3. Run the project
4. Prepare the local environment for development
5. Reset the database
6. Freeze and unfreeze the state of the database
7. Update non-local environments
8. File generation scripts
9. Run arbitrary command
10. Migrate database
11. Check code
12. Bump version
13. Run unit/acceptance tests
14. Open the profiling tool
Conclusions

Introduction

Make and other similar expert/automation systems like PyInvoke (I highly love this one over other build systems, even over Makefile), Rake, Cake, etc... avoid the necessity of having a bunch of scripts (bash, python...) to perform certain repetitive tasks. Even then, you often chain them in succession to achieve a certain goal (e.g. you know you have to run script_1.sh before script_2.sh, but that dependency is probably not made explicit in the project file structure). These dependencies are difficult to remember as they become more and more complex.

Or you might install a whole bunch of other tools for your project: poetry, gulp or docker just to name a few. Eventually recalling the arguments required to perform a certain action becomes difficult. But it's never difficult to recall the action you need to perform.

Dumb metaphor: It's easier to remember that you need an spanish omelette than remember that you need eggs, potatoes, onions and salt to build such delicious dish. Wouldn't it be easier if you had the power to imagine an omelette and PUFF!, it materializes in front of you, than building the omelette by yourself each and every time? That's where make and other derivatives can help you.

Here I present you with several suggested recipes that I use on a daily basis on the backend projects I have been working. The example are expressed in Makefile syntax, but the very idea can be applied to every build system.

Some Recipes/Tasks for your backend projects

Build the project (`make build`)

Pretty self explanatory. Every project of a reasonable size and maturity needs to be built, either by compilation, building the docker image, you name it. Therefore, the responsibility of this recipe should be to build/update the project and leave it in a runnable state. Optionally it could run migrations as well to keep the database schema up to date.

Make recipe:

build:
        # Put here the commands to compile, install dependencies, build docker images...

Invocation:

make build

Clean up the workspace (`make clean`)

This one is the inverse of build. Its responsibility is to clean up the docker images, volumes, generated files... created by the build command.

Make recipe:

clean:
        # Clean docker images, dist files, etc...

Invocation:

make clean

Run the project (`make run`)

This one runs the project. Prior to spinning up the application (running docker-compose up, executing the main script, etc.), it could prepare the environment to ensure that the project starts up successfully: invalidating caches, creating an AWS session, running database migrations, etc...

This is the command we tell the frontend team to run after building the application with make init to spin up the web server for theirs to run their HTTP requests.

Make recipe:

run:
        # Run your project here

stop:
        # In the same sense that there is a run, it might be interesting for you to have a stop command that kills the process/es that starts up the `run` command.

Invocation:

make run
make stop

Prepare the local environment for development (`make setup-dev`)

This one is intended for those that are going to do development over the project. It installs dependencies locally, set up pre-commit hooks, etc... basically setup and install anything required for local development.

Make recipe:

setup-dev:
        # Run pre-commit scripts, install dependencies locally...

Invocation:

make setup-dev

Reset the database (`make reset`)

Eventually and through your testing or the testing of your colleague developers, the database is going to be heavily polluted, and this command should reset the database to factory settings (destroying and creating the local database, re-running the initial data migrations, etc.)

Make recipe:

reset: down
        # Run db scripts to purge the DB/s, remove docker volumes...

Invocation:

make reset

Freeze and unfreeze the state of the database (`make freeze` and `make unfreeze`)

I came up with these after some people of my team suggested that it would be cool, from a testing perspective, if we could save a snapshot of the database and then restore it. It proved very useful.

Make recipe:

# Example with postgres, leveraging docker

freeze: down
        docker run -v my_postgres_vol:/volume -v /tmp:/backup --rm loomchild/volume-backup backup postgres

unfreeze: down
    docker run -v my_postgres_vol:/volume -v /tmp:/backup --rm loomchild/volume-backup restore postgres

Invocation:

make freeze
make unfreeze

Update non-local environments like DEV/TEST/UAT/PROD (`make update-remote`)

Often non-local environments have different characteristics than your machine (they might not run via docker, for example). Having a recipe to handle migrations and updates to those environments is very useful to avoid forgetting to run something.

Make recipe:

update-remote:
        # call here the commands that are intended to be called when you deploy to your remote environment: Database migrations, cache invalidations, scripts to fix something...

Invocation:

make update-remote

File generation scripts (`make generate-XXXX`)

Sometimes you might need to generate files in your code (documentation, protobuf code, etc...). These kind of commands are intended for such tasks.

Make recipe:

generate-docs:
        # call your doc generation tool (doxygen, sphynx...)

Invocation:

make generate-docs

Run arbitrary command (`make run-command`)

I work with Django (but this probably applies to other frameworks as well). And wrapping these commands into a recipe is often a good idea to keep the execution centralized.

Make recipe:

run-command:
        # e.g. python src/manage.py ($cmd)

Invocation:

make run-command cmd="XXXXX"

Migrate database (`make migrate`)

This recipe usually is a dependency of another of the previously mentioned recipes like init and up. As the name implies, it applies the migrations to the database.

Make recipe:

migrate:
        # Run here the command to execute the migrations

Invocation:

make check

Check code (`make check`)

Any serious project run some kind of code formatting tool to comply with your company coding guidelines and best practices. This command performs these checks on the code.

Make recipe:

check:
    # Example leveraging pre-commit
    pre-commit run --all-files

Invocation:

make check

Bump version (`make bump-version LEVEL=patch|minor|major`)

This recipe updates the version of your project. Each time I commit code the pre-commit hook calls this command with the patch argument, and each time I deploy to production I manually call it with the minor|major argument depending on the type of code change I'm conducting.

Make recipe:

bump-patch:
    # Example with bump2version
    poetry run bump2version $(LEVEL)
    git add _version.py
    git add .bumpversion.cfg

Invocation:

make bump-patch LEVEL=patch

Run unit/acceptance tests (`make run-tests` and `make run-acceptancetests`)

This recipe runs the tests of my application with some default parameters I like to establish. Sometimes I create other similar recipes to run these very tests but with some quirks, like generating coverage and result reports when running.

Make recipe:

run-tests:
        # Call here the command to execute your tests: py.test, NUnit... just make sure that the environment variables needed by your tests are loaded. This is why I like to run my tests with docker compose!

Invocation:

make run-tests

Open the profiling tool (`make open-profiling`)

You might have been profiling some code to find bottlenecks in your application and generated a report you'd like to visualize with tools like snakeviz or pyprof2calltree. This command will open the latest report generated, ready for visualization.

Make recipe:

open-profiling:
        # Example of command, adjust to your preferred visual tool
    pyprof2calltree -i profiling/results/$(FILE).prof -k

Invocation:

make open-profiling FILE=results

Conclusions

I exposed several recipes that I found useful when developing. Even though I gave some clues of the tooling you could potentially use, it's impossible to provide a detailed implementation since it wildly depends on the programming language you use, the environment, etc... However, what's really important about each recipe is the main objective it tries to achieve, and that's a fact (building the project, running tests...).

I'd like to hear from other fellow developers what other recipes you have come up with that could make our lives a little bit easier. If you have any to share please, feel free to contribute in the comments section!

Lessons learned after being a Tech Lead for three years

David Jiménez — Mon, 19 Jul 2021 18:48:18 +0000

A great man doesn't seek to lead: he's called to it.

- Duke Leto, Dune -

After being a technical lead for three years in my current company I decided it was a good time to retrospect on it and share my experience and conclusions. I hope you find my experiences helpful.

Disclaimer: Any statement here are solely based on my experiences and knowledge. I am by no means a Leadership Coach so some conclusions I provide might be clash with some established ideas and conceptions. And that's ok, challenging what's established is always enriching!

Introduction

We are developers. By default we are not the most charismatic people in the world (this is why we are developers after all, otherwise we might have picked another less solitary profession). However, I felt like working up and improving certain soft skills at the same pace as technical skills could boost my career. I found the chance to be on that role and I took it. And I never could have been more certain.

During these years I learnt a lot. I made mistakes. I messed up. And on occasions maybe, and just maybe, I was right.

I'm going to highlight those aspects of the tech lead role I found the most challenging and important, and most of them, surprisingly, are not especially tech-related; most of them refer to communication with your team and behavioral aspects.

The topics I'm going to pinpoint in this article are:

Communication and self-retrospection
You are a selfless enabler
"The teacher"
It's not about being bossy
Manners and feedback
It's good to be a point of technical reference, but it's not mandatory
Share ownership
Ultimately, it's about being an actual leader

Communication and self-retrospection

It looks pretty obvious, but communicating effectively is an essential part of being a tech lead. You will have to talk with a lot of people, from developers to clients, crossing product owners and testers in the path. You don't necessarily need to be always talkative, it is way more important to listen and understand foreign ideas while realizing that not everybody will understand your technical jargon.

When communicating with your team, sometimes you might be a grumpy furious monkey without being aware of it. This is why, from time to time, you should sit down and look at yourself from the outside in order to find gaps in your communication skills or behavior. Even when writing these lines I'm just realizing some communication mistakes I did several days ago which I was completely oblivious.

You are a selfless enabler

Tech Lead is a role meant to be selfless. Yes, we are all busy with our own share of work. But after all your teammates and you navigate in the same direction inside the context of the project. I like to think about it in an epic manner where you are a Spartan hoplite carrying a spear and a shield, but your shield is not meant to protect you, but your comrade at the left.

Is a fellow developer blocked and you can help? Go help him ASAP and, if you cannot do it right now for whatever reason, kindly tell him/her to meet up at a specific, later point in time to tackle the issue (note the word specific) or delegate the responsibility to another team member.

Neither you should hoard knowledge. I remember a debate with a former colleague from another company where he defended that, if you are good, it's better to keep your knowledge and best practices for yourself, with the statement that most people he shared his knowledge with were ungrateful and took advantage of that knowledge to step over him. Clearly a resentment from a past experience that sometimes gives you a lesson learned that, when you ponder about it with a good heart you think to yourself "that's a sad way to think about it".

Even if you had any experience like that, I prefer to "flow in the good, selfless direction". Share your knowledge and eagerness to help by default, and if people are ungrateful and no one appreciates it, either you should reflect on how you are providing this help or maybe, just maybe, it could be time to move on to another job/place (yes, there are workplaces which are utter shitholes).

"The teacher"

This follows up a little bit the last statements of the previous section. Another of the roles I have at my current company is that of a "Technical Coach", bringing the more junior employees or new recruits up to speed on the project regarding the architecture of the software solution, best practices, coding techniques, etc.

When I was at university, I wanted to be a professor there, teaching Software Engineering to students, sharing my knowledge while researching something awesome. I just loved that idea, and I studied a master oriented towards a Ph.D. on the area of Software Engineering.

So, when I receive a question from one of my fellow dev mates, this part of me triggers and I just fire the cannon of knowledge and experience on the topic relentlessly, even when unrequested. Not all people are equal though, and while some will appreciate your wisdom, others will frown upon that.

You do not have to always be "the teacher". It's a bit like the classical advice you are given when flirting in your early 20's: do not go all out unless you are requested so and keep a juicy bait on the hook.

Even then, I have a personal conflict over this. I like the TV show Avatar: The last airbender, and one of my favorite characters is Uncle Iroh, the uncle of the main antagonist of the show. He spreads wisdom everywhere, even if his own nephew doesn't want it. Still, I love Iroh at heart. I would love any reader to share his opinionated view on this.

It's not about being bossy

The role is not meant to exercise authority and supremacy over others. I like flat hierarchical structures, since it promotes the feeling of ownership of the solution and the flowing of ideas from all directions in a team. Even though formally in your contract you are the technical lead, you are still one more developer and you must consider your fellow teammates as an equal.

Yes, as the tech lead you might be the only technical guy from your team invited to certain meetings for the sake of keeping noise under control. But that doesn't mean that you belong to the Bilderberg Club. Try to act as a voice and speaker for your team, not yourself.

A subtle, yet significant change I noted and changed: When introducing myself to either new job candidates at technical interviews or clients, I started stating my role as the Technical Lead of the development team. Recently, for the sake of that flat hierarchy inside the team, I changed that and started introducing myself simply as a software engineer or backend developer. It made me realize it's just not necessary to boast yourself at the slightest opportunity (unless by protocol they need you to introduce yourself as such)

This brought me to the following conclusion: Humility is a good quality for a leader. Your contributions to the conversations, your communication skills and your confidence will eventually give a clue to the rest of the team who is the leader.

As I was writing this though, a thought clashed my mind. In this era of big CEOs on big companies or big personalities where everybody thinks of them as true leaders, I perceive most of them are everything but humble. Just one more example that sometimes statements and lessons learned depend on perception.

Manners and feedback

Yes, you and your teammates are going to screw up. Someone will not follow gitflow to the line and have a polluted remote git repository. Some code will make you pour blood from your eyes when reviewing a pull request.

There are people, however, willing to improve and listen to feedback. If you find those, kindly address them and provide them with constructive feedback. People will appreciate it. Just remember this quote from Lincoln when giving "constructive feedback": "A drop of honey catches more flies than a gallon of gall".

AND REMEMBER, the same works the other way around. Feedback is a gift, a very difficult gift someone gives to you (it's always way easier to stay silent and let people fail). Take the time to appreciate it if you receive some.

It's good to be a point of technical reference, but it's not mandatory

There are a number of reasons you might have obtained the role of tech lead of a team: it may have been your technical prowess, your communication skills, a mix of both...

Remember though that you are not a walking encyclopedia: you do not know everything, nor you should force yourself to attain such state, nor you should look pedantic. It's ok to admit you don't know something. If a teammate is blocked though and you are required to solve a certain problem do what any other developer worth their salt would do: research, test and solve it.

Neither you need to be the role-model or that steadfast statue one-hundred percent of the time. We are humans after all: we laugh, we explain bad jokes that make our mates either fall from the chair or frown upon us, we may have bad days. Showing weakness it's ok (and in fact, deeply, people appreciate it, it makes you more approachable than keeping the barrier up all the time), just make sure you always keep cool and don't let any storm you generate fall upon your teammates.

Share ownership

As a tech lead, the definition of the architecture and implementation of parts of the software system are your responsibility. But that doesn't mean that you should define it on your own. It's a good idea to build something considering the input of your team. It's easier for developers to build something if they contributed to the solution actively.

Do not be afraid to share ownership and collaborate with others to build a software solution. They are not paying you to be a hero. While making developers feel like the project is their child is cool, it's actually not yours unless you are the owner of the company. They are paying you to promote best development practices inside the team, coach junior members and new recruits, and offer a working, maintainable solution to the problem at hand.

Ultimately, it's about being an actual leader

Even though your contract might state the word "lead", it's not enough. A leader is someone that people follow by their own will. Remember that a leader that people follow by force is not a leader, but a mere boss.

And trust me, it's way, way, way more rewarding for everybody if people follow you because they want to. Not everybody is going to like you and that's ok, try though that statistically most people like you or retrospect about why it's not that way.

Conclusions

I'm a developer at the core, but having had the opportunity to develop a good share of soft skills in the shape of technical leadership has been one of the most rewarding and personality-bending steps I ever took on my career. It has changed me for good: it boosted my communication skills both inside and outside the technical team and gave me the chance to be able to share the joy of facing new technical challenges with my team.

Nevertheless, being a technical lead (in fact, any leading position in general, not necessarily technical) is deeply bound to your own personality. If your personality clashes with some of the aspects required for the role (you're selfish and have a boundless evil ambition, or you prioritize yourself over your teammates), you'll eventually show up your true colors. So TL;DR: don't be an asshole.

DEV Community: David Jiménez

Reaching consensus and empowering discussions in your technical team

Trivia

How the Colemak layout distribution and a proper keyboard doubled my coding performance

Table of contents:

Introduction

The Colemak layout

Install instructions

Tools of the trade

Touchtyping

Your keyboard

Conclusions and final tips

References

Getting started on profiling with python

Motivation

The example I'm going to use for all the demonstrations

A quick and dirty time measure

cProfile and Yappi decorators

PyInstrument

Conclusions

References

Techniques to declare settings in a third party Django library

The problem of using django.conf.settings directly in a third-party library

A proposed better solution: the wrapper class

Alternative technique

How to set up and run migrations in third-party Django packages

Pre-requisites

The solution

Ease your development hurdles with these [M]ake recipes

Table of contents

Introduction

Some Recipes/Tasks for your backend projects

Build the project (make build)

Clean up the workspace (make clean)

Run the project (make run)

Prepare the local environment for development (make setup-dev)

Reset the database (make reset)

Freeze and unfreeze the state of the database (make freeze and make unfreeze)

Update non-local environments like DEV/TEST/UAT/PROD (make update-remote)

File generation scripts (make generate-XXXX)

Run arbitrary command (make run-command)

Migrate database (make migrate)

Check code (make check)

Bump version (make bump-version LEVEL=patch|minor|major)

Run unit/acceptance tests (make run-tests and make run-acceptancetests)

Open the profiling tool (make open-profiling)

Conclusions

Lessons learned after being a Tech Lead for three years

Introduction

Communication and self-retrospection

You are a selfless enabler

"The teacher"

It's not about being bossy

Manners and feedback

It's good to be a point of technical reference, but it's not mandatory

Share ownership

Ultimately, it's about being an actual leader

Conclusions

The problem of using `django.conf.settings` directly in a third-party library

Build the project (`make build`)

Clean up the workspace (`make clean`)

Run the project (`make run`)

Prepare the local environment for development (`make setup-dev`)

Reset the database (`make reset`)

Freeze and unfreeze the state of the database (`make freeze` and `make unfreeze`)

Update non-local environments like DEV/TEST/UAT/PROD (`make update-remote`)

File generation scripts (`make generate-XXXX`)

Run arbitrary command (`make run-command`)

Migrate database (`make migrate`)

Check code (`make check`)

Bump version (`make bump-version LEVEL=patch|minor|major`)

Run unit/acceptance tests (`make run-tests` and `make run-acceptancetests`)

Open the profiling tool (`make open-profiling`)