DEV Community: Dominik Weber

Developing a reading list for programmers - Motivation

Dominik Weber — Mon, 22 Mar 2021 08:27:37 +0000

A few months ago, the reading list of my choice, Pocket, made a change that I didn't agree with. A minor change, but nonetheless extremely annoying.

That sparked a thought. I love working on side-projects, why not develop my own?

Following that decision, I started thinking of other features a reading list could have.

The ideas ranged from simple, like showing if an article was already read (in case it's added a second time), to complex, like keeping a backup of the website in case it goes down or the article is removed.

Grand vision(s)

After a few weeks the idea expanded. It doesn't have to be a reading list only. It could include any kind of information that can be linked to. Books, Podcasts, online courses, videos, lectures.

Such a product could become a central hub of information.

Not only listing, but organizing it as well. Features like showing the source of an article, who recommended the article, book or video, adding notes, following specific high-quality blogs, and many more crept into my mind.

I use a reading list because I often come across great content, but don't have the time or energy to consume it in that moment. So I add it to the list for later.

That's the idea of a reading list. But it could be so much more...

What if I didn't view it as only a list of information, but as a central place to manage information consumption.

The difference being that it doesn't just list, it also helps to retain the knowledge that these articles contain. In this area I'm still a little short in what I could do. Spaced repetition is one thing, but I'm sure there's a lot more possible.

As you can see, this little project evolved into something huge. Only in thought for now, it'll probably take years until I have realized that vision. But every great thing starts at the beginning.

Opportunity to tackle societal problems?

Not being short of visions, I found two more that are important to me and I might be able to do with this project.

The first is battling fake and false news. The difference I make for this article is that fake news are deliberately presented as true, while false news is simply a mistake.

For false news I already have an idea. Articles are often corrected, and usually there's then a correction section at the end. But very few people check back to see if there was a correction. So the feature could be to automatically watch changes on articles, and if a correction appears, show it.

I have no idea yet what to do against fake news though.

The second is spearheading a move to higher quality information.

In the past years, with the commodization of content publishing, a lot more high- and low-quality content became available, but the ratio shifted towards low-quality.

It also seems to me that searching for specific topics doesn't list the high quality information. What I usually get is listicles or short articles about the topic. Long, well-researched writups are rarely found.

I have an idea of how to tackle this, but it's not refined enough yet to share. I can only say that for it to happen I need users. Many users. Which means I'll create a valuable product first.

Development

Rome wasn't built in a day, and it'll be the same with my little project.

The first step is to build a basic reading list. Then I'll add more and more features to improve the product and serve the vision.

A first version is done already. I named it ReadingDragon and you can find it here: https://www.reading-dragon.com/

It is in alpha now. You can use the code 'alpha' to get 66% discount. Or 'free' if you don't share my vision, then you can try it out for free.

Since it's a side-project, and software development is my passion, I'll also share my progress on Twitter and via articles like these (which I'll also share on Twitter).

I'll blog about product evolvement and about technical aspects, like the architecture, testing strategy and so on.

Conclusion

The whole idea developed from a small and simple reading list to a really ambitious project. It is a bit intimidating, but I like the fact that I can potentially work on it for years to come.

Wish me luck! :)

Thoughts on Developer Autonomy

Dominik Weber — Fri, 06 Dec 2019 05:33:14 +0000

The level of autonomy developers get influences their learning speed, job satisfaction and how fast teams can deliver software. As a person seeking as much freedom as possible this is a topic dear to my heart.

Disclaimer: I write as if all of what I'm saying is true. But this is just my writing style. It's still only my current thoughts on this topic.

I have seen a few different levels of planning out there.

1) People are told what task to work on
2) People can choose tasks from a predefined list
3) People get the vision, discuss how to best fulfill it, and then execute on that
4) People get the vision and work on whatever they want to fulfill it
5) People work on whatever they want

I personally experienced the first 3, heard first hand about the 4th one. The last one is just there for completeness sake, I have actually never heard or read from anyone having such a job, but I guess they exist somewhere.

In general autonomy is lowest in the first one and goes up the further you go down the list. There's another axis impacting freedom though, task size.

If people are told exactly which task to work on (1), but they are huge, say 2 weeks of work, they are more autonomous than in a system where it's discussed how to best fulfill the vision (3), but a sync and evaluation happens every day.

As soon as a task is estimated, team leads and managers want to keep close to that to not mess up their plans for the team and projects. Going on a tangent to future-proof code which doubles the task size is often not desired.

With the estimated time being at least somewhat fixed, doing their own estimates gives engineers the ability to calculate in important adjacent activities (like code improvements) that others might overlook. If team leads do the estimate all on their own, this freedom is lost.

The worst in terms of freedom is a team where the team lead estimates tasks, these tasks are tiny and people are told exactly what to work on at which time. And to top it all off, it's even defined exactly how to do them. Micromanagement in its purest form.

Jokes aside, I didn't hear of any organization yet where it's that bad.

Impact on Developers

Particularly young developers profit enormously from increased levels of autonomy. It provides a wider area for mistakes. And the mistakes made are their own. Which is crucial for learning.

If the someone else decided how to do it, it's too easy to reject all responsibility. In this case the learning effect is weak.

This is also why side-projects are so powerful. Developing one entirely on your own means you have total control, all decision are yours and all mistakes too. You see the impact of everything you do, of every line of code you write.

This is not to say that you can hire people directly out of university, let them alone for 2 years, and suddenly they are the best developers in the world. No. Inexperienced engineers need guidance and support. Sometimes there are issues they just can't figure out, or a quick explanation saves them days of research. It's equally important to show them what to learn, so they invest their time effectively, and to help them achieve their best.

For senior developers autonomy is equally important, just in a different way. They already know how to program and write software. To improve they need a wider area of impact, get exposed to more and different technologies, and learn the "softer" aspects of software engineering. Not only soft skills, also how their choices impact the consumer, how to create a good architecture, which technologies to use, how to fulfill requirements without overengineering, so on so forth.

I'd say that in general, people strive for autonomy and freedom in their lives. Developers maybe even more so than the average person. Regardless of how much experience an engineer has, autonomy will impact their job satisfaction. Exactly how much is different for each person, also because more autonomy usually also means more responsibility. This is for their team leads to figure out. Overall I'd say though the more the better for employee happiness.

Impact on Team Leads?

Team leads are responsible for the team's speed and set, or at least have a strong influence on, all of the 3 axis of autonomy I described above (task size, definition and estimation). The way this is done not only affects the developers, but also the team lead.

The more detail a task description has the better, as long as it describes what to do. Team leads often got their position because they are great developers and therefore usually have an idea on how to achieve the desired result. For the sake of team efficiency and velocity they add their knowledge to the description.

For issues where speed or correctness are highly important this makes sense, but that are by far not all. For the others detailed descriptions about what to do just has the potential to reduce the lead's productivity. If there is a pointer of how to implement something, questions for more details will come, which sporadically interrupt both parties. When enough of that happens it becomes difficult for the team lead to focus on bigger tasks.

The increased time needed to do closer managing of developers (not just answering questions) reduces the size the team can have. It's possible to micromanage 3 people, with more macro management it can go up to 10 or maybe even more. So higher levels of autonomy save time.

The goal is usually not to reduce the autonomy of developers. How much they have is usually a side-effect of everything else they do, often from measures to increase the team's development speed or make planning more accurate. For example splitting tasks up because small ones can be estimated more accurately or exactly knowing how to achieve the desired result to reduce estimation variance.

Over the long run increased autonomy engages the engineers and uses their full potential, knowledge and creativity for solving problems. As mentioned above, this maximizes skill improvement and therefore increases the team's velocity.

All of that is justified by developers learning faster with more autonomy. I'm quite certain that this is true, but of course cannot be 100%. It might also be different for each person. Full autonomy doesn't mean no guidance or mentoring though. This can and should still happen. It could even have more impact in high-autonomy environments.

Actively giving more freedom is not always an easy choice. It means not always knowing the status of tasks, where difficulties are and when they'll be done. It makes coordination and future planning difficult. Also reporting is affected by that.

So currently I'm thinking that high autonomy makes the team lead's job is more difficult with the long-term reward of a faster team.

Conclusion

The topic of autonomy is a hard one. Most of the time it's a side-effect of other measures and not explicitly thought about.

My current view is that autonomy is the key to being able to manage more and more people, utilizing their full potential and keeping developers engaged. The exact way to achieve all that depends on the company, the environment, the team lead and the people making up the team.

There is, as always, but unfortunately, not a one size fits all solution.

Type-Safe Redux Reducers

Dominik Weber — Fri, 23 Aug 2019 00:00:00 +0000

The main problem that has to be solved is how to show TypeScript what the concrete action type is after checking the type property of Redux actions.

Straightforward Solution

The most straightforward way to do this is creating a type for every action and using type guards to distinguish them.

Type guards are boolean functions that check if a parameter is a specific type, indicated by the return value being parameter is Type.

Used in a condition, TS assumes the variable passed to the type guard is of the type the guard checks for. Only in the true branch of course.

An example solution with type guards:

interface Action {
  type: string;
}

interface StartAction extends Action {
  date: Date;
}

interface SuccessAction extends Action {
  data: number;
}

interface ErrorAction extends Action {
  error: string;
}

export const createStart: () => StartAction = () => ({
  type: 'Start',
  date: new Date()
});

export const createSuccess = (data: number): SuccessAction => ({
  type: 'Success',
  data
});

export const createError = (error: string): ErrorAction => ({
  type: 'Error',
  error
});

// TYPE GUARDS
const isStartAction = (action: Action): action is StartAction => 
  action.type === 'Start';

const isSuccessAction = (action: Action): action is SuccessAction => 
  action.type === 'Success';

const isErrorAction = (action: Action): action is ErrorAction => 
  action.type === 'Error';

export interface State {
  lastStarted?: Date;
  data?: number;
  error?: string;
}

const defaultState: State = {
  lastStarted: undefined,
  data: undefined,
  error: undefined
};

const reducer = (state = defaultState, action: Action) => {
  if (isStartAction(action)) {
    return {
      lastStarted: action.date,
      data: undefined,
      error: undefined
    };
  }

  if (isSuccessAction(action)) {
    return {
      data: action.data,
      error: undefined
    };
  }

  if (isErrorAction(action)) {
    return {
      data: undefined,
      error: action.error
    };
  }

  return state;
};

This works, but is quite verbose.

For every action, it's necessary to define

The action type
The action creator
The type guard

And whenever an action changes, a property is added/removed/changed, it has to be adapted on 2 places, the specific action interface and the action creator.

It's possible to get rid of both, but that requires a little detour first.

Inferring Concrete Types from Unions

TS has a neat type inference feature. Under specific circumstances, it can infer the concrete type from a union. What I mean is this

const x: A | B = someObject;
if (someCondition) {
  // x is of type A here
}

For this to work A and B must have a property in common. In Redux actions it would be type. This is necessary so that TS knows this property can be accessed in the condition.

Additionally the type property must be a literal type, for example a literal string (e.g. 'Start') or number (e.g. 123).

This type is checked in the condition. Since each type's type property can only be a specific value, TS can infer the type based on the given value.

interface A {
  type: 'Start';
  propA: number;
}

interface B {
  type: 'End';
  propB: number;
}

if (x.type === 'Start') {
  // x is of type A here
}

For this reason it's necessary that the type properties are literal types. If they were a generic string, there'd be nothing to go on for type inference.

Using Unions to Infer the Action Type

Armed with that knowledge, it's possible to get rid of the type guards, in exchange for adding ActionTypes, which is a union of all action types handled in that reducer.

interface StartAction {
  type: 'Start';
  date: Date;
}

interface SuccessAction {
  type: 'Success';
  data: number;
}

interface ErrorAction {
  type: 'Error';
  error: string;
}

type ActionTypes = StartAction | SuccessAction | ErrorAction;

export const createStart = (): StartAction => ({
  type: 'Start',
  date: new Date();
});

export const createSuccess = (data: number): SuccessAction => ({
  type: 'Success',
  data
});

export const createError = (error: string): ErrorAction => ({
  type: 'Error',
  error
});

export interface State {
  lastStarted?: Date;
  data?: number;
  error?: string;
}

const defaultState: State = {
  data: undefined,
  error: undefined
};

export default (state = defaultState, action: ActionTypes) => {
  if (action.type === 'Start') {
    return {
      lastStarted: action.date,
      data: undefined,
      error: undefined
    };
  }

  if (action.type === 'Success') {
    return {
      data: action.data,
      error: undefined
    };
  }

  if (action.type === 'Error') {
    return {
      data: undefined,
      error: action.error
    };
  }

  return state;
};

Removing Action Types with `ReturnType`

The last improvement to arrive at the final example is automatically inferring action types based on the return value of action creators.

This can be done with the helper type ReturnType.

As the name says, it is a generic type that, if passed a function, returns the type of the returned value. For example

type F = (p1: string, p2: number) => boolean;
type T = ReturnType<F>;
// T = boolean

If you're interested how it works, check out my other blogpost List of Built-In Helper Types in TypeScript.

By using that helper type it's possible to create the union type directly from the return types of the action creators.

One thing to notice here is that the returned objects define the type like this: 'Start' as 'Start'.

This is so the inferred type is of the literal string ('Start'), because by default, when assigning a string, TS infers it to be of type string.

type ActionTypes =
  | ReturnType<typeof createStart>
  | ReturnType<typeof createSuccess>
  | ReturnType<typeof createError>;

export const createStart = () => ({
  type: 'Start' as 'Start',
  date: new Date();
});

export const createSuccess = (data: number) => ({
  type: 'Success' as 'Success',
  data
});

export const createError = (error: string) => ({
  type: 'Error' as 'Error',
  error
});

export interface State {
  lastStarted?: Date;
  data?: number;
  error?: string;
}

const defaultState: State = {
  data: undefined,
  error: undefined
};

export default (state = defaultState, action: ActionTypes) => {
  if (action.type === 'Start') {
    return {
      lastStarted: action.date,
      data: undefined,
      error: undefined
    };
  }

  if (action.type === 'Success') {
    return {
      data: action.data,
      error: undefined
    };
  }

  if (action.type === 'Error') {
    return {
      data: undefined,
      error: action.error
    };
  }

  return state;
};

This is the final example, as concise and with the least typing effort possible.

All solutions are valid though, and depending on the rest of the codebase, the team and your preference you might want to choose a more explicit (first) way to implement type-safe reducers.

Benefits of a Component Library

Dominik Weber — Sun, 09 Jun 2019 09:03:36 +0000

At my work we're currently starting to create a company-internal component library.

It will be a bit of work, but significantly improve our UI development.

The most obvious benefits are:

Reduced code duplication
Increased speed
Improved UI consistency

One benefits that is less obvious, but underpins all those is reduced decision making effort. For designers, developers and the interaction of those groups.

In my opinion a component library is there to limit the possible designs, not just to have some base components.

And with limited possibilities, the thinking process will be faster. Instead of literally endless options, designers only have a few. The same thing goes for developers. They see the design and know which components to use. Overriding styles is not an option, only if it is explicitly mentioned and approved by all stakeholders.

If you're not thinking about component libraries as a way to limit design possibilities, you're only getting half the benefit.

The logic will be abstracted, yes, but the styling will still often be overwritten, which might even need custom logic at some point.

This is how I currently believe we can get the most of our component library.

What is your opinion? Do you think it makes sense? Maybe it would be too limiting for a designer? That it'd hamper their creativity? Or even facilitate it?

Do you know of any other benefits component libraries have?

Setting up a Reverse-Proxy with Nginx and docker-compose

Dominik Weber — Mon, 23 Jul 2018 12:56:22 +0000

Nginx is a great piece of software that allows you to easily wrap your application inside a reverse-proxy, which can then handle server-related aspects, like SSL and caching, completely transparent to the application behind it.

This is a cross-post from my personal website.

Introduction

Some aspects of web applications, like SSL encryption, request caching and service discovery can be managed outside of the application itself. Reverse-proxies like Nginx can handle many of those responsibilities, so we as developers don't have to think about it in our software.

Additionally, some software is not meant to be available over the internet, since the don't have proper security measures in place. Many databases are like that. And it is good practice in general to not make internal services public-facing that don't have to be.

All of that can be achieved with docker-compose and Nginx.

docker-compose

docker-compose is a neat little tool that lets you define a range of docker containers that should be started at the same time, and the configuration they should be started with. This includes the exported ports, the networks they belong to, the volumes mapped to it, the environment variables, and everything else that can be configured with the docker run command.

In this section I'll briefly explain how to configure the docker-compose features used in this article. For more details take a look at the documentation.

The main entry point is a docker-compose.yml file. It configures all aspects of the containers that should be started together.

Here is an example docker-compose.yml:

version: '3'
services:
  nginx: 
    image: nginx:latest
    container_name: production_nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    ports:
      - 80:80
      - 443:443

  ismydependencysafe:
    image: ismydependencysafe:latest
    container_name: production_ismydependencysafe
    expose:
      - "80"

As you can see, there are 2 images specified.

First nginx, with the name production_nginx. It specifies a volume that replaces the default Nginx configuration file. Also a mapping of the host's ports 80 and 443 to the container's ports 80 and 443 is defined.

The second image is one is one I created myself. It exposes port 80. The difference to the ports configuration is that they are not published to the host machine. That's why it can also specify port 80, even though nginx already did.

There are a few other configuration options used in this article, specifically networks, volumes and environment variables.

Networks

With networks it is possible to specific which containers can talk to each other. They are specified as a new root config entry and on the container configurations.

version: '3'
services:
  nginx:
    ...
    networks:
      - my-network-name

  ismydependencysafe:
    ...
    networks:
      - my-network-name

networks:
  my-network-name:

In the root object networks, the network my-network-name is defined. Each container is assigned to that network by adding it to the network list.

If no network is specified, all containers are in the same network, which is created by default. Therefore, if only one network is used, no network has to be specified at all.

A convenient feature of networks is that containers in the same one can reference each other by name. In the example above, the url http://ismydependencysafe will resolve to the container ismydependencysafe.

Volumes

Volumes define persistent storage for docker containers. If an application writes somewhere no volume is defined, that data will be lost when the container stops.

There are 2 types of volumes. The ones that map a file or directory to one inside the container, and the ones that just make a file or directory persistent (named volumes), without making them accessible on the file system (of course they are somewhere, but that is docker implementation specific and should not be meddled with).

The first type, volumes that map a specific file or directory into the container, we have already seen in the example above. Here is it again, with an additional volume that also specifies a directory in the same way:

version: '3'
services:
  nginx: 
    image: nginx:latest
    container_name: production_nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - /etc/letsencrypt/:/etc/letsencrypt/
...

Named volumes are specified similar to networks, as a separate root configuration entry and directly on the container configuration.

version: '3'
services:
  nginx:
    ...
    volumes:
      - "certificates:/etc/letsencrypt/"

    ...

volumes:
  certificates:
...

Environment Variables

Docker can also specify environment variables for the application in the container. In the compose config, there are multiple ways to do so, either by specifying a file that contains them, or declaring them directly in docker-compose.yml.

version: '3'
services:
  nginx:
    ...
    env_file:
      - ./common.env
    environment:
      - ENV=development
      - APPLICATION_URL=http://ismydependencysafe
    ...

As you can see, both ways can also be used at the same time. Just be aware that variables set in environment overwrite the ones loaded from the files.

The environment files must have the format VAR=VAL, one variable on each line.

ENV=production
APPLICATION_URL=http://ismydependencysafe

CLI

The commands for starting and stopping the containers are pretty simple.

To start use docker-compose up -d.

The -d specifies that it should be started in the background. Without it, the containers would be stopped when the command line is closed.

To stop use docker-compose down.

Both commands look for a docker-compose.yml file in the current directory. If it is somewhere else, specify it with -f path/to/docker-compose.yml.

Now that the basics of docker-compose are clear, lets move on to Nginx.

Nginx

Nginx is a web server with a wide array of features, including reverse proxying, which is what it is used for in this article.

It is configured with a nginx.conf. By default it looks for it in /etc/nginx/nginx.conf, but it is of course possible to specify another file.

As a reverse proxy, it can transparenty handle two very important aspects of a web application, encryption and caching. But before going into detail about that, lets see how the reverse proxy feature itself is configured:

http {
  server {
    server_name your.server.url;

    location /yourService1 {
      proxy_pass http://localhost:80;
      rewrite ^/yourService1(.*)$ $1 break;
    }

    location /yourService2 {
      proxy_pass http://localhost:5000;
      rewrite ^/yourService1(.*)$ $1 break;
    }
  }

  server {
    server_name another.server.url;

    location /yourService1 {
      proxy_pass http://localhost:80;
      rewrite ^/yourService1(.*)$ $1 break;
    }

    location /yourService3 {
      proxy_pass http://localhost:5001;
      rewrite ^/yourService1(.*)$ $1 break;
    }
  }
}

The Nginx config is organized in contexts, which define the kind of traffic they are handling. The http context is (obviously) handling http traffic. Other contexts are mail and stream.

The server configuration specifies a virtual server, where each can have its own rules. The server_name directive defined which urls or IP addresses the virtual server responds to.

The location configuration defines where to route incoming traffic. Depending on the url, the requests can be passed to one service or another. In the config above, the start of the route specifies the service.

proxy_pass sets the new url, and with rewrite the url is rewritten so that it fits the service. In this case, the yourService{x} is removed from the url.

This was a general overview, later sections will explain how caching and SSL can be configured.

For more details, check out the docs.

Now that we know the pieces, lets start putting them together.

Setup Nginx as a Reverse-Proxy inside Docker

For a basic setup only 3 things are needed:

1) Mapping of the host ports to the container ports
2) Mapping a config file to the default Nginx config file at /etc/nginx/nginx.conf
3) The Nginx config

In a docker-compose file, the port mapping can be done with the ports config entry, as we've seen above.

    ...
    ports:
      - 80:80
      - 443:443
    ...

The mapping for the Nginx config is done with a volume, which we've also seen before:

    ...
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
    ...

The Nginx config is assumed to be in the same directory as docker-compse.yml (./nginx.conf), but it can be anywhere of course.

Cache Configuration

Adding caching to the setup is quite easy, only the Nginx config has to be changed.

In the http context, add a proxy_cache_path directive, which defines the local filesystem path for cached content and name and size of the memory zone.

Keep in mind though that the path is inside the container, not on the host's filesystem.

http {
    ...
    proxy_cache_path /data/nginx/cache keys_zone=one:10m;
}

In the server or location context for which responses should be cached, add a proxy_cache directive specifying the memory zone.

  ...
  server {
    proxy_cache one;
  ...

That's enough to define the cache with the default caching configuration. There are a lot of other directives which specify which responses to cache in much more detail. For more details on those, have a look at the docs.

Securing HTTP Traffic with SSL

By now the server setup is finished. docker-compose starts up all containers, and the Nginx container acts as a reverse-proxy for the services. There is just one thing left to set up, as this site so beautifully explains, encryption.

To install certbot, the client that fetches certificates from Let’s Encrypt, follow the install instructions.

Generating SSL Certificates with certbot

certbot has a variety of ways to get SSL certificates. There are plugins for widespread webservers, like Apache and Nginx, one to use a standalone webserver to verify the domain, and of course a manual way.

We'll use the standalone plugin. It starts up a separate webserver for the certificate challenge, which means the port 80 or 443 must be available. For this to work, the Nginx webserver has to be shut down, as it binds to both ports, and the certbot server needs to be able to accept inbound connections on at least one of them.

To create a certificate, execute

certbot --standalone -d your.server.url

and follow the instructions. You can also create a certificate for multiple urls at once, by adding more -d parameters, e.g. -d your.server1.url -d your.server2.url.

Automating Certificate Renewal

The Let's Encrypt CA issues short-lived certificates, they are only valid for 90 days. This makes automating the renewal process important. Thankfully, certbot makes that easy with the command certbot renew. It checks all installed certificates, and renews the ones that will expire in less than 30 days.

It will use the same plugin for the renewal as was used when initially getting the certificate. In our case that is the standalone plugin.

The challenge process is the same, so also for renewals the ports 80 or 443 must be free.

certbot provides pre and post hooks, which we use to stop and start the webserver during the renewal, to free the ports.

The hooks are executed only if a certificate needs to be renewed, so there is no unnecessary downtime of your services.

Since we are using docker-compose, the whole command looks like this:

certbot renew --pre-hook "docker-compose -f path/to/docker-compose.yml down" --post-hook "docker-compose -f path/to/docker-compose.yml up -d"

To complete the automation simply add the previous command as a cronjob.

Open the cron file with crontab -e.

In there add a new line with

@daily certbot renew --pre-hook "docker-compose -f path/to/docker-compose.yml down" --post-hook "docker-compose -f path/to/docker-compose.yml up -d"

That's it. Now the renew command is executed daily, and you won't have to worry about your certificates' expiration date.

Using the Certificates in the Nginx Docker Container

By now the certificates are requested and stored on the server, but we don't use them yet. To achieve that, we have to

1) Make the certificates available to the Nginx container and
2) Change the config to use them

To make the certificates available to the Nginx container, simply specify the whole letsencrypt directory as a volume on it.

  ...
  nginx: 
    image: nginx:latest
    container_name: production_nginx
    volumes:
      - /etc/letsencrypt/:/etc/letsencrypt/
  ...

Adapting the config and making it secure is a bit more work.
By default, a virtual server listens to port 80, but with SSL, it should also listen to port 443. This has to be specified by 2 listen directives.

Additionally, the certificate must be defined. This is done with the ssl_certificate and ssl_certificate_key directives.

  ...
  server {
    ...
    listen 80;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/your.server.url/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/your.server.url/privkey.pem;
  }
  ...

These small changes are enough to configure nginx for SSL.

It uses the default SSL settings of Nginx though, which is ok, but can be improved upon.

Improving Security of Nginx Config

At the beginning of this section I should mention that, if you use the latest version of nginx, its default SSL settings are secure. There is no need to define the protocols, ciphers and other parameters.

That said, there are a few SSL directives with which we can improve security even further.

Just keep in mind that by setting these, you are responsible for keeping them up to date yourself. The changes Nginx does to the default config settings won't affect you, since you're overwriting them.

First, set

ssl_protocols TLSv1.1 TLSv1.2;

This disables all SSL protocols and TLSv1.0, which are considered insecure (TLSv1.0, SSLv3, SSLv2). TLSv1.1 and TLSv1.2 are, at the time of writing (July 2018), considered secure, but nobody can promise that they will not be broken in the future.

Next, set

ssl_prefer_server_ciphers on;
ssl_ciphers ECDH+AESGCM:ECDH+AES256:ECDH+AES128:DHE+AES128:!ADH:!AECDH:!MD5;

The ciphers define how the encryption is done. Those values are copied from this article, as I'm not an expert in this area.

Those are the most important settings. To improve security even more, follow these articles:

You can check the security of your SSL configuration with a great website SSL Labs provides.

Wrap up

In this article we've covered how to setup docker-compose, use its network and volume feature and how to set environment variables, how to use Nginx as a reverse proxy, including caching and SSL security. Everything that's needed to host a project.

Just keep in mind that this is not a terribly professional setup, any important service will need a more sophisticated setup, but for small projects or side-projects it is totally fine.

Amendment

Here are the resulting nginx.conf and docker-compose.yml files. They include placeholder names, urls and paths for your applications.

docker-compose.yml

version: '3'
services:
  nginx: 
    image: nginx:latest
    container_name: production_nginx
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - ./nginx/error.log:/etc/nginx/error_log.log
      - ./nginx/cache/:/etc/nginx/cache
      - /etc/letsencrypt/:/etc/letsencrypt/
    ports:
      - 80:80
      - 443:443

  your_app_1:
    image: your_app_1_image:latest
    container_name: your_app_1
    expose:
      - "80"

  your_app_2:
    image: your_app_2_image:latest
    container_name: your_app_2
    expose:
      - "80"

  your_app_3:
    image: your_app_3_image:latest
    container_name: your_app_3
    expose:
      - "80"

nginx.conf

events {

}

http {
  error_log /etc/nginx/error_log.log warn;
  client_max_body_size 20m;

  proxy_cache_path /etc/nginx/cache keys_zone=one:500m max_size=1000m;

  server {
    server_name server1.your.domain;

    location /your_app_1 {
      proxy_pass http://your_app_1:80;
      rewrite ^/your_app_1(.*)$ $1 break;
    }

    location /your_app_2 {
      proxy_pass http://your_app_2:80;
      rewrite ^/your_app_2(.*)$ $1 break;
    }
  }

  server {
    server_name server2.your.domain;
    proxy_cache one;
    proxy_cache_key $request_method$request_uri;
    proxy_cache_min_uses 1;
    proxy_cache_methods GET;
    proxy_cache_valid 200 1y;

    location / {
      proxy_pass http://your_app_3:80;
      rewrite ^/your_app_3(.*)$ $1 break;
    }

    listen 80;
    listen 443 ssl;
    ssl_certificate /etc/letsencrypt/live/server2.your.domain/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/server2.your.domain/privkey.pem;
    include /etc/letsencrypt/options-ssl-nginx.conf;
  }
}

Follow me on Twitter for more of my thoughts, articles, projects and work.

Do password rules impact security?

Dominik Weber — Sun, 08 Jul 2018 12:59:32 +0000

Recently I had a shower thought about passwords, namely that any rules we apply limit the possible passwords the user can choose. Which also means that there are less passwords an attacker has to go through when brute-forcing a password.

Setup

For simplicity I chose 4 character groups:

lowercase letters: 26
uppercase letters: 26
numerals: 10
special characters: 34

The calculations assume that there has to be at least 1 character of each group in the password.

This is not exhaustive of course, but I have to start somewhere.

Impact

In total there are 96 allowed characters.

This means, for an 6 character long password, without any rules, there are 96^6, or 782.757.789.696, options. Almost 800 billion.

With the rules above applied, there are only 26*26*10*34*96^2, or 2.118.205.440, options. Just a bit more than 2 billion.

That's a 370x difference!

To put that in perspective, instead of 1 year, it will take an attacker less than 1 day to brute-force that password.

Is that bad?

To be honest, I don't know. It sounds like much, but on the other hand, increasing the length by ~1.3 characters will negate the difference. So a password that's 1 character longer will almost do so, and one that's 2 characters longer will provide even better security.

What are your opinions on that topic? Please comment below 👇

Follow me on Twitter for more of my thoughts, articles, projects and work.

Coloring white Images with CSS filter

Dominik Weber — Sun, 17 Jun 2018 00:00:00 +0000

Recently I had to change the color of a white image. Since this image was rendered by a library, which did not allow replacing it by another one, it had to be done with CSS. Although there is a relatively simple solution to it, it proved to be quite a challenge to come up with that.

Lets get to it.

Should be easy!

Just add a hue-rotate filter, that should work.

But it doesn’t. Of course not, that would be too easy. The image is still white.

Why didn’t it work?

After checking a few times if the filter really was applied, it dawned upon me.

White has a luminocity value of 100%, so changing the hue won’t do anything. In fact, white has a hue value of 0, which is red. So I should have known beforehand that changing the hue won’t work, silly me!

Change the luminosity then

Since 100% luminocity is white, and 0% is black, 50% should result in a rich color. Lets try that.

Changing the luminocity can be achieved by the brightness filter, which changes the brightness based on the given factor.

Okay, something changed, that’s good. But the result is not as expected. There is no color. The image is just gray.

What happened?

The luminocity is now 50%, as expected. But I forgot about the saturation. Since white has 100% luminocity, I thought that saturation would also be 100%. Unfortunately, it is 0%, which causes the gray color.

So change the saturation

If there is a brightness filter, there surely is a saturate filter. And yes, there is!

Lets set the saturation to 100% then:

Again, nothing changed.

Turns out, the saturate filter works the same way as the brightness filter. It just changes the value based on the given factor. Now that’s a problem. As primary school math taught us, everything times 0 is 0.

This makes the saturate filter useless for this purpose.

Time to give up?

Not yet! There are a few other filters to try. Lets go through them:

blur: obviously doesn’t change the color
contrast: does nothing in a gray image
drop-shadow: also doesn’t change the color
grayscale: well, can’t make a gray image gray_er_
hue-rotate: the whole point is to make the hue visible, a different hue won’t change anything
invert: inverting gray, sadly, just results in a different variation of gray
opacity: it would be possible to make another color shine through, but that’s not what I’m after
sepia: that’s the last one. And finally, it looks like it could work!

The MDN documentation example looks very promising.

Let’s try it out:

Did that change something? It looks like white. Almost. By checking with the color picker I found out that the actual color is hsl(60, 100%, 97%).

That means, we’ve got saturation! Finally some results 🎉

From there it is now possible to apply the other filters to achieve the desired result.

Lets see some color!

I have written enough about trial and error already. Let me save you some time and show you how to get a pure red:

In my opinion this is the easiest color to work from to get whatever color you want.

For example, to get green, just add a hue-rotate(120deg) at the end.

Wrapping up

White is a nasty color to change with filters. With the typical filters, it is impossible to create a color other than grayscale.

The sepia filter might not have been designed for this purpose, but it saves the day!

One more thing: black

To turn black into a color, turn it to white and work from there:



img {
  filter: invert(1) brightness(50%) sepia(100%) saturate(10000%);
}

Follow me on Twitter for more of my thoughts, articles, projects and work.

Hosting Asp.Net Core Applications on Windows Server Core

Dominik Weber — Wed, 04 Apr 2018 22:13:11 +0000

Recently, I've found myself in the position of having to host an application on Windows Server. Having never managed a Windows Server before, I struggled to find relevant information, especially since most of it is written for a Windows Server with installed UI, and the default image on Azure is a Core image, without UI. This is mostly documentation for myself, but maybe you find it helpful too.

This is a cross-post from my personal blog.

Introduction

This is a step by step introduction of how to host an Asp.Net Core application on Windows Server Core with IIS (Internet Information Server).

We will cover how to set up IIS, how to configure it, how to deploy to it with Web Deploy in Visual Studio and securing connections to that application with https.

I'm using a virtual machine from Azure, which provides a nice UI for managing firewall rules. That is probably very different for you, so I'll just say which ports have to be open, and not cover how to do that.

Setting up the Server

After logging in on the server, you are greeted by a command prompt. Since most commands we will use are PowerShell commands, we have to start it.

Just enter powershell and execute it. After that you should see a PS in front of the prompt.

Now IIS has to be installed. This is done with this command:

Install-WindowsFeature Web-Server

While installing, PowerShell shows a nice little progress bar:

Enabling Remote Management

Per default, the server does not allow remote management. It has to be enabled by installing the Web-Mgmt-Service and setting the registry entry HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WebManagement\Server\EnableRemoteManagement to 1.

Keep in mind that the registry key is only available after Web-Mgmt-Service is installed.

Install-WindowsFeature Web-Mgmt-Service

Set-ItemProperty -Path Registry::HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\WebManagement\Server -Name EnableRemoteManagement -Value 1

After executing those commands restart the web server so that the changes take effect:

net stop was /y
net start w3svc

Also start the Web Management Service, otherwise you won't be able to connect to it.

net start wmsvc

Note: IIS Manager connects via port 8172, so make sure it is open on your server.

Enabling Management on your Windows 10 Device

To remotly manage an IIS server, the IIS Manager has to be installed on your device. This can be done in Control Panel -> Programs -> Programs and Features -> Turn Windows features on or off. Activating IIS Management Console is sufficient, IIS itself does not have to be installed.

Out of the box IIS Manager cannot manage remote servers. That features has to be added with IIS Manager for Remote Administration. You can download it here.

After it is installed, IIS Manager will have the menus enabled to connect to a remote IIS.

Now the connection to the remote IIS can be added. Just go to File -> Connect to a Server and fill out the required information.

Note: If you can't connect, most likely the Port 8172 is not open, or the Web Management Service is not started. Do that with

net start wmsvc

Configuring IIS to host Asp.Net Core Applications

By default IIS cannot host Asp.Net Core applications. The Asp.Net Core Module is needed for that, which is installed with the .NET Core Windows Server Hosting bundle.

1) Go to the .Net all downloads page
2) Select the .Net Core runtime you need
3) Download Server Hosting Installer (this is just to copy the download url, we need it on the server, not locally)
4) Copy the download url
5) Download the installer on the server with the command

Invoke-WebRequest https://download.microsoft.com/download/8/D/A/8DA04DA7-565B-4372-BBCE-D44C7809A467/DotNetCore.2.0.6-1-WindowsHosting.exe -OutFile C:\Users\YourUsername\Downloads\DotNetCore.2.0.6-1-WindowsHosting.exe
#This is the download url for the latest non-preview runtime at the time of writing (2.0.6).

6) Execute the installer

C:\Users\YourUsername\Downloads\DotNetCore.2.0.6-1-WindowsHosting.exe

Now, this is what was really surprising for me. The installer executes with a UI, the same as on any Windows. Being on a Core installation, I thought there would be absolutely no UI, but I was wrong.

This also opens the interesting option to install Chrome and download all necessary files with it.

Restart the web server so that the changes take effect:

net stop was /y
net start w3svc

Preparing IIS for Web Deploy

Since this is a small project, the most convenient deploy option is Web Deploy directly in Visual Studio.

As with almost everything else, this is not supported out of the box, but can be added.

Web Deploy can be downloaded from the Microsoft Download Center.

Use the same process outlined above, or Chrome, your choice :-)

Invoke-WebRequest https://download.microsoft.com/download/0/1/D/01DC28EA-638C-4A22-A57B-4CEF97755C6C/WebDeploy_amd64_en-US.msi -OutFile C:\Users\dominik\Downloads\WebDeploy_amd64_en-US.msi
#This is the download url for the latest Web Deploy at the time of writing (3.6).

Also execute that installer

C:\Users\dominik\Downloads\WebDeploy_amd64_en-US.msi

Note: I've read somewhere that all features have to be installed, and that the installer's Complete option does not actually install everything. So just select Custom and make sure to that all features are enabled.

Deploying an Asp.Net Core Application

Now we are finally ready to publish the application. Well, almost. A publish profile has to be created first.

1) Right-click on the Asp.Net Core application in the Solution Explorer
2) Select Publish
3) Click on Create new Profile
4) Select IIS, FPT, etc.
5) Select Create Profile where by default Publish is entered

6) Enter the required information
- Site name is either Default Web Site, or, if you created a different one in IIS, the name of that one.
7) Click Validate Connection to check if everything was entered correctly
8) If it was, click Save
9) Select the created profile
10) Click Publish and watch the magic happen :-)

Configuring SSL

We've achieved what we wanted, hosting the application. Now there is only one step left: securing it with SSL. Don't worry, it's not difficult, I promise.

There is a great project out there, called Windows ACME Simple, which makes this process really simple.

1) Download the latest release (you can get the download link from the release page of the Github project)

Invoke-WebRequest https://github.com/PKISharp/win-acme/releases/download/v1.9.10.1/win-acme.v1.9.10.1.zip -OutFile C:\Users\dominik\Downloads\win-acme.v1.9.10.1.zip
#This is the download url for the latest version at the time of writing (1.9.10.1).

2) If this fails with the message The request was aborted: Could not create SSL/TLS secure channel., try execute [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12 beforehand (from StackOverflow).

3) Extract the zip file

Expand-Archive C:\Users\dominik\Downloads\win-acme.v1.9.10.1.zip -DestinationPath C:\Users\dominik\Downloads\win-acme.v1.9.10.1

4) Execute letsencrypt.exe

C:\Users\dominik\Downloads\win-acme.v1.9.10.1\letsencrypt.exe

5) Select N to create a new certificate in simple mode
6) Select 1 to create a single binding of an IIS site
7) Now you should see a selection of sites you have configured. Select the site you want to secure
8) After you've added an email address and agreed to the subscriber agreement, it does its magic
9) If all goes well, your site is now encrypted and you can quit Windows ACME Simple (Q)

Closing

That's it. The application is now fully set up. I hope this walkthrough helped you as much as it undoubtedly will help me in the future, the next time I have to set up a Windows Server.

Resources

Follow me on Twitter for more of my thoughts, articles, projects and work.

Using Scripting to quickly execute repetitive Tasks

Dominik Weber — Sun, 18 Feb 2018 11:59:39 +0000

This is a cross-post from my personal blog.

Sometimes I have to do a repetitive task for a project, that would take hours if done manually. Shell commands are normally the way to go, but I've always had a hard time learning them, and after I did, I forgot them quickly, because I need them very infrequently. So I found another solution.

Scripting! Especially with interactive ruby.

For example, recently I had to rename a bunch of files, removing ".inline".

The files I had looked like this:

icon.inline.svg
logo.inline.svg
(and a bunch more)

And they should be renamed to

icon.svg
logo.svg

Okay, lets do this.

Navigate to the directory where the files are and start the interactive ruby shell with irb (if Ruby is installed)
Get all filenames in a variable:

irb(main):001:0> filenames = Dir.glob("*.svg")

Rename them in a loop:

irb(main):002:0> filenames.each { |f|
irb(main):003:1* File.rename(f, f.sub(".inline", ""))
irb(main):004:1> }

Done. Quite easy, right?

Now, this is a simple use-case. Looking up a shell command wouldn't take too long.

But what if it gets more complicated?

What if the file needs to contain a special text? What if the file size has to exceed a particular limit? What if ...

You know where I'm getting at.

With Ruby, I know what to do. Just look up whatever I don't know and combine everything with the constructs I, as a developer, am familiar with.

Since I have never extensively used it, I'd have a lot more problems with shell commands/scripts. I'd need to look up how to do the subproblems (e.g. getting the file size) AND how to combine them with pipes, arrows and whatever else exists.

Additionally, Ruby statements are usually a lot easier to remember. Staying with the example of getting the file size, this is how it works with bash (accepted answer in the first Google result):

stat -f%z icon.inline.svg

With Ruby:

File.size("icon.inline.svg")

Doesn't Ruby look a lot more straightforward?

So, if you're like me and don't know how to do complex tasks with shell, give interactive ruby a chance. It will infinitely speed up those annoying repetitive tasks.

Follow me on Twitter for more of my thoughts, articles, projects and work.

Keeping the footer at the bottom with CSS Flexbox

Dominik Weber — Sun, 04 Feb 2018 22:13:37 +0000

Just now I've read the post Keeping the Footer at the Bottom with CSS-Grid.
While reading, I somehow got the impression that doing the same with 'just' flexbox is difficult. It's not.

This is not to belittle the referenced post in any way, it is very well explained, and in my opinion more elegant than with flexbox. I just want to explain that it is as easy with flexbox as it is with grid, so that if, for whatever reason, you cannot use grid, you don't shy away from implementing it with flexbox.

I'll use the same HTML as in the referenced article:

<html>
    <body>
        <article>
            <header>
            </header>
            <main>
            </main>
            <footer>
            </footer>
        </article>
    </body>
</html>

And here the CSS to put the footer at the bottom, where it belongs:

html, body {
    width: 100%;
    height: 100%;
}

article {
    min-height: 100%;
    display: flex;
    flex-direction: column;
    align-items: stretch;
}

main {
    flex-grow: 1;
}

header, main, footer {
    flex-shrink: 0;
}

On the container, we set flexbox to align the contents in a column. The elements should stretch so that they span the whole width, and not just the width their contents take up.

Setting flex-grow: 1 on main makes it grow to fill the available space. This puts the footer at the bottom, since main takes up all the space in the middle.

The use for flex-shrink: 0 is probably less obvious, and it is often forgotten. At least I forgot it way more often than I'd like to admit.

By default, flex-shrink is set to 1. This makes the items shrink if there is not enough space, which happens if the content is larger than the screen. The results can look very weird, e.g. a button that is smaller than the text it contains. Setting it to 0 stops that behavior.

That's it. That's everything you need to know to position the footer at the bottom with flexbox. :)

Follow me on Twitter for more of my thoughts, articles, projects and work.

What are good Resources to learn about core IT Topics?

Dominik Weber — Sun, 17 Dec 2017 14:04:19 +0000

Recently, I've talked to some engineer friends, and some core IT topics came up. Topics like virtualization, specific protocols like HSTS or how browsers work.

During this conversation, I realized that I don't know enough about these topics, and now I want to learn.

The thing is, getting an overview of these areas is enough for me. 10 hours of knowledge on a specific topic is too much, and I won't remember the details anyway.

Finding such resources proves difficult, because I cannot even properly put them together in a category.

So, there are 2 questions. Do you know what I mean? And if so, how would you categorize it? And do you know about any resources for that?

Podcasts would be ideal, so I can listen to it on my commute, but anything is welcome.

Follow me on Twitter for more of my thoughts, articles, projects and work.

The Bliss of Automating my Server Setup

Dominik Weber — Thu, 14 Dec 2017 20:15:56 +0000

I have a VPS, on which a few of my side-projects are hosted. Recently, I had to set it up a few times, because I made it unusable (please don't ask me how, it was really stupid).

Every time this happened, it took me hours to set it up again. Granted, I don't know much about maintaining a server, and you can probably do it a lot faster than me, but you can never be faster than a script. So, after the second time, I finally decided it is time to automate that process.

Goals

I'm hosting ElasticSearch and a few .Net Core applications. They are all accessible via an nginx reverse proxy.

Additionally, there are a few cronjobs.

While writing the script, I decided to add Docker support to all my .Net Core applications, so there is one less thing needing to be installed.

This is extremely easy btw, just right-click -> Add Docker support -> Done. But I digress.

The script should

install Docker, docker-compose and nginx
pull all Docker images
copy the nginx and ElasticSearch configuration files to the right locations
add cronjobs
start all applications
configure the server so that after rebooting the applications start automatically
install other applications that I like (e.g. htop)

The Script

Writing the script was simple, quite a lot simpler than I thought it would be. It was basically copy&pasting the commands from the vendor websites and using the Docker CLI, which is extremely well documented.

Docker: https://docs.docker.com/engine/installation/linux/docker-ce/ubuntu/#set-up-the-repository

docker-compose: https://docs.docker.com/compose/install/

nginx: https://www.nginx.com/resources/admin-guide/installing-nginx-open-source/#prebuilt

I did have some minor difficulties with specific software though, which are described in the following 2 sections.

ElasticSearch

ElasticSearch has a small fallacy. By default, the vm.max_map_count environment variable is set to 65536.

This is too low for ES, which results in out of memory exceptions. This is very confusing, if there are a few GB of free memory.

The solution is setting vm.max_map_count to at least 262144, as mentioned in the docs, which is quickly found after some googling.

Cronjobs

Sadly, crontab does not offer a way to simply add an entry. But as expected, others had the same problem, and Stackoverflow has an easy to use solution for it.

(crontab -l 2>/dev/null; echo "@reboot nginx -c /home/ubuntu/nginx/nginx.conf") | crontab -

This also demonstrates how applications start after a server reboot. Simply with an @reboot crontab entry.

Conclusion

Automating the setup process has brought down the time until my server is ready from hours to minutes. And in those blissful minutes, I don't have to do anything, just watch the script work.

Additionally, I feel much more comfortable experimenting. If something breaks, I just have to start the script.

Automation is great!

PS: If you are interested in the whole script, you can find it here, minus my credentials of course ;)

Follow me on Twitter for more of my thoughts, articles, projects and work.

DEV Community: Dominik Weber

Developing a reading list for programmers - Motivation

Grand vision(s)

Opportunity to tackle societal problems?

Development

Conclusion

Thoughts on Developer Autonomy

Impact on Developers

Impact on Team Leads?

Conclusion

Type-Safe Redux Reducers

Straightforward Solution

Inferring Concrete Types from Unions

Using Unions to Infer the Action Type

Removing Action Types with ReturnType

Benefits of a Component Library

Setting up a Reverse-Proxy with Nginx and docker-compose

Introduction

docker-compose

Networks

Volumes

Environment Variables

CLI

Nginx

Setup Nginx as a Reverse-Proxy inside Docker

Cache Configuration

Securing HTTP Traffic with SSL

Generating SSL Certificates with certbot

Automating Certificate Renewal

Using the Certificates in the Nginx Docker Container

Improving Security of Nginx Config

Wrap up

Amendment

Do password rules impact security?

Setup

Impact

Is that bad?

Coloring white Images with CSS filter

Should be easy!

Why didn’t it work?

Change the luminosity then

What happened?

So change the saturation

Time to give up?

Lets see some color!

Wrapping up

One more thing: black

Hosting Asp.Net Core Applications on Windows Server Core

Introduction

Setting up the Server

Enabling Remote Management

Enabling Management on your Windows 10 Device

Configuring IIS to host Asp.Net Core Applications

Preparing IIS for Web Deploy

Deploying an Asp.Net Core Application

Configuring SSL

Closing

Resources

Using Scripting to quickly execute repetitive Tasks

Keeping the footer at the bottom with CSS Flexbox

What are good Resources to learn about core IT Topics?

The Bliss of Automating my Server Setup

Goals

The Script

ElasticSearch

Cronjobs

Conclusion

Removing Action Types with `ReturnType`