DEV Community: Mehmet Barış Kalkar

Opinionated Frameworks

Mehmet Barış Kalkar — Sat, 16 Apr 2022 14:48:56 +0000

Opinionated

Definition of the word opinionated by merriam-webster is:

firmly or unduly adhering to one's own opinion or to preconceived notions

Opinionated Frameworks

In software development related discussions however, this word is used together with languages, frameworks or tools and it has a more specific meaning.

An opinionated framework/language has an “intended” or “right” way of doing things.

Python is considered opinionated because there is a “pythonic” way to solve problems.

Spring boot, Django, Ruby on Rails and Nestjs are pretty good examples to opinionated web application frameworks. Because they have well accepted ways of implementing things.
For some of the more common encountered issues, they even have default configurations and implementations.

This does not mean opinionated tools are not flexible. It means, if you use the language, library or the framework correctly, everything should be straightforward and reasonably simple.

Or in other words, it should “just work” if you are not doing anything crazy.

Why?

We have similar frameworks in all languages and there is a good reason it. When we build production ready web applications, we need to think about a lot of common issues.
These are not related to the business solution we want to create but they still take time to implement properly, even if we assume developers have the expertise.

Request routing
Logging
Error handling
Validation
Request/Response Serialization/Deserialization
Database access, possibly with an ORM tool
Authentication/Authorization
Request filters

Pros

Most opinionated frameworks not only have solutions for all these issues, they even have sane defaults
So we might not need to think about asynchronous logging or Serialization at all during development.
Or when we are implementing authentication, we might use a lot of code from the framework instead of writing it ourselves.

Similar to how design patterns help the development process, having a shared set of rules and processes to implement functionality can help productivity and maintainability.
This is especially helpful to prevent bike shedding around folder structure, naming things etc.

Working in multiple projects using the same framework makes switching between them easier and this is a huge boost in productivity if we can leverage it.

When building services without much experience, using a good framework can teach us about best practices and industry standards.

Cons

We have to follow a structure and it might not be ideal for our use case.

If we leave the golden/happy path things start to get difficult.

We might not need all the complexity or the default functionality provided.

Even sane defaults might be too magical if we are not aware of them. This is one of the pain points for Spring Boot from my experience.

Conclusion

Benefits certainly outweigh the cons in my opinion but the decision of using an opinionated vs non-opinionated framework depends a lot on the team size, complexity of the application, number of projects running in parallel and experience of the developers.
In any case, understanding why these frameworks exist and the problems they solve is important before this decision.

Java Records, Lombok and POJOs

Mehmet Barış Kalkar — Wed, 17 Nov 2021 15:24:54 +0000

For various reasons like carrying or storing data, deserializing/serializing api requests or simply organizing our code base, we use java entities called POJOs or Plain Old Java Objects. They usually have multiple variables with accessor methods and little/no business logic. In most cases, they are immutable and short-lived.

POJO

A simple POJO would typically have a public constructor, getters and setters, equals, hashCode and toString methods. Since these classes are used frequently, IDEs have shortcuts to create these methods. So it is actually not a big loss of time to create them. There is still the possibility of forgetting to update an equals or toString method after adding a new field though. Considering how many of these we have, it is still a lot of code to look at and manage.

public class User {

    public User(String id, String name, String address) {
        this.id = id;
        this.name = name;
        this.address = address;
    }

    private String id;
    private String name;
    private String address;

    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }

    public String getAddress() {
        return address;
    }

    public void setAddress(String address) {
        this.address = address;
    }

    @Override
    public boolean equals(Object o) {
        if (this == o) return true;
        if (o == null || getClass() != o.getClass()) return false;
        User that = (User) o;
        return Objects.equals(id, that.id) && Objects.equals(name, that.name) && Objects.equals(address, that.address);
    }

    @Override
    public int hashCode() {
        return Objects.hash(id, name, address);
    }

    @Override
    public String toString() {
        return "User{" +
                "id='" + id + '\'' +
                ", name='" + name + '\'' +
                ", address='" + address + '\'' +
                '}';
    }
}

Lombok

Lombok is an annotation processor which works at compile time to add getters, setters and constructors to the code. It is a pretty popular library since java is incredibly verbose compared to other languages and there is always a desire to
have simpler and shorter code. Lombok has more features related to logging, builders, validations etc.

With this library however, we need to understand that the code we are working on is not syntactically correct until compilation. So we also need a plugin to tell our IDE everything is fine. There are less popular alternatives
like Immutables and AutoValue and they use a more natural way of annotation processing.

With Lombok, we could have the same functionality as the POJO above with a much shorter definition

@Getter
@Setter
@AllArgsConstructor
@ToString
@EqualsAndHashCode
public class User {
    private String id;
    private String name;
    private String address;
}

To make this even simpler, we can use the @Data annotation which is equivalent to @Getter @Setter @RequiredArgsConstructor @ToString @EqualsAndHashCode. We still need the @AllArgsConstructor to have a constructor with all variables though. Also, we might not want to have @Setter or a @NoArgsConstructor in many cases.

@Data
@AllArgsConstructor
public class User {
    private String id;
    private String name;
    private String address;
}

Records

Records are a relatively new addition to java that solves the same problem without any external libraries, annotation processing or plugins. You can refer to JEP-384 and JEP-359 for more context about this feature. Motivation, Goals and Non-Goals are pretty well-defined in these JEPs.

record User(String id, String name, String address) {
}

With this definition, we can create new User instances with the new keyword like a normal class. Similar to how @Data from Lombok works, we get public getters for all fields, equals, hashCode and toString methods. However, since every field is final, we do not have setters for records.

Designed to have a very specific role in java, they have some restrictions.

They only have private final fields for each parameter in their definition, no more instance fields
They can not extend any other class
They are implicitly final and cannot be abstract
They are designed to be immutable, so no setters

While records may not be suitable to work with Spring Data JPA since they are immutable, they work really well as API layer request and responses or messaging DTOs.
They also work really well when you just need them to carry data across layers instead of passing down 5 parameters. Immutability is also an advantage to consider when dealing with records.

Compact Constructor

If we want our constructor to do something after initializing the final fields, we can use something called compact constructors. Note that we are not defining a typical constructor, this code block will be running before the actual compiled constructor, and we can use the parameters from the actual constructor itself.

record User(String id, String name, String address) {
    public User {
        if (StringUtils.isBlank(id) || StringUtils.isBlank(name)) {
            throw new ValidationException();
        }
    }
}

We can also use annotations like javax validations on records

record User(@NotBlank String id, @NotBlank String name, String address) {
}

Static Methods in Records

To have more customizations on the record, we can have a static factory method inside the body.

public record User(String id, String name, String address) {
    public static User create(String name, String address) {
        return new User(UUID.randomUUID().toString(), name, address);
    }
}

I think records are really useful in all places where immutability is not a concern. Using a native solution instead of libraries like Lombok is a good move for Java overall.

Preventing Alert Fatigue

Mehmet Barış Kalkar — Sun, 07 Nov 2021 19:26:04 +0000

Expectation for software reliability and availability is at an all-time high now. With money and prestige on the line, every application and website wants to avoid downtime while building new things as fast as possible. However, perfect infrastructure or perfect software does not exist, so incidents do happen.

What we can do is minimizing the impact by making sure we are responding quickly and efficiently. This makes on-call rotations a part of our lives now.

If structured and compensated correctly, on-call rotations can improve the overall quality and the response time for incidents quite a lot. However, it is not always a positive experience to be on call. One reason for an unpleasant on-call is getting many alerts.

Alert Fatigue

Alarm fatigue or alert fatigue describes how busy workers become desensitized to safety alerts, and as a result ignore or fail to respond to such warnings.

This term did not originate in software, but it is applicable to many industries and works well to describe the problem. It is dangerous for both developers and companies because

when developers start ignoring all alerts, they also miss or respond later to critical ones
their productivity suffers
they are most likely dissatisfied if they are waking up in the middle of the night for alerts multiple times a week
they are more likely to burn themselves out

How to avoid it?

Create Actionable Alerts

Rule #1 in creating effective monitors is having actionable alerts. If you are not going to take any action when you get the alert, you might need to re-evaluate your thresholds or change what you are monitoring.

Prevent Over-Monitoring

We are collecting a lot of metrics from our services, but it does not mean we should create monitors for every single one of them. We should be deliberate while creating more monitors and make sure every alert should be useful.

Misconfigured monitoring is a common cause of operational overload. Paging alerts should be aligned with the symptoms that threaten a service’s SLOs. All paging alerts should also be actionable. Low-priority alerts that bother the on-call engineer every hour (or more frequently) disrupt productivity, and the fatigue such alerts induce can also cause serious alerts to be treated with less attention than necessary.

Delay Non-Urgent Notifications

We really do not need to fix every single issue when we notice them. If something can wait until the morning and we know it in advance, delay the notification until working hours. Some examples for this could be problems in non-production environments, temporarily having longer response times from a non-critical service or any other trivial issue.

Trying to deploy changes at 3AM in the morning, on your own, without any reviewers stresses me out just by thinking.

Set Up Priorities

Not every alert has the same priority. We can have many alerts and monitors but only the critical ones should wake us up or call us. Some can be just push notifications or informational alerts without any notification which are closed automatically if the situation does not get worse. We can also use different priorities with escalations and notification settings.

Owners Should Be On-call for their services

Being on call was a sysadmin/ops task traditionally, but now it is the norm for developers to be on-call for the services
they maintain. Ideally, we should only be getting alerts only from services we own.

This structure helps in multiple ways:

since you have more expertise, you are more likely to know what to do
you do not need to ask questions to other people so you can act quickly when a situation arises
you feel more responsible for your service
you can analyze the impact and priority of problems better
you are more motivated to fix recurring issues

This changes according to business domain and company but, we might also consider having on call-rotations for non-engineering teams like business or marketing too.

In some situations, having an on-call for more than one services might make sense. Not having enough experienced engineers in a team could be one reason for this because being on-call every other week is an even larger problem.

Create Incident Runbooks

Having incident runbooks is a good way to reduce stress associated with getting paged. Instead of thinking for a solution, we can follow a list of steps. Creating these runbooks also makes sure we are creating meaningful alerts. Because if we can't think of anything to do when we get the alert, it might be unnecessary.

Have a good escalation plan

There might be extraordinary cases where we are not available to respond to an alert. Alternatively, we might need help from our teammates to solve a certain issue. In this case, the person or teams we are asking for help should be already clear to us.

Be comfortable to ask for help

Asking for help should feel natural during incidents. On-call rotations are set up so not everyone is pinged for every single alert. A second person is really useful in most problems because bouncing off ideas or an extra set of eyes helps a lot while solving problems. If the issue actually needs everyone in the team to work on it, they should. This is also a great way to share expertise within teams because incidents are a great source of experience for everyone involved.

Have a clean alerts dashboard

Having a clean alerts dashboard is a good indicator for a team which is on top of their alerts and monitors. If you have dozens of open alerts all the time, you might be missing important alerts or ignoring some problems with your service. If having an empty dashboard is the norm, every alert feels important

Anatomy of a Dockerfile

Mehmet Barış Kalkar — Sat, 21 Aug 2021 18:27:36 +0000

What is a Dockerfile?

Dockerfile is a text document containing commands which can be run in sequence to assemble a docker image.

A sample Dockerfile from the official docs looks like this

# syntax=docker/dockerfile:1
FROM node:12-alpine
RUN apk add --no-cache python g++ make
WORKDIR /app
COPY . .
RUN yarn install --production
CMD ["node", "src/index.js"]

Why do we need them?

There are millions of images on dockerhub that we can directly start using with a command like this.

docker run -it --rm -d -p 8080:80 --name web nginx

For various reasons, we may want to customize these base images. Docker images are immutable, so we can't exactly modify them.
We can technically run a container using an existing image, make some changes on it and then create a new image with these modifications using the commit command but there is a better way to accomplish this task
However, before we start modifying images, we need to understand the concept of layers in docker

Images, Layers and Containers

Each Docker container consists of a readable and writable layer on top of multiple read only layers.
These read only layers represent instructions in Dockerfiles, and they are deltas on previous layers(similar to git commits)

Multiple containers can share the underlying layers since they have their own writable/readable layer on top.
The readable and writable layer is a thin layer which has a lifespan associated with the container.

# syntax=docker/dockerfile:1
FROM tiangolo/uvicorn-gunicorn-fastapi:python3.7
LABEL maintainer="Mehmet Baris Kalkar"
LABEL version="1.1"
RUN addgroup api && adduser fast && adduser fast api 
USER fast:api
ENV GREETING="hola"
COPY ./app /project/app
WORKDIR /project
EXPOSE 8090
CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8090"]

If we create a container from this same dockerfile, we will see a log similar to this:

 => [1/4] FROM docker.io/tiangolo/uvicorn-gunicorn-fastapi:python3.7@sha256:a0e0188a485fd8c232d8774ae4680d3b834f95dd2deccdb0211ce71cfd778b97
 => [internal] load build context
 => => transferring context: 56B
 => [2/4] RUN addgroup api && adduser fast && adduser fast api
 => [3/4] COPY ./app /project/app 
 => [4/4] WORKDIR /project 
 => exporting to image 
 => => exporting layers 
 => => writing image sha256:3cef1a7b7ddc037fa375a1fb37daa907bc31031fedb4142b98e98e582c0bead5
 => => naming to docker.io/library/fastapi

One important thing to understand is how these instructions are cached.
The result of some commands like FROM, COPY/ADD, RUN and WORKDIR can be cached.

Cached instructions are marked in the build command. If we build the same image by changing only the WORKDIR instruction to project2, we would see something like this.

 => CACHED [1/4] FROM docker.io/tiangolo/uvicorn-gunicorn-fastapi:python3.7@sha256:a0e0188a485fd8c232d8774ae4680d3b834f95dd2deccdb0211ce71cfd778b97
 => [internal] load build context
 => => transferring context: 56B
 => CACHED [2/4] RUN addgroup api && adduser fast && adduser fast api
 => CACHED [3/4] COPY ./app /project/app
 => [4/4] WORKDIR /project2
 => exporting to image
 => => exporting layers
 => => writing image sha256:fe482845750cf79708d1a6cc107578e76bd843f92fb3092d636180547b32b897
 => => naming to docker.io/library/fastapi

Let's take a look at this Dockerfile line by line

# syntax=docker/dockerfile:1

(Optional) syntax is only enabled if we are building the image with BuildKit
In this line, we can inform the Dockerfile builder which syntax to use while parsing the Dockerfile

FROM tiangolo/uvicorn-gunicorn-fastapi:python3.7

FROM instruction is used to set the base image that we are going to use.
This should always be the first instruction in a Dockerfile.

LABEL maintainer="Mehmet Baris Kalkar"

LABEL version="1.1"

LABEL instructions are used to add metadata to images.

Side note, There used to be a MAINTAINER instruction in the past, but it is deprecated now.

RUN addgroup api && adduser fast && adduser fast api

RUN instruction is used to execute commands in a new layer on top of the current image and commit changes.
Following steps will use the new image.

USER fast:api

USER instruction sets the user and group for the following steps.

ENV GREETING="hola"

ENV is used to add environment variables to the container. This variable can be used in the following steps during build as well.
If we want to use a variable in only a single command and not in the image, we can define use the RUN command with a variable instead.

RUN LOCUST_LOCUSTFILE=custom_locustfile.py locust

COPY ./app /project/app

COPY [--chown=:] ... copies files from source and adds it to the file system of the container
Target path is always relative to the working directory.

ADD command also has a similar function, but it can also be used to fetch files from a remote URL or extract tar files.

It is preferred to use COPY
over add because COPY is a more transparent and simple instruction.

WORKDIR /project

WORKDIR Sets the working directory to run instructions like CMD, RUN, ENTRYPOINT and COPY after this step.

EXPOSE 8090

EXPOSE is an informational instruction. It does not actually publish any ports, but it is used as a documentation to let
users know which ports should be published to use the image.

CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8090"]

CMD is the instruction to define the command you want to execute when run a container from an image.
It is possible to override this command while actually running the image, so it acts as a default.

ENTRYPOINT and CMD are similar commands, the differences are explained here pretty well

VOLUME command is used to create mounting points within the container.
We can use these volumes to share files between containers or the native host.