DEV Community: Ben

Remote Onboarding as a Software Engineer

Ben — Wed, 19 May 2021 00:00:00 +0000

If you are a software developer and you have been working remotely during this pandemic, it might come as no surprise that due to recent studies over 80% of software engineers want to continue working at least partly remote after the pandemic. While this opens up new possibilities in the way we live and work, it comes with certain difficulties. Not only do companies need to learn how to manage their remote workforce, but also employees need to find new ways to be successful at their job while wearing no pants.

Remote onboarding is difficult

The most difficult part of working remotely is joining a new company. Companies themselves invest a lot of time and budget to work and improve on their onboarding process. If they manage to onboard their employees quickly, they will be more useful to them and their employees will be happy. But how can we as software engineers make the most of this phase? As software engineers, we need to understand the problem we have to solve. Making sure we get this right is essential for our personal success. Moreover, in order to make a good job, we need to understand company dynamics, communication structures and find a way to socialize when there is no way to randomly meet people in the hallway.

Lay the groundwork

When you start out at a new job, the one thing you want to get right is to understand what your job actually is. Most of the times, it varies quite a lot from what the job description says. Start out and figure out what the company mission is. From there, go a step deeper and figure out what your team or department is responsible for. Ultimately, make sure you know how you fit into the puzzle and why you were hired. Reading through the company wiki is a good start to get a broader sense of what is out there, what topics are discussed and also how well people document. Aim for broad knowledge gathering in the beginning and focus on the details later. Try to understand as quickly as possible how the development process from a commit on a feature branch to the actual production deployment works, so you yourself can be productive quickly.

Be humble and patient

Most of us are aware of developer bias for certain technologies, patterns and frameworks. However, we tend to fall into the same trap of believing our way of writing code is the way it should be done again and again. Even if you have years of experience, when you join a new team and work on a new code base stay humble and be patient. It is easy to criticize a project that evolved over years. Some things won't make sense on the first view, but on the second. Most of the times, there were good reasons why a team chose to use technology X or design a system a certain way. Even if these reasons are not viable anymore at this current point in time, try to understand why choices were made in the past. Your coworkers will fell much more respected.

Ask

Especially in the beginning of a new job, you will have a lot of questions and that is completely normal. While undisturbed, focused time is sacred to developers, you will need to ask questions to your co-workers at some point. Try to get an answer by yourself by looking at the wiki or other internal guides and documentation, but don't hesitate and waste too much time. People are usually friendly and helpful and getting quick answers will prevent bigger misunderstandings in the future.

Communicate outside of your team

Getting in touch with co-workers outside of your team is difficult. In a remote environment, you cannot randomly meet people in the hallway or kitchen. You need to get out of your comfort zone. Ask your employer if you could join one of the other teams to get to know them and their way of work. Their might even be something like an exchange program between teams in place. Look out for projects and opportunities to work together with other teams. Maybe you have done something in the past or experience in a topic another team is working on? Share your knowledge. Your coworkers will appreciate it and you will boost your impact.

1:1s

While this also depends on your manager, making use of your personal 1:1s can help immensely with your remote onboarding experience as a software engineer. When you join a new company, you certainly have some expectations and goals in mind. Make the effort and write them down. Share them with your manager and look together if they align with the goals of your manager and the goals of the company. Try to learn what the goals of your manager are to align them with your goals. This way, you guarantee to work in the same direction and both of you share the same expectations.

After all, we are all in the same boat. Do not forget that if you fit into the job quickly, this also a success for your manager and team. Be open with your expectations and your concerns and raise any issues quickly and your coworkers will most likely repay you.

Learnings From Three Years Into Software Development Without A Degree

Ben — Tue, 23 Feb 2021 00:00:00 +0000

Three and a half years ago, I got my first job as a software engineering intern at my current employer. At this point in time, I barely knew that Java was not only an island but also a programming language. I was coming from a complete different field of study, originally on my way to become a teacher in Germany. Today, I am leaving my employer, starting my second job as a software developer without a degree.

These are my learnings and tips for everyone considering going this path (especially in Germany).

Spoiler: I won't advise you to drop out of college and get into tech like so many people advocate. If you are twenty something most likely you should finish your studies or at least join a bootcamp, if this is possible for you.

Find a sponsor

No, not that kind of sponsor that gives you money and baseball caps. A sponsor is a person that believes in you and supports you. In Germany, getting into a company without a degree or anything to show off is near impossible. I was lucky enough to have a friend who offered me an internship position in his startup team, which was originally not even about software development. You will need someone who believes in you and supports you to a) get into the job and b) pull you through when you start struggling and questioning yourself. To get off the ground you might want to consider doing some work for little or no money to show off your potential.

Work on your social skills and learn to adapt

You can only do so much to learn and teach yourself about software development. In my experience, the best software developers were those which were also fit to work in a team. You might be surprised, but if you support others and help them reach their goals, chances are high, they will pay you back with their help. There will be many questions and sometimes you simple might not know what search for in Google. Care for your peers and they will care for you.

At some point, some of your peers will leave the company, you get a new boss or just a new team member. Every time one of those events happen, your personal situation changes with it. You will have a hard time, if you are not able to adapt to new situations, so learn to embrace change and watch out for new opportunities for yourself.

Develop a constant learning habit

If you take anything from this, it should probably be this. Software developers are knowledge workers. Every developer needs to learn new languages, frameworks or design patterns at some point. If you start with little to no knowledge in a team of peers that have some kind of education in software development, you will need to outrank them in your ability to learn something new. This seems hard (and it is), but if you manage to do it, it has the nice side effect that you will likely keep this habit throughout your career.

Just to be clear: this does not mean to work 12+ hours a day, but to improve the way you learn and work. There is a whole industry around productivity. Look out for ways to improve the quality of your work rather than the pure amount of time you work. Working 40 hours a week in front of a screen is exhausting. I have done the same mistake and I have seen many peers do it: working 60+ hours a week, because you just want to finish that feature or fix that bug. Get some sleep and take breaks and fixing that bug, which would have taken you four hours after a hard working day, might just be fixed in a couple of minutes the next morning.

Favour fundamentals over frameworks

Especially at the beginning of your journey the amount of frameworks, languages and design patterns seems overwhelming and staggering. While you probably will need to learn some framework for your job, focus your studies on fundamental concepts of software development. I was working for a company doing mainly backend web development, so I had to teach myself everything from HTTP over databases to a suitable language like Java. There are many great resources out there, but if you are trying to get into web development, I recommend taking a look at these developer roadmaps or your welcome to reach out to me. You will have a much easier time following a structured learning path than I had blindly stumbling from one topic to another and hammering my peers with questions.

T-shaped skills

As described above, you should pick a niche in software development. It is easy to jump from one broad topic to the other, because there is so much out there in surely a real developer knows everything about them, don't they?

They don't.

As developers we all specialise in something and this is important. If you are looking for jobs, you want to differentiate yourself from others and most companies have specific requirements for the people they hire. There is a well-known concept among developers, which is called the T-shaped developer. It is based on the notion to specialise in one topic and then develop broader understanding of other topics around that.

Take responsibility and get shit done

Even though the mantra of developing a "get things done"-attitude seems kind of obsolete, it is not. If you take responsibility and commit to goals with your team lead, you do not only develop a better feeling to estimate the time of your work, you actually learn to "own" your work. There are only few things more rewarding than working on something that is actually yours. This can lift you through hard days when you may lack motivation and it will show your boss and peers that you actually care and they can rely on you.

To make this happen you will also need to let some things go and learn to prioritise. From my experience, there is no need for a super framework or tool to organise your work. I use very simple and effective methods:

Define long term goals for the next six months (you might want to look at SMART goals)).
Choose a simple tool like a todo list or KanBan tool to write down your tasks for every day and keep a backlog (I organise everything in Notion).
Define your daily tasks on the evening before, starting with the most important to the least important.
Every morning, check if your tasks today are in line with your long term goals. Try to eliminate everything that is not.
Be consistent with your daily tasks but be flexible with goals, if new opportunities arise.

The hard part here is to really be consistent in prioritising the right tasks and then sticking to them. Defining my tasks on the evening before is probably the most valuable productivity hack. If you start planning your day in the morning, you are tempted to avoid difficult tasks.

Promote yourself

One thing many people feel uncomfortable with, including me, is promoting themselves. As a software developer without a degree you will need to do this at least to some extent. You do not need to be a Twitter influencer with 100k followers, but you should make sure to always have an up-to-date LinkedIn profile , a GitHub profile with some projects that showcase your skills and a CV. I do like Twitter as well, since you can easily connect with people from the tech industry, but it has not provided any value in terms of getting a new job or project so far.

Setting up a small portfolio or blog like this one can also be beneficial. First, you have one platform that you own to link to all your social media accounts and promote yourself. Second, blogging about what you learn and do does not only help with promoting, but also with understanding more deeply what you write about. I am not super consistent with blogging so far, but every time I do, I am suprised by how much value I get out of it for myself. Third, building a blog can be a perfect sample project to showcase your engineering skills.

Conclusion

As I mentioned in the beginning, I would not advise you to take this path, if you have the possibility, time and money to get a proper education in computer science. Having a degree and something that proves your skills will make your life much easier and less stressful. But if you do not have that for any reason, it can be done and the job as a software developer can be very rewarding one, not just financially.

If you are considering a career in software development without a degree, I would love to know more about where you are at and I hope my learnings were helpful. Feel free to reach out to me on Twitter or via mail and tell me your story :)

API Usability Testing in 8 Simple Steps

Ben — Fri, 05 Feb 2021 00:00:00 +0000

If you are building an API product, developer experience should be at the top of your mind. Testing your APIs for usability is crucial to offer your customers the best experience possible. While there are many people who offer usability tests for user-facing products, API usability testing is a bit more tricky, since your customers are developers. In this blog post, I will provide you with a guide on how you can get started with API usability testing without burning through your valuable startup budget. We will look at when to test, who to test and eight simple steps to organize your testing.

The Timing

The best time to start usability testing is right at the beginning of your project. If you have not done that, the second best time is now. It is true. Do not fall into the trap of testing your APIs for usability shortly before a release. Make it part of your development process early on and gather feedback as early as possible, so you can iterate on that feedback and change your product while it is easy to do so. If you are working in a somehow agile development process, which I will assume at this point, your development cycle will probably look something like this. The best time to do your usability testing is after you have designed your product or in our case, the API. With modern tools like Swagger and the OpenAPI standard you can easily generate Mock APIs which can then be used for usability testing with real users or your team. If you do not have mock APIs and prefer to develop first, usability tests can also be done after the Development stage.

The Users

Developers come in different forms and shapes. They vary in their roles and skill sets, but also in the way they perceive content. If you can, try to determine which roles and skills your customers have and derive user personas from it. This will help you to view your APIs from your customers perspective and find suitable candidates for a usability test. We will not go into detail on how to define user personas, but I recommend watching this short video by Pluralsight to get an idea. In reality, we often do not know the developers who will integrate our APIs when we start developing a new API. Therefore, it is helpful to assume a broad variety of different skill sets and find users of different backgrounds. As we do not want to hire an agency and spend as little money as possible on usability testing, there are mainly two easy options:

Freelance developer platforms: Upwork, fiverr, etc.. Look out for developers of various backgrounds and hire them for a 30-60 minute usability test. We will go into detail on how to do that in the next section.
Onboarding of new developers: When a new developer joins your team, he can be tasked to make a usability test of the API. This has the nice side effect that they get to know the product/API and a unique view of how users perceive it.

API Usability Testing in 8 Steps

To learn as much as possible from our tests, we need a bit of preparation. These are eight simple steps, to get everything out of your usability tests:

Define an explicit task on what your tester is supposed to test.
Give as little information as possible, but as much as needed to get started with the assigned task.
Find a way to observe the user when using your API. Nowadays, where in-person meetings are less common, you can make use of screen sharing/recording tools.
Make sure, the user speaks out loud while testing your product, explaining what he or she is thinking at each step.
Provide a possibility for questions, if the user gets stuck at a certain point of the process and does not find any other option than asking for help.
Create a detailed report on the observations during the test.
Derive learnings and tasks from your observations.
Share your learnings and observations with your team.

One other way to get started with usability testing is to do a Hackathon based on your API. This has the positive side effect that you do not only learn about the usability but also potential use cases of your API.

Conclusion

API usability testing does not have to be an expensive and tedious task. Getting started with it will immediately improve your understanding of your own product and how users perceive it. You will be astonished by how much your personal conceptions may differ from a user's fresh view on your API and your customers will thank you for providing the best developer experience. Since, API products are still quite new, there is not much content around usability testing for APIs. If you want to share your learnings and discuss, feel free to reach out to me on Twitter or subscribe to my Newsletter if you want to read more on this topic.

Exposing microservices with an API gateway and the API composition pattern

Ben — Fri, 17 Jul 2020 10:57:28 +0000

Microservices are a widely spread pattern in modern software architectures. Tools like Docker, Kubernetes and modern CI/CD pipelines have enabled software teams to build highly scalable distributed systems in which each microservice can be operated by a small team. In fact, the implicit notion of microservices is to be self-contained and focused on a single purpose. Thus, implemented correctly, microservices enable your system to be highly flexible and your teams to operate independently. While this is a major upside for the development process itself, it comes with certain challenges when you want to expose your system to external users.

Check out this well-written blog series of NGINX, if you want to get deeper into microservices architecture.

Flexibility leads to complexity

Don't get me wrong. The microservices pattern takes away a lot of complexity introduced by highly cohesive and monolithic systems, but it introduces a new kind of complexity. The fact that teams are flexible in the way they build their microservices - even if you have certain guidelines in place - makes it more difficult for your clients to access your system. They possibly need to

talk with all the different services
understand which service provides the data they need
communicate via different protocols.

In terms of user experience, this is not optimal and might lead to frustration on the client side. The complexity increases the learning curve of your system.

A simplified overview of clients A,B and C which require different data from individual services X,Y, and Z of a microservice architecture. All are accessible via different protocols: REST, GraphQL and gRPC.

The picture above illustrates such a microservice architecture. Even if your microservices do not use different communication protocols such as in this example: Talking to multiple hosts might still require multiple ways of authentication, understanding different styles of API design, etc..

An API gateway as the single entry point

An API gateway might help you with this complexity. As the term "gateway" implies, the main purpose of an API gateway is to be the single entry point for your microservice architecture. It is build on top of your individual services and routes requests of clients to these services. It does not contain or know of any business or domain logic of the individual services.

An API gateway sitting between your microservices and the clients who want to access these services. The API gateway provides a unified interface to the clients and hides implementation details of the services X, Y, and Z.

Upsides of API gateways

As the illustration shows, an API gateway makes it possible to easily hide implementation details of our microservice architecture and unify the external interface. This enables clients to have a single point of entry to your system, thus, making it more accessible. Besides these UX improvements, the API gateway can be used to centralize and handle topics such as

authentication
permission handling
API security
policies (rate limits, allow-listing, etc.)

This enables the individual microservice teams to concentrate on the core business/domain logic of their service and improves development speed. The API gateway might further support your system by providing extensive (access) logging and collecting performance as well as business related metrics of your clients.

Downsides of API gateways

Introducing a new part in your architecture always comes with certain challenges and so does an API gateway. The API gateway is another piece of software in our system that has to be developed, deployed and maintained. While this was also one of the upsides listed above, a single point of entry to our application might also become a bottleneck in two different ways.

First, it has to be highly scalable. Since it is another layer between every request of the client and our microservices, it needs to have as little impact on the request latency as possible. Second, it can slow down development, since every microservice has to be exposed via the gateway.

The API composition pattern

So far, we have introduced the API gateway simply to route each request of a client to a microservice and back. With microservices being self-contained and the owner of their domain and data models, they often fail the expectations of clients on how to consume the data. Clients might want to query data of multiple sub-domains at the same time or just a small portion of the data a single microservice offers. To fit these needs, the API composition pattern can be applied.

Overview of an API gateway with API composition. Composition 1 composes data of service X and Y to a single resource, while Composition 2 composes data of service Y and Z to a single resource.

Even though the API composition now manipulates the payloads of one or multiple microservice(s), an API composition still remains free of any domain or business logic. Each composition could also be defined as an aggregate of the microservices behind the API gateway to fit the clients need and improve the developer experience.
This pattern introduces some more advantages. With API composition in place, we suddenly can

make up for bad design decisions and apply a new shiny interface
hide legacy systems and replace these whenever we have the time for it
enable specialists for API design and developer experience to create a consistent design of the overall API.

On the other hand, as with the API gateway itself, we introduce another layer in our overall system. Development teams have to be careful that API composition does not evolve to a replacement for good API design of the individual microservices or mixing up the domain logic and models within the API gateway.

Conclusion

API gateways and API composition are a useful tool to reduce complexity of microservice architectures. They can help with handling access and monitoring of the overall system, as well as enabling clients to request data as they expect it. Nevertheless, one has to be careful when introducing a new layer to a system, since it inevitably leads to more dependencies in the development chain.

What do you think of this approach? Have you faced similar challenges? If yes, how did you solve them?
If you want to talk more on the topic of APIs and microservices, feel free to contact me on Twitter or via e-mail.

Docker Hands-On Start for Developers - Part I

Ben — Mon, 20 Apr 2020 09:52:38 +0000

If you are working as a software developer in 2020, you probably have heard about Docker. In fact, by now over 20% of all hosts are running Docker with increasing numbers for the upcoming years.

If you have no experience and you want to get started with this technology, you have come to the right place. You will get a succinct overview of the topic and some best practices on how to leverage Docker for your own projects or your projects at work. In this part, we will learn a short definition of the technology and we will dive into more detail on the basic elements of Docker: images and containers. To get some immediate hands-on experience, we will build our own docker image and spin up containers from it.

What Docker is and why you want to use it (Part I)
Images (Part I)
Containerizing you application (Part I)
Containers (Part I)
Best Practices (Part II)

What Docker is and why you want to use it

You probably know about the concept of virtual machines (VMs). VMs are simulating operating systems with system-specific pre-installed software. They are managed by a hypervisor (virtual machine monitor) which manages and shares hardware for multiple VMs on a single physical machine. VMs need a big amount of memory and computing power and therefore are expensive to setup and maintain. Furthermore, moving VM workloads between hypervisors and cloud platforms is not as easy and intuitive as it could be.

In contrary, Docker or other containerization technologies are running on the host operating system (OS). Containers contain your application code, dependencies, your runtime system and system libraries, settings and tools. That is it. Nothing more than the bare minimum of what you need to run your application. They do not need a full-blown OS to setup, instead they free up a lot of CPU, RAM and storage for running applications. As long as a host has the Docker Engine running, you can easily port Docker containers from one host to another. You will never have to worry, what kind of server your application is running on or how to ship between hypervisors. You simply pick or create your own docker image and focus on developing your software.

This image visualisation is copied from the Docker website and illustrates the difference:

Source: https://www.docker.com/resources/what-container#/package_software

Images

As a developer, you should be familiar with the concept of object-oriented programming. In a way, you can think of images as classes and containers as objects. Images are your templates to build containers on.

To start out, you usually pull an image from a docker registry like Docker Hub. Images are made of one or multiple stacked read-only layers. The only writable layer is build by the container as the illustration below shows. We will discuss what exactly a layer is in a minute.

Source: https://docs.docker.com/v17.09/engine/userguide/storagedriver/imagesandcontainers/#container-and-layers

Getting started

So, to dive deeper, let’s start and play around with Docker. If you have not done so already, go here and install the Docker Engine on your host. When you’re done, open your command line tool and type:

docker pull node:latest

Congratulations! You have pulled your first Docker image. Let’s look at what happened here:

The docker pull command initiates a call to a Docker registry (the default is Docker Hub) to pull an image with the name node and the tag latest. A Docker registry serves as a storage server for docker images. From the command above, we can extract the following syntax: docker pull <name>:<tag>

The name specifies the name of the image and the tag specifies the version of a particular image. (Attention: Image tags are mutable. To make sure, you can rely on an image version, you might want to consider image digests).

In this case, we pulled an image with a pre-installed version of Node.js. Your output should look like this:

Working with images

Let’s have a closer look at the image we just pulled. Execute the following command in your terminal:

docker images

This will return a list of all your images like this:

You can see the name of the image, the tag we specified before, a unique image id, a timestamp when it was created and the size of the image. To further inspect the image execute:

docker inspect node:latest

Here you can see a lot of metadata for the image like the size of the image, the different layers, the underlying OS and much more. We will come back to the inspect command in the the Best Practices section.

Now, let's remove the image with:

docker rmi node:latest

This way, we removed the image and the respective layers of the image from our Docker host.

Note that all these commands can be executed with the image id, but they will only work, as long as no container based on the image is running. To understand how images are built and why they are structured into layers, we will build a simple Node.js server, containerize it and spin up our first containers from it.

Containerizing your application

To keep things simple, I set up a simple Node.js server. You can find the source code here. For this example, you do not need any experience with Node.js. You simply need to have node and the node package manager npm installed and follow along the tutorial. You can install both as described here.

Clone the repository, cd into the repository’s folder and execute the following commands to build the application and install all dependencies.

npm install && npm run build

Now, let’s run the server with

npm start

Now, our server should be running. If you navigate your browser to http://localhost:8080, you should see a "Hello World of Docker" message.

Great! Now, let’s see how we can make an image of this application and spin up a container of it. Remember, our image is our template for all our containers and it should do only one thing: run our application.

In the project repository, create a new file and name it Dockerfile. In there, copy the following code:

FROM node:latest


WORKDIR app


COPY . .


RUN npm install && npm run build


EXPOSE 8080


ENTRYPOINT ["npm", "start"]

To build your image, execute docker build -t my-first-docker . in the same directory as your Dockerfile. You should see the different steps building. When we execute docker images, we can see our newly created image:

There are two kind of instructions in a Dockerfile. Those, which add data and therefore create a new layer and those which simply add metadata to the image. I will walk you through those steps and explain in detail what they are doing.

The FROM command specifies a base image, on which we want to build our image. In our case, we use the latest image from the node repository. This image has Node.js pre-installed since we want to build a Node.js server. In general, it is recommended to use base images from official repositories since you do not have to worry about installing Node.js properly.

WORKDIR creates a working directory for your application. It is seen as best practice, because we avoid unreadable syntax like cd .. && do foo. The working directory will be your default directory in which the following RUN, COPY, etc. commands will be executed. Our working directory will be app.

COPY copies files from your project directory into the Docker image. For now, we will just copy all files in our working directory, even though this is not considered best practice. Notice: this is the first time we are adding data to our image. Thus, this instruction creates a new layer in your image. Each instruction of this type creates its own layer.

RUN executes a command in the default shell of the image. Since it adds data, it creates another layer. In our case, we add data by installing all dependencies with npm install and building the compiled files with npm run build.

EXPOSE exposes a specified network port for our container.

ENTRYPOINT sets a default shell command, which will we be executed when we start a container and run it. This command makes the Docker image an executable when run. We could also use the CMD instruction to achieve this behaviour. Nevertheless, it is recommended to set ENTRYPOINT whenever you want your users to use this image in a specific way. If you want to give your users more flexibility, choose CMD, because it can be overwritten more easily.

Containers

As explained, containers are the runtime counterpart to images. Containers accept an image and a command as arguments. The image is used to run the container and the command specifies the application, the container should run. Let’s use our example to elaborate on this. Run:

docker run -it my-first-docker

And just like that, you have your first docker container running! The run command starts a container. You pass an image as an argument and with the -it flag you will attach your shell to the container’s shell. To exit the container, you can simply press CRTL-C on Linux/Win or CMD-C on MacOS.

Now, we do not want to attach our shell every time we build a Docker container. In these cases, you can add the -d flag to run a daemon of the container. We can try it like this:

docker run -d -p 8080:8080 my-first-docker

Your docker container is now started in the background. At the same time, we mapped the exposed port 8080 of the container to our local port 8080. Run:

docker ps

As you can see, our docker container is running. Docker assigns a unique CONTAINER ID to every container. We also can see the command, which is executed as specified in our ENTRYPOINT command in the image, the mapped ports and a randomly assigned name for the container (can be specified manually with the —name flag). If you move back to your browser and http://localhost:8080, you should see your “Welcome to Docker World” message again, since all traffic on our local port 8080 is routed to our running docker container.

To stop the container, simply use:

docker stop <CONTAINER ID>

Now that our container is stopped, we can see the running and stopped containers with:

docker ps -a

You can see the container still exists. To remove our container completely, we can use a final:

docker rm <CONTAINER ID>

Conclusion

For now, this should be everything to get started with Docker and experiment a bit on your own. We learned, why Docker is such a great alternative to VMs, what images and containers are, how we can containerize our own app and how to interact with our running containers. Hopefully, this has helped you to dive into this amazing technology.

In the next part, we will have a look at some general best practices when working with Docker, especially on how to build your images and set up your Dockerfiles.

Stay tuned!

Explore Arrays & Slices in Go

Ben — Tue, 31 Mar 2020 15:59:57 +0000

There are many ways to organize sequential data in the programming world. The most common ones are the abstract data types array and list. They are at the beginning of every programmer's learning journey. With arrays and lists we can implement our first for loops, we can store those notes for the note taking app we are building, or implement our own buffer.

In this post, we will have a look at how these data types are implemented in Go. After defining what exactly those types are, we will inspect static arrays and why we have slices instead of lists in Go. Finally we will look at the complexity of working with these types to get a better grasp on when to use these data types.

Static Arrays

Static arrays are at the core of every programming language. They have a fixed length of n, which has to be defined upfront. In memory, all elements of the array are stored adjacently. If you want to iterate over an array, this can be achieved via an index starting at 0 for the first element and ending at n - 1 for the last. In Go, arrays and slices are both evaluated during runtime. Furthermore, the length of a slice is also part of the array type, so int[10] and int[20] are different types.

Declaration and initialization

There are different ways to declare and initialize an array. We can first declare it and set its values explicitly.

var array [3]string
array[0] = "all"
array[1] = "the"
array[2] = "elements"

Or we declare and initialize it with one line.

var array = [4]int32 {23, 12, -1, 8}

Accessing elements

Elements are accessed by their index. The following statement gives us the second element of an array.

var secondElement = array[1]

Specifying an index lower than 0 or greater than n - 1 results in an index out of bound error. In Go, specifying an index out of range will not even compile.

Searching elements

Searching elements can be done via iteration and a simple if condition.

for i := 0; i < len(array); i++ {
  if array[i] == "foo" {
    fmt.Println("Found foo")
  } 
}

Time complexity

Since static arrays are fixed in size, our options to work with them are limited to only accessing and searching for data.

Action	Complexity
Accessing elements	O(1)
Searching elements	O(n)

Slices

With slices things get interesting. The ability to grow and shrink this dynamic-array-like data structure in size gives us a new set of possibilities to work with. Before inspecting how they are used, let's have a closer look at slices and how they are special compared to similiar data structures of different languages.

Compared to other languages, creating a slice from an array is basically just a struct with a pointer to the array and two variables holding length and capacity of the array. It does not allocate any new memory. As a consequence, basic operations on slices are very cheap. Creating, moving on or expanding the respective array (until the capacity is reached) can all be done without any new memory allocation. This leaves the user with the indicated dynamic-array-like feeling when working with slices.

Declaration and initialization

Here, we also have different ways to declare and initialize an array. This time we start with the declaration and initialization combined. This will create a slice with an underlying array of size and capacity 4.

var slice = []int32 {23, 12, -1, 8}

This is fairly convenient, but may not be the most efficient way. This way, when adding an element via the append()function, a new array has to be allocated in memory. On the other hand, Go gives us the tools to handle the allocation ourselves.

var slice = make([]int, 2, 3)
slice[0] = 6
slice[1] = 12

This make function creates a new slice with an underlying array of length 2 and capacity 3.

Appending elements

When we add one more element to the slice above, no new memory is allocated, because of the big enough capacity, which improves our efficiency. Another element more and a new array hast to be created.

slice = append(slice, 7) // still working on the same array
slice = append(slice, 14) // allocates new memory, since we expanded the capacity

You can easily test this behaviour on your own. If you want to know more about what exactly happens when appending new elements in Go, I recommend this article from The Go Blog.

Slicing a slice

We can create a "sub-slice" of a slice by specifying the upper and lower index like this.

var subSlice = slice[2:5]

Note that this does not create a new slice. We are still operating on the same data in memory. You should be aware that this is behaviour might lead to the fact that your program holds more data in memory than you might expect.

Deleting elements

Since Go does not have a respective method in its standard library, let's implement a method to delete an element of a slice of strings.

func remove(slice []string, element string) error {
    i := -1
    for j := 0; j < len(slice); j++ {
        if slice[j] == element {
            i = j
        }
    }

    if i == -1 {
        return errors.New("element not found in slice")
    }

    slice = append(slice[:i], slice[i+1:]...)

    return nil
}

If you watch closely, you can see that we can use our sub-slicing syntax with variadic arguments to cut the element from our slice. Another quite convenient feature at this place.

Inserting elements

We do the same for inserting new elements.

func insert(slice []string, index int, element string) []string {
    return append(slice[:index], append([]string{element}, slice[index:]...)...)
}

In this case, we have no other option than creating a new slice with the append() function. Again we can use variadic arguments as a shortcut.

Time complexity

Now, let's enhance our time complexity chart for the available operations appending, inserting and deleting.

Action	Complexity
Accessing elements	O(1)
Searching elements	O(n)
Appending elements	O(1)
Inserting elements	O(n)
Deleting elements	O(n)

Conclusion

As we can see, working with arrays and slices in Go is quite convenient while there are some specialities which might be surprising for developers with a background in other languages. Generally spoken, working with arrays is very foreseeable and easy, which is why they are so popular. Nevertheless, when we work with large or specially structured data, they might not always suite your use case and come with certain performance downsides.
I will elaborate on this in future posts on data structures in Go. For now, I hope I could shed some light onto these data types and their implementation. If you have any questions, feedback or just want to discuss about the topic, feel free to contact me via mail or follow me on Twitter.

Keep your service alive with a circuit breaker

Ben — Thu, 19 Mar 2020 16:51:39 +0000

Chances are high you are familiar with microservices architecture. If you do not work in such an environment, you should at least be familiar with the concepts of services running in a network, communicating with each other. With this architectural approach come certain challenges. When sending network calls, our service might run into timeouts of a downstream service. Not handling those timeouts or errors might result in processing or memory overload. To fix this, we can apply a pattern from the electrical hardware world: circuit breakers.

Why you would want a circuit breaker in place

The circuit breaker pattern is an architectural pattern to make a specific function call fault tolerant and robust. It is used in distributed systems where remote or network calls take place. Its main goal is to quickly analyse if a downstream service is unhealthy and to stop calling it, thus risking memory or processing overload. You might be familiar with this term in the context of electrical switches where a circuit breaker is used to avoid damage on a electrical circuit by overload or a short circuit.
If you are dealing with a high amount of traffic which depends on a certain downstream service, you should consider this pattern to improve your system’s overall stability and end user experience.

How does it work?

The concept of a circuit breaker is fairly simple: whenever a downstream service or function stops working, our circuit breaker "trips" and stops calling it over and over until it works again. Meanwhile all attempted calls result in a specified error.

To apply this pattern effectively in software, we need three steps:

failure monitoring
the circuit state machine
error handling

To illustrate the power of the circuit breaker pattern, we will implement a simple example which will focus on the circuit state machine. We will outline a very simple failure monitoring strategy, trigger the circuit state machine to “trip” and show up some real world use cases. The respective code for the state machine we are building can be found here.

I do not recommend to use this code in a production-like context. There are already battle proven frameworks with this pattern. For Go you can check out gobreaker or if you are looking for a HTTP client with the respective functionality, have a look at Heimdall.

Overview

Before we dive into the hands-on action, let’s have a look at a visual representation of this pattern.
The circuit breaker can have three different states: closed, open and half-open. The default state is closed. While being closed, the circuit breaker will forward all requests to a certain function or downstream service. Meanwhile, it monitors all requests and errors for the execution of a given function. When the circuit breaker is open, all incoming requests will return an error immediately. Anyhow, before switching from a closed to an open state, the circuit breaker will try to heal itself by switching to a half-open state.
For the sake of simplicity, we will follow a rather simple approach for our failure monitoring strategy. Let’s say, we want our circuit to trip when a call throws a certain amount of consecutive errors.
The circuit breaker pattern will look like this:

Implementation

Now, let’s see how this might be implemented in code. At first, we define our different states.

type State int

const (
   Closed   State = 1
   HalfOpen State = 2
   Open     State = 3
)

Our circuit breaker should take in a function and return a generic interface and an error. Therefore, we will implement a respective interface.

type CircuitBreaker interface {
   Execute(func() (interface{}, error)) (interface{}, error)
}

The next step is to define all required variables for our failure monitoring strategy. In the beginning we said, the circuit breaker should start to trip when a certain error threshold is exceeded. Therefore, let’s add a strategy which holds the threshold to our circuit breaker, a counter to count our consecutive errors, and the current state.

type Strategy struct {
   threshold     int
}

type circuitBreaker struct {
   strategy          *Strategy
   state             State
   consecutiveErrors int
}

type CircuitBreaker interface {
   Execute(func() (interface{}, error)) (interface{}, error)
   GetState() State
   GetName() string
}

For now, this is enough to make the circuit breaker switch from closed to open whenever our threshold is exceeded. Therefore we need to count the consecutive errors. We will also send an alert whenever our state has switched to open to signal that our function is not healthy. To keep things simple, our alert will just be a print of the open state to the console.
Here is what our Execute function might look like.

func (c *circuitBreaker) Execute(f func() (interface{}, error)) (interface{}, error) {
   switch c.state {
   case Closed:
      res, err := f()

      if err != nil {
         c.handleError(f)
         return res, err
      }
   case Open:
      message := fmt.Sprintf("%v circuit breaker open", c.name)

      fmt.Printf("ALERT: %v", message)    
      return nil, errors.New(message)
   }
   return f()
}

func (c *circuitBreaker) handleError(f func() (interface{}, error)) {
   c.consecutiveErrors++
   if c.consecutiveErrors > c.strategy.threshold {
      c.state = Open
   }
}

This is basically all you need to reproduce the behaviour of an electrical circuit breaker in code.
Every consecutive call will result in an immediate error without even calling the original function. In a distributed system, this might keep you from stacking up calls to a downstream service and risking processing or memory overload. Nevertheless, this will keep the circuit breaker open forever. When we want to close it, because our downstream service is working again, we have to switch the state back manually. Therefore, we can implement a simple recovering mechanism. With Go’s fairly simple approach to concurrency, we can execute a go routine which checks if the function we want to execute is still returning an error.

To enable this, we update our strategy to hold a retryMax counter and retryInterval which specify how often and with what interval we want to check if a service is up again.

type Strategy struct {
   threshold     int
   retryInterval int
   retryMax      int
}

Now let’s add the new state to our Execute function and implement our error handling while in half open state.

func (c *circuitBreaker) Execute(f func() (interface{}, error)) (interface{}, error) {
   switch c.state {
   case Closed:
      res, err := f()
      if err != nil {
         c.handleError(f)
         return res, err
      }

      c.handleSuccess()
   case HalfOpen:
      return nil, errors.New("circuit half open. trying to recover")
   case Open:
      message := fmt.Sprintf("%v circuit breaker open", c.name)
      fmt.Printf("ALERT: %v", message)
      return nil, errors.New(message)
   }
   return f()
}

func (c *circuitBreaker) handleError(f func() (interface{}, error)) {
   c.consecutiveErrors++
   if c.consecutiveErrors > c.strategy.threshold {
      c.state = HalfOpen
      go c.recover(f)
   }
}

func (c *circuitBreaker) handleSuccess() {
   c.consecutiveErrors = 0
}

func (c *circuitBreaker) recover(f func() (interface{}, error)) {
   retries := 0
   for c.state == HalfOpen {
      // Open circuit breaker when recovering fails
      if retries > c.strategy.retryMax {
         c.state = Open
         return
      }

      time.Sleep(time.Second * time.Duration(c.strategy.retryInterval))

      // set state to closed if request is successful
      _, err := f()
      if err == nil {
         c.state = Closed
      }

      retries++
   }
}

Now we got it! Our circuit breaker can heal itself by checking upon the given function if it still continues to throw errors. Instead of switching to the open state immediately, we switch to half-open and fire up a go routine which executes the recover function. While the retryMax threshold is not exceeded, it will asynchronously continue sending new requests to the service to check if it is still returning errors. Meanwhile, the Execute function is still returning immediate errors to the consumer of the original function, thus preventing our service from overload. If the retryMax threshold is exceeded, we will set the state to open and send the alert we configured before. If our function stops returning errors, we reset the state to closed and the consecutiveErrors to 0. If you want to see this in action, you can have a look at the tests in the GitHub repository.

By now, you should have a good understanding of how this pattern might help you in a distributed system and you might consider it the next time you are dealing with systems with high throughput. If you have any
questions or any kind of feedback, feel free to leave comment, contact me or follow me on Twitter.

Go Big O

Ben — Mon, 09 Mar 2020 10:54:03 +0000

Software development has become widely accessible in today’s world. Besides the traditional academic way,
there are endless ways to study the concepts of software development.
Getting acquainted with modern software development frameworks is just as easy as watching an online course on platforms like Udemy or Coursera. As long as you have a computer with internet connection at hand, you can teach yourself software development with basically no money. So did I.

What is widely missed in these courses and often also underrated in the workforce are the basic concepts of computer science.
This post aims at getting started with these concepts, explicitly the Big O notation, which every developer should not only be familiar with but master it, if he or she wants to do a good job. Furthermore, having a solid knowledge of Big O will be fundamental to survive a technical job interview. Since I am currently learning Go as a new programming language, all examples will be provided in Go. You can find all examples and algorithms in this GitHub repository. The algorithms are implemented by me, so there will probably be more efficient implementations. Any feedback is welcome!

What is Big O?

Big O notation is a mathematical notation that describes the limiting behavior of a function when the argument tends towards a particular value or infinity. (Source: Wikipedia)

In other words, with Big O we are looking for a generalised way to describe how well an algorithm will perform when maximising its input. Therefore, we are not looking for the exact number of executed lines of codes but rather for the growth rate of the function in respect to executed lines of code. This can be in a matter of time as well as space/memory.
We will have a closer look at time and space complexity within this article.

An Example

To get a better sense of how Big O is calculated, let’s have a look at the classic (not so efficient) selection sort algorithm.
Here is a simple implementation in Go:

func SelectionSort(list []int32) []int32 {
   for i := 0; i < len(list); i++ {
      for j := i; j < len(list); j++ {
         if list[j] < list[i] {
            swap := list[i]
            list[i] = list[j]
            list[j] = swap
         }
      }
   }
   return list
}

We iterate over every element in the list and compare its value by iterating over every element after the given one.
Now, let’s call the size of the list n. First, we iterate n times over the list, then the inner loop iterates over the following elements n-1 times. This leaves us with n * (n-1) which is equal to (n^2 )/2 - n/2. In Big O notation this can be reduced to O(n^2 ). To understand why, let's have a closer look.

Best Case, Worst Case, and Expected Case

To describe the runtime of our algorithm, we actually have three different ways: the best, worst and expected case. To illustrate this, let's have a look at this implementation of the Quick Sort algorithm.

func QuickSort(array []int) []int {
    n := len(array)
    pivotIndex := n - 1
    return partition(array, pivotIndex, 0, pivotIndex - 1)
}

func partition(arr []int, pivotIdx int, leftIdx int, rightIdx int) []int {
    if len(arr) < 2 {
        return arr
    }

    foundLeft := false
    for leftIdx < rightIdx && leftIdx < pivotIdx {
        if arr[leftIdx] > arr[pivotIdx] {
            foundLeft = true
            break
        }
        leftIdx++
    }

    foundRight := false
    for rightIdx >= leftIdx && rightIdx > -1 {
        if arr[rightIdx] < arr[pivotIdx] {
            foundRight = true
            break
        }
        rightIdx--
    }

    if foundLeft && foundRight {
        swap(arr, leftIdx, rightIdx)
        partition(arr, pivotIdx, leftIdx, rightIdx)
    }

    if leftIdx > rightIdx {
        swap(arr, leftIdx, pivotIdx)

        partition1 := arr[0:leftIdx]
        pivotIndex1 := len(partition1) - 1
        partition2 := arr[leftIdx:]
        pivotIndex2 := len(partition2) - 1

        partition(partition1, pivotIndex1, 0, pivotIndex1-1)
        partition(partition2, pivotIndex2, 0, pivotIndex2-1)
    }
    return arr
}

func swap(arr []int, leftIdx int, rightIdx int) {
    swap := arr[leftIdx]
    arr[leftIdx] = arr[rightIdx]
    arr[rightIdx] = swap
}

If you are not familiar with the implementation of Quick Sort, I recommend this short video.
Quick Sort is a very efficient sorting algorithm which consists of recursively partitioning an array into two subsequent arrays around an initially determined pivot. The left array holds smaller values than the pivot, the right one bigger values. This implementation of quick sort sets the last element of the array as the initial pivot.

Now, let's have a look at what might happen
during execution.

Best case: All elements are equal. In this scenario, quick sort will just traverse through the array and return the array as it is. This leaves us with a complexity of O(n).
Worst case: The chosen pivot is repeatedly the highest element in the array. In this case, partitioning the array will only result in reducing the array by one element and will effectively end in a O(n^2 ) runtime.
Expected case: The above examples are rare scenarios which will, depending on the implementation of th algorithm, happen from time to time given a certain input. But on average, Quick Sort has a O(n * log n) runtime.

If you are not familiar with the Big O notation and you feel a bit confused by now, don't worry! We will take a step back
and have a close look at the most common scenarios of time complexity now.

Typical Big Os and their Complexity

Below, we can see a Big O complexity chart. It shows the different Big O types and their growth rate for an increasing
number of elements.

(Source: https://www.bigocheatsheet.com/)

In our leading example, we said, with Big O we can reduce a complexity of (n^2 )/2 - n/2 to O(n^2 ). The reason why we only focus on the dominant term of a function is, because it is dominating the growth rate in the long run.
In addition, we can drop the constants leading the dominant term. This might be a bit confusing in the beginning, but you can easily verify this behaviour with inserting very high values. The only thing which really interests us with Big O is, how the algorithm scales in a generalized way.

To demonstrate this, I will show some common examples of these Big O types and briefly explain why they perform as they do.

O(1)

Given a list, print the first element of that list. No matter how big that list is, we will always have only that one operation.
If there is any way, this is what we should aim for.

func PrintFirstElementOfList(list []int) {
   fmt.Println(list[0])
}

O(log n)

The classical binary search algorithm is a perfect example. Given a sorted list, search a number by cutting the list in half and comparing its value to the ends of the two halves. Halving the input with each iteration leaves us with a time complexity of log n.

func BinarySearch(a int, sortedList []int) int {
    left := 0
    right := len(sortedList) - 1

    for left <= right {
        i := int(right + left / 2)

        if a == sortedList[i] {
            return i
        } else if a > sortedList[i] {
            left = i + 1
        } else {
            right = i - 1
        }
    }
    return -1
}

O(n)

Traversing a list or array. One operation for each element. While it is still worse than O(log n), O(n) is fairly acceptable and easy to estimate.

func PrintEveryItem(list []int) {
   for i := 0; i < len(list); i++ {
      fmt.Println(list[i])
   }
}

O(n log n)

This one is a bit more tricky and famous case for technical interviews. One classical example for this case is the Merge Sort algorithm. At first, a list is divided into halves until we have only lists with one element (log n) and then all elements are merged together iteratively (n). It is more complex than O(n) but still not as bad as polynomials.

func MergeSort(arr []int) []int {
   if len(arr) == 1 {
      return arr
   }

   left, right := divide(arr)

   return merge(MergeSort(left), MergeSort(right))
}

func divide(arr []int) ([]int, []int) {
   length := len(arr)
   m := int(length / 2)
   left := arr[0:m]
   right := arr[m:]
   return left, right
}

func merge(left []int, right []int) []int {
   merged := make([]int, len(left) + len(right))

   i := 0
   for len(left) > 0 && len(right) > 0 {
      if left[0] < right[0] {
         merged[i] = left[0]
         left = left[1:]
      } else {
         merged[i] = right[0]
         right = right[1:]
      }
      i++
   }

   for j := 0; j < len(left); j++ {
        merged[i] = left[j]
        i++
    }
    for j := 0; j < len(right); j++ {
        merged[i] = right[j]
        i++
    }

   return merged
}

O(n^2 )

Whenever you deal with this pattern, be cautious. As in this example, when we print all possible pairs of an array of integers which leads us to a time complexity of O(n^2 ). Especially nested loops should let your alarm bells ring.
You should only consider these algorithms if you can estimate the input size.

func PrintAllPairs(list []int) {
   for i := 0; i < len(list); i++ {
      for j := 0; j < len(list); j ++ {
         fmt.Printf("%v:%v", list[i], list[j])
      }
   }
}

O(2^n )

These algorithms are extremely bad, since they rise meteorically. As in this example, for a recursive calculation of the fibonacci numbers every additional n doubles the growth of the algorithm. Just avoid these.

func RecursiveFibonacci(num int) int {
   if num <= 1 {
      return num
   }
   return recursiveFibonacci(num - 2) + recursiveFibonacci(num - 1)
}

O(n!)

The most trivial example for an algorithm with factorial time complexity is - well - factorial addition.

func FactorialAddition(n int) int {
   for i := 0; i < n; i++ {
      return factorialAddition(n-1)
   }
   return n
}

Time vs. Space Complexity

By now, we should have a good understanding of how complexity is computed and described. So far, we only looked at the time it takes to execute an algorithm and the respective growth rate. In computation, besides time, space is one of our concerns which must be analysed.
If we look at our example of the beginning, the selection sort algorithm, the space complexity is O(1). We only need our swap variable to keep the current value to be swapped even though our time complexity is O(n^2 ). Compared to the O of time complexity, we now have to analyse how much space in memory has to be allocated. In most cases, space complexity is not as high as time complexity.
To illustrate this, we will have a look at a simple example.

First of all, we need to know much memory is used by different data types:

Type	Size
bool, char, unsigned char, signed char, int8	1 byte
int16, short, unsigned short, wchar_t	2 byte
float, int32, int, unsigned int, long, unsigned long	4 byte
double, int64, long double, long long	8 byte

Now let's have a look at a simple sum function

// n is the length of the array
func sum(arr []int, n int) int {
    var result int // 4 bytes for result
    for i := 0; i < n; i++ { // 4 bytes for i
        result += arr[i]
    }
    return result
}

Our space complexity scales linearly with the size of our array. Besides the three variables result, i and n which take up 3*4 bytes of memory, each element of the array takes up 4 bytes as well. This leads us to 4*n + 3*4 bytes. Drop the constants and we are left with O(n).

Summary

By now, you hopefully have an enhanced understanding of Big O and its role in software development. Even though, in your daily
business as a software developer you will most likely not be confronted with this topic, it is important to inherit these
principals to be able to analyse your code and make a profound statement about its complexity.

If you have any questions on this topic, feedback or you have any ideas on new topics that you want me to write about, feel free to contact me or follow me on Twitter.
Besides this being a way of improving my own learnings, I enjoy technical writing, so I am happy for new ideas and any discussions.

DEV Community: Ben

Remote Onboarding as a Software Engineer

Remote onboarding is difficult

Lay the groundwork

Be humble and patient

Ask

Communicate outside of your team

1:1s

Learnings From Three Years Into Software Development Without A Degree

Find a sponsor

Work on your social skills and learn to adapt

Develop a constant learning habit

Favour fundamentals over frameworks

T-shaped skills

Take responsibility and get shit done

Promote yourself

Conclusion

API Usability Testing in 8 Simple Steps

The Timing

The Users

API Usability Testing in 8 Steps

Conclusion

Exposing microservices with an API gateway and the API composition pattern

Flexibility leads to complexity

An API gateway as the single entry point

Upsides of API gateways

Downsides of API gateways

The API composition pattern

Conclusion

Docker Hands-On Start for Developers - Part I

Contents

What Docker is and why you want to use it

Images

Getting started

Working with images

Containerizing your application

Containers

Conclusion

Explore Arrays & Slices in Go

Static Arrays

Declaration and initialization

Accessing elements

Searching elements

Time complexity

Slices

Declaration and initialization

Appending elements

Slicing a slice

Deleting elements

Inserting elements

Time complexity

Keep your service alive with a circuit breaker

Why you would want a circuit breaker in place

How does it work?

Overview

Implementation

Go Big O

What is Big O?

An Example

Best Case, Worst Case, and Expected Case

Typical Big Os and their Complexity

Time vs. Space Complexity

Summary