DEV Community: Niklas Engberg

How to fix the nasty ‘bash: ./wait-for-it.sh: Is a directory’ on Windows

Niklas Engberg — Sat, 29 Jun 2019 17:02:06 +0000

How to fix the nasty ‘bash: ./wait-for-it.sh: Is a directory’ on Windows 💪

Nowadays when building applications they’re usually built using a distributed microservice architecture that consists of many services. Many of those services have dependencies on other services, usually infrastructure that needs to be booted before. To manage multiple services in an controlled an easy way Docker and Docker Compose can help you.

Docker Compose is a tool for defining and running multi-container Docker applications. What is limited though is to know when a container is ready to start accepting traffic. You can control startup order of containers by using depends_on. So for instance:

services:
   service1:
     ...omitted
     depends\_on:
     - "rabbitmq"

   rabbitmq:
     ...omitted

This will only verify that the container is running before service1 boots. To ensure that you do not start your container before the dependency is ready to accept traffic wait-for-it is a good script to use.

services:
  service1:
    ...omitted
    volumes:
      - "./scripts/wait-for-it.sh:/app/wait-for-it.sh"
    entrypoint: ["bash", "-c", "./wait-for-it.sh rabbitmq:15672 -t 0; ./start.sh"]

  rabbitmq:
     ...omitted

This will make sure that the rabbitmq service is available on port 15672 before the next script is executed (in this case a simple start.sh-file). This approach usually works very well but you can run into trouble when you are mainly developing on Windows:

bash: ./wait-for-it.sh: Is a directory

In this post I will share how I have solved this issue. I’ve spent maaany hours troubleshooting so hopefully I can save you some time.

Wrong line endings

Your wait-for-it.sh file is using windows line endings (CRLF). Files created on Windows have different line endings than files created on Unix. If you are running Linux containers you need to make sure your script file uses LF and not CRLF. You can easily find out by opening the file in some modern text editor like Notepad++ or VSCode.

Line endings in repository

You probably version your software and when you are using git there is a setting that you can configure to handle line endings. If you are collaborating across platforms you usually have the setting configured to auto. That means that any time you add a file to the git repo that git thinks is a text file, it will turn all CRLF line endings to just LF before it stores it in the commit. Whenever you dogit checkout, all text files automatically will have their LF line endings converted to CRLF endings.

Make sure you have a present .gitattributes file in your repo that has this setting:

*.sh text eol=lf

This will tell git to apply LF line endings to all .sh files when checking out.

Shared drive

Make sure you share your driver(s). In the example above I am using a host volume mount to share files from my host into the container. To be able to do this you need to share your drive. You can do this in the Docker Settings by right clicking the icon in the tray.

https://token2shell.com/img/howto/token2shell/docker/docker-settings-shared-drives.png

In our environment (I assume that this might be the case for you) we are using specific local admin accounts for elevated permissions. To be able to share your drive you need authenticate using this account. When the password for this local account expires, Docker won’t tell you or prompt you again to enter your password — unless you untick the box, hit apply, tick the box and hit apply again. Frustrating!

Hopefully when you have fixed the line endings and shared your driver(s) your issue will go away and you’ll see something like this:

service1 | wait-for-it.sh: waiting for rabbitmq:15672 without a timeout

These are two things that might bite you. Hopefully I have saved you some time troubleshooting! 😆

My way of working with internal nuget packages

Niklas Engberg — Mon, 26 Nov 2018 14:36:01 +0000

Picture borrowed from https://cdn.pixabay.com/photo/2013/07/13/10/22/box-157074_1280.png

The company I’m working at is big enough to span over multiple development teams. To be able to share a common way of working and to prevent that the wheel is invented over and over we rely on shared nuget packages.

I am author of many packages and the main contributor of those and that requires me to spend some time maintaining them. To be able to do this I need a working method that enables me to do this efficiently.

In this post I’m going to share my way of doing this, step by step.

Version control

To version control the packages git is used. Git is a de facto standard and need no further description :)

Naming

Decide on a good naming convention that will work for you. We use a prefix with our company separated by points narrowing down. For instance: CompanyA.AspNetCore.Mvc would be our own package with custom classes for ASP.NET Core MVC.

Project structure

To stay consistent a project structure should be used. I base mine on this Gist by David Fowler. It is not strictly followed, but rather inspired by. For example, the test project is not separated into its own test folder, instead it is included in the src folder. This setup is done once, when I introduce additional packages I just copy the structure from another project. This structure could easily be extracted into a dotnet template.

Nuspec file

Follow thisreference by adding your nuspec file.

Tests

Make sure you cover your packages with unit tests. That will give you confidence that your code works as you expect it to work and will also help other contributors. I’m using xUnit and in some projects AutoFixture to maximize maintainability and reduce code in the Arrange phase.

Documentation

Add a README.md to the project that explains how to use the package. More on the README.md below in the section about Crafting the package. I am considering looking into Read the Docs as a complement.

How to introduce changes?

The workflow that is used is Git Flow which means that we do not directly push changes to the master branch, but rather to feature branches that we integrate against the develop branch. I am not going to go into detail about Git Flow here.

When a feature is done, a pull request is opened and stakeholders are invited. And hopefully the feature is merged back to develop and ready for the next upcoming release. The git tag should reflect the version to be released. I explain my process of doing this in the section Crafting the package.

Breaking changes or not?

So, what is a breaking change?

A change in one part of a software system that potentially causes other components to fail; occurs most often in shared libraries of code used by multiple applications — Wikipedia

If you alter the definition of an existing method that you are exposing to the consumer or remove properties from an object that is a breaking change. Please make sure you have a consuming test client targeting your packages so that you can verify that the changes you’ve made are correctly done.

When it’s time to update an already existing package you need to take your consumers into consideration. You need to ask yourself if it’s a breaking change or not that you are introducing. If it isn’t, then it’s fine. That change should then only alter the minor and/or patch version.

If you are about to introduce a breaking change you need to keep support for the deprecated feature at least one version before removing it. I tend to do it this way

Add the new feature
Deprecate the old feature by adding an ObsoleteAttribute and describe how the consumer should migrate.
Bump the minor version and release a new version

In the next major release, remove the obsoleted feature. In this way the consumer is given some time to migrate their existing code to use the new version.

Crafting the package

Before publishing a new package I make sure that

The dependencies in the nuspec file is updated and that the supported target frameworks are in place
The README.md is updated with new functionality and examples
Versioning is correct

Dependencies in nuspec

There is a section in the nuspec file manifest that enabled to specify the dependencies that the package has. Even if packages only support one target framework they are included in a dependency group. That makes it easy in the future to add additional target frameworks.

Updating the README.md

To make sure that I update the README.md so that it is easy to follow I use Markdown Live Preview when editing it. That gives me a visual editor on how the file will be displayed. I then copy the content into my README.md and commit it.

Versioning the package

I’m using semantic versioning. To version the package the numbers must be consistent in *.csproj, .nuspec and in the git repository. For this I’m utilizing custom made PowerShell scripts.

This is executed by running it from the src folder.

.\bump-assembly-version.ps1 -Version 0.0.1

This is excuted by running it from the src folder.

.\bump-nuspec-version-ps1 -Version 0.0.1

I then commit those changes

git commit -am "Incremented project version"

Before pushing anything I make sure that my repository is tagged with the same version so that we have a tag that corresponds to the release

git tag -a 0.0.1 -m "Release 0.0.1"

(All of the above could easily be built into one single script file. I still do it separately but might consolidate in the future :))

And then push the tag

git push origin 0.0.1

And the code

git push

Publishing the package

There are different ways you can do this. Either locally using the nuget CLI to push the package to your feed, or let your CI service do this work.

I am letting Azure DevOps and Azure Pipelines do this. The build process consist of standard .NET build / run tests tasks. If you are interested in how that might look you can find examples here.

Finally the nuget package is crafted based on the nuspec file. The package is then pushed to our internal nuget feed. For this to work seamless a service connection to our nuget feed is added in Azure DevOps. The package manager that is used is ProGet.

When this package published I usually notify all the developers on Slack that new packages are available and take any further discussions from there.

This is a process that is working for me. If you have any opinions or other ways of doing this I’d like to hear. What features should be extracted into nuget packages really depends on your company and how you share code between team. I prefer to have a consistent way of doing things and I’ve noticed that we save a lot of time in our projects if we can reuse good things that others already have built.

If you liked the post, you know what to do! 👏

MediatR behaviors to validate API Resource existence

Niklas Engberg — Wed, 12 Sep 2018 14:42:09 +0000

MediatR logo borrowed from https://bit.ly/2N4Jz6S

When building robust APIs it it really important with validation and descriptive error messages and status codes.

In this post I am going to show you how you can get rid of duplicated code in your Request Handlers by adding MediatR behaviors to take care of validation for resource existence.

MediatR is an in-process messaging library in .NET that supports requests/responses, commands, queries, events etc. Go check it out if you haven’t.

In MediatR there is something called behaviors which allow you to build your own pipeline for requests. I use them for cross cutting concerns such as logging and validation of requests parameters.

Recently when some endpoints were developed for a microservice I struggled with duplicated code in my Request Handlers that was necessary to have in place.

To give you an example lets start with a couple of API endpoints, and also the implementation of the Request Handlers with the duplicated code.

The endpoints in this example are:

PUT /groups/{id}

POST /groups/{id}/users

The controller:

Both endpoints need to verify that the id-parameter actually matches a given group to be able to update a group / add user to a group (in my example I’m using EF Core as persistence, but you can safely ignore that).

As you can see there is an if-check for group existence in both the handlers.

How should we get rid of this duplication? Wouldn’t it be good to use some sort of “validation” here as a pre-step, so that the Request Handlers could look like this?

Thank behaviors for that! Let’s introduce an interface that both the requests: UpdateGroupRequest and AddUserToGroupRequest implement.

Then create a behavior implementation class with generic constraint to the interface defined above so that the behavior only applies to requests implementing the IGroupRequest interface.

And a registration example using Autofac:

Now we can rewrite our Request Handlers without the if statements, but still get the benefit of validation!

Lets imagine we now add a couple of more endpoints:

GET /groups/{id}

POST /groups/{id}/permissions

Add the IGroupRequest interface to your requests and you will get this feature out of the box. Pretty simple! I find this kind of approach really useful. What do you think?

We have to keep in mind that there are both some cons and pros to this approach. What I’ve found out is:

Cons

Multiple roundtrips if you have a database that you validate the existence against.
Validation as a separate class which might confuse some of the developers

Pros

Cleaner code in the actual request handler
No duplicated code scattered all over all Request Handlers that need to validate an entity’s existence

If you liked this post. You know what to do! 👏

Apply the decorator pattern in .NET using Autofac

Niklas Engberg — Tue, 06 Mar 2018 05:21:33 +0000

Russian Dolls — metaphor to extend a given functions behavior. Photo borrowed from Maxpixel.

To follow good design principles and structure when you develop applications is something I advocate.

As the development process goes on we add new functionality to our applications: it could be a feature request from your client or it could be improvements that you’ve suggested. No matter what, these upcoming changes will also change the already given codebase. When changes are introduced we want this to be as easy as possible.

In this post I’m going to show you an example of how you can utilize the decorator pattern with the powerful IoC container Autofac in scenarios where you want to extend functionality to an existing feature.

Lets say you are about to build a service that serves products. We will call this service IProductRepository and for simplicity we only use one method called GetById. It could look something like this with an implementation connected to it:

Normally you would register your concrete implementation XmlProductRepository as IProductRepository with the following code line to support the dependency inversion principle:

Lets say that you after a while need some logging when products are fetched. How do you accomplish this?

It is tempting to just modify the implementation XmlProductRepository by injecting a logging service like this:

But that violates the open/closed principle. The principle states that you are not allowed to modify classes, but you can extend them. Here the decorator pattern is really handy.

Instead of cluttering down your original implementation you extend it by adding a logging proxy, and register it as a decorator of it.

You will have to modify one code line though, and that one is the registration of the original implementation. It will need to be a named registration now so we can refer to it when we register our decorator:

And then connect and register the decorator:

The key that we use to refer to the implementation here is productRepository. When you have this code in place the first class that will be invoked when you inject your IProductRepository is the LoggingProductRepositoryProxy, and as we have injected the inner implementation to it it will just work as a proxy (russian doll style) following the open/closed principle.

Lets say that you experience your XmlProductRepository slow and you need to cache your products for a while. How would you implement that? It is pretty easy to just add another decorator on top of your logging implementation:

Pretty flexible!

The decorator pattern in really powerful when it comes to supporting cross cutting concerns within applications.

Did you find this post useful? You know what to do! 👏

Find your Circle of Competence — how to do well as a Software Architect

Niklas Engberg — Sun, 04 Mar 2018 16:52:36 +0000

As a Software Architect you are the link between requirements and implementation. It’s an ongoing challenge to make decisions that have a solid ground, decisions that enables future features to fit into an existing ecosystem that well aligns with good software principles such as SOLID, DRY, YAGNI to name a few.

Building software requires constant improvement work. New type of technologies bubbles up every day and it is hard to keep you updated with everything that is happening. The more you learn, the more you realize how little you know, as the famous Socrates said. My experience of roughly 3–4 years as a Software Architect (by title, not if you ask me in person ;) has taught me to delimit myself into specific areas instead of jumping on everything that at first glance seems awesome and hot.

The Circle of Competence

I recently read about something called the Circle of Competence. The Circle of Competence comes from one of the best investors, Warren Buffet, and is the subject area that matches a person’s skill or expertise. This principle originates from investing money within areas that you have a solid understanding in.

The reason I’m mentioning the principle and why it caught my focus is that this kind of principle does not only apply to financial decisions, but really to everything. Including Software Development, and I’ve applied it.

The more experienced you get, you realize that you can’t do everything well. I don’t think this is anything new or spectacular but to zoom out and evaluate yourself and how you are doing is important. How do you stay focused on the things that matters to you? I would love to hear!

The Circle of Competence applied

In October last year I took the opportunity to join the financial industry as an architect / developer for a global bank. I previously worked in a similar role but in another industry so this was something completely new for me, business and company size-wise.

My former job and company were ahead in many technical areas, but rarely had those challenging projects that enabled us techies to use our toolbox fully. That was one of the reasons I left. Anyway, that’s not the point here. :)

The point is that when signing this new job, I knew that there’s going to be a lot of challenges such as:

define visions and standards within the software area
set coding standards and best practices

Those challenges are quite wide and requires a lot of decisions to be made. How do you make sure you can accomplish those? I knew that I could benefit from my experience that worked out for us in my previous work to some extent but some kind of system would be great.

Why not see how to use the principle Circle of Competence to my situation.

Okay, so this is a picture of how the circle looks:

But I think it is more accurate like this:

Split what you know into your comfort zone and what you don’t outside the zone.

Comfortable in

Well, to start with I have mainly been focusing to back-end development and especially the .NET platform.
I advocate test (TDD), have been working hard to apply clean code and good software principles etc. So within my Circle of Competence I can add this.
I have experience in configuration build pipelines and continuous deployment.
I am also quite good at communication and to transfer technical terms to non-technical people verbally. So to communicate those things should be doable.

Not comfortable in

What about networking, VIPs, bastion, firewall rules, DMZ and all this? Nah, I’m not really comfortable there.
Core Banking Solutions? No.
UX and front-end
What about bank regulation and laws? Noo, not really, shivers…

Okay, so that’s it. I’ve now divided what I know and what I don’t. This is roughly an overview and examples just to give you a hint of how I split things up. The list in reality is much longer and complex.

By doing this little exercise I can then look back at the two challenges mentioned above and answer the question:

How can I define visions and standards within the software area and to set coding standards and best practices given what I know and where I can contribute?

If I look at the perspectives of the challenges with my confident zone in mind the probability that I will make good decisions increases than if I should start focusing on aspects outside the circle.

Outside the circle, which are the areas you do know you are not an expert in, invite others to help you out, and accept that. Let them be the experts teaching you. This will help you expand your circle and hopefully you can expand others’. That is one of the philosophies I’m living after. It keeps me focused on the areas that I know I am good at.

Note that this has nothing to do with what you want to learn but rather a method to keep you focused in the areas you know you can do well in.

How do you stay focused as a Software Architect?

Did you like this article? You know what to do! 👏

Guidelines how I developed my personal skills (and helped others) by initiating a side project

Niklas Engberg — Mon, 16 Jan 2017 16:57:36 +0000

This post is x-posted from Medium here. If you like the article, please feel free to give some applause and follow me! :)

Every developer should have a side project I’ve heard. And yes, I can agree that it can be good for your personal career but it isn’t easy to get started with. If you are one feeling that, or if you are just interested in my story and how it became reality for me, please continue reading!

I always try to improve my programming skills in the best possible way. As a developer you are solving problems most of the time of your day-to-day work. This problem solving tend to be tied to a specific stack or programming language that you are already familiar with.

It is nothing bad about that but to broaden your skills outside your comfort zone you need to put in some extra effort, and that is hard. You can play around with hello world-examples and coding katas but for me, it took me nowhere.

I currently work at a digital agency as a developer and one day I identified a problem. Our company have invested well in health and wellness in form of a gym and instructor-led training activities such as Yoga, Pilates, Circuit Training and much more.

A year ago (before my side project was born) to be able to participate in these occasions you had to sign up by communicating with our agency assistants. They then took notes for the participants. They kindly informed you back either: yes, you have been enrolled / disenrolled or: sorry, this occasion is full.

I noticed this process and started to think that there should be an application that replaced this work and that it would fit perfectly as a side project. And so it did. We now have a fully digitalized booking system for training activities!

It’s been almost a year since I shipped the product and recently I evaluated my work and here I’m going to share my findings from this project, and what have been essential for getting things done.

Find a problem to solve

What made sense to me was that my product would actually solve a problem. This gave me momentum and helped me motivate myself.

This is probably the hardest one but it will make you commit. It does not have to be a big problem, it can basically be anything. If you are employed by a company, a recommendation would be to check with them. Do they feel the absence of any technical product / tool that can help them? Ask!

Github with it’s open source culture is full of different projects that need contributors. Check out Up For Grabs.

Nothing is perfect

For the product that I built I started to think big, wondered, and felt that I needed to deliver something that was perfect.

Let’s face it, your work will never be perfect. At some point you just have to ignore that when starting a new project. We would all love the perfect idea that no one have ever thought of or implemented yet, but there is no such idea. This is something that I’ve learnt throughout the process.

Keep it simple, don’t get lost in details

In my case with the booking application I decided to start off really simple: to initially replace the manual work by implementing an easy web interface. All other features would have been improvements (since nothing existed at this point). I just needed to keep telling myself that. And when I finally got comfortable about it, I started to become productive.

Side note: The application has been running for almost a year now and no one has ever complained about the minimalistic user interface I initially shipped. Why? Because the application does what it should and doesn’t need a fancy UI.

It’s just a matter of execution

I’ve seen ideas that just remains as ideas, and never really turns into something tangible. My recommendation is that if you believe in something, you should just go for it! Make a plan.

This goes hand in hand with keeping it simple. If you cannot manage to get your product up and running pretty soon, it will probably start to decay.

Start with the simplest solution of the problem you are facing. Ship a minimum viable product that fairly soon hits the market. This will give you an early response to your work.

Evaluate your work and be proud!

What I never did until recently was to look back and analyze my work. What have I actually delivered? Learnings? Could I have done anything differently? These questions are good to ask yourself.

Besides the findings that I’ve presented this project has taught me a new programming language and how to organize my work. And as the headline for this post states it has also made a huge impact on the employees since they easily themselves can manage training activities. So, check on that :)

I try to keep it as a habit to continuously improving the product as much as I can. And it is a lot easier now since the product is being used by others, so I get their input and their ideas for free— which in turn generates more work :)

I would love to hear your thoughts. What have worked out for you?

If you are interested in details about what technical products I used for this application I might write a more technical post about that :)