DEV Community: Andrew Cahill

Are We Really Doing Continuous Integration?

Andrew Cahill — Sun, 28 Feb 2021 16:50:32 +0000

Preamble

After seeing some recent tweets and blog posts by Martin Fowler and Kief Morris on the topics of Continuous Integration, Pull request reviews and branching and how we may not be doing it quite right at least internally within an organization(references below), led me to explore this practice a little further.

First, let’s define what Continuous Integration is

Continuous Integration is the practice of merging all developers working copies to a shared mainline daily if not multiple times a day.

I really like this idea as it brings with it very clear benefits including

– reducing the possibility of large merge conflicts

– reduces peer review effort as the code to review is smaller and thus easier to reason about

– smaller commits make it easier to find issues and roll back if necessary

A little background

I have been a proponent of feature branching and pull request review ahead of merging(pre-integration review) within an organization’s team for some time. The reasons are as follows:

Feature branching allows the reviewers to review a PR that encompasses the entire body of work that is aligned to a feature enabling them to know the scope of work associated.

Pre-integration reviews provide value in decreasing potential issues that otherwise would not have been caught by say unit, integration, functional tests and analysis, this is especially true if the codebase is a brownfield project that may not already contain a high level of test coverage.

The Issues

However, we noticed quite early on that there were some drawbacks to these practices such as:

Feature branches required an entire feature to be completed before a PR be created, well if a feature required several days to complete this right away violated the CI principle and thus does not bring with it the benefits above.

Pre-integration reviews incurred delays in completing tasks due to relying on team members being available to perform this review at the time of task completion, as team members were typically busy completing their own tasks we typically waited for them to complete their tasks to avoid them context switching unless there was a dependency on this task. So this meant the developer who completed the task would pick up a new feature to switch to and wait for the reviewer to be free, which may be when they develop in this case is now busy so no matter which way you went there was a hit on throughput/productivity and led to tasks taking many days to complete again violating CI.

As you can see these practices may not be the most effective way of going about doing things especially as speed is of priority which almost always is the case.

One exception to this where such a level of rigour may be required for compliance reasons.

So with feature branches and pre-integration review causing such issues I was curious about solutions to these problems or if these were simply a trade-off to accept for apparent higher quality. After reading further into the aforementioned blog posts there were a few good thoughts that made me rethink my ideas around quality oversight.

Possible solutions

Well, I think we have to look at how we may have got to perform such practices -> Open Source.

Pre-integration review and feature branches make complete sense for open source development as you are working with external contributors who may not have the same standards, styles and objectives as you, wherein an organization it would be assumed this would not be the case.

So what are the alternatives to these practices in an organization setting?

Instead of performing pre-integration reviews, we could apply pair programming or periodic code reviews:

Pair programming, one of the practices in XP, suggesting 2 programmers work on a task together one acting as a developer and another acting as a reviewer in real-time reducing the time of developing something incorrectly and re-implementing. Shortening the development – review – improve cycle

Periodic reviews where you can perform scheduled reviews of committed code, this takes an assumption that code committed is of a reasonable quality to be committed through training etc. and if there are issues they can be fixed through subsequent refactoring.

Instead of creating feature branches we could allow partially completed features to the codebase. This way you can push your changes as frequently as you like and be able to adhere to the CI principle or merging at least once per day. A side benefit of performing this rapid commit style is that it encourages refactoring, and following a clean as you go mindset. What I mean is that you can make any refactoring you notice during your development whether it’s part of the feature you are working on or not.

You might say well how do I do this as it will not be a fully working feature and we can’t release partial features to end users, well one popular approach to get around this exact issue is to begin using feature flags and switch on this feature & make it visible when ready.

Performing refactoring like this is a safer approach for two reasons, one is that it provides you with a better chance of getting to fixing an issue sooner rather than later before complexity increases or it becomes a more serious issue down the line and two hoping to get time at a later time to apply refactoring is a big challenge get buy-in as delivering features pressures will always be present and this is a good thing as we don't really want things to get quiet do we!

Summary

Again the above is not to say one way is necessarily better than the other, depending on your circumstances one way may be more appropriate than the other. This post is just to provide further options if you find yourself in a similar situation where such processes may be hampering your delivery velocity.

References:

https://infrastructure-as-code.com/book/2021/01/02/pull-requests.html

https://martinfowler.com/bliki/RefinementCodeReview.html

https://martinfowler.com/bliki/PullRequest.html

https://martinfowler.com/articles/branching-patterns.html

Adding resilience to my http calls using Polly

Andrew Cahill — Sun, 31 Mar 2019 20:26:41 +0000

As I am building out my MyReverie(https://github.com/andrewcahill/myreverie) project in GitHub, I wanted to add some resilience to my http calls. I had heard of the Polly framework that made doing this extremely easily.

Polly is an open source framework that provides resilience capabilities on transient errors that you can leverage in your own applications thus removing the burden of having to write this code out manually yourself. Types of capabilities include Retry, Circuit Breaker, Fallback among others.

If you were to try write this code yourself and for it to be done correctly can be a little tricky and error prone due to inconsistencies that may develop over time as these are cross cutting concerns it should be abstracted to a library to be reused – so why reinvent the wheel!

Type of resilience’s you may want to consider could be against network outages or systems being busy attending to other work, this is particularly important in larger microservice applications that rely heavily on intercommunication of services so instead of timing out or returning with an error wouldn’t you like it to retry and handle this scenario a little more elegantly and smarter.

For my purposes I wanted to simply add an ability that if the endpoint was for some reason unavailable that it would ‘Retry’ a certain number of times and not simply fail after the first attempt. It is a simple use case but it allowed me to easily become familiar with this framework.

To get Polly up and running is quite straight forward.

In my startup class I define the http client

services.AddHttpClient("api").AddTransientHttpErrorPolicy(p => p.RetryAsync(5));

This effectively means to retry the http call five time before returning with a failure response. I chose 5 as a reasonable amount of times to retry but you can choose whichever number you wish.

I need to of course add the Polly package(in my case I am using v2.2.0):

Microsoft.Extensions.Http.Polly

add then add the using statement

using Polly;

In my service class within my web project where I make my http calls to the API I inject ‘IHttpClientFactory’ using Dependancy Injection which comes out of the box with .Net Core as a constructor parameter.

IHttpClientFactory httpClientFactory

I then assign this parameter to a global variable ‘_httpClientFactory’ in the constructor as below

_httpClientFactory = httpClientFactory;

I initialize a new HttpClient object to the client I defined in the startup class by

HttpClient client = _httpClientFactory.CreateClient("api");

after which I make the http calls as you would normally do, but this time incorporating using Polly.

In my case it is:

await client.GetStringAsync(_appSettings.GoalsUrl);

Now if this http call fails it will try another four times before returning to the caller thus improving the resilience of my system in case of intermittent connectivity issues!.

The post Adding resilience to my http calls using Polly appeared first on Avid Developer.

My Git Command Line Toolbelt

Andrew Cahill — Thu, 18 Oct 2018 18:22:36 +0000

Overview

Lately I have been getting a little more familiar with the Git command line and I have to say I am liking it. The reason why I had stayed away from it until now was that I found the GUI(SourceTree in my case) to be a safe/easier interactive medium for something I was not at all that familiar with. However I realize the CLI is faster for issuing commands so I made a conscious effort to start familiarizing myself with it.

The idea of this post was to demonstrate and explain the most frequent commands I typically use and will hopefully help others interested in understanding the Git CLI basics but may be a little apprehensive.

I will demo operations from both a repository owner as well as a repository contributor perspective.

Disclaimer: I know there are many more commands that I am excluding however I just wanted to focus on what I believe are the most frequent most used day to day commands for a beginner.

I wanted to not only list the commands with their explanations but provide a real example against a sample application for better context.

To follow these examples, you will need to have some pre-requisites installed.

Pre-requisites

Git installed on your machine – you can download from here: https://git-scm.com/downloads
You have the .Net Core SDK installed(as I will be demonstrating these command on a sample .Net Core application – if you do not have it you can browse to : https://www.microsoft.com/net/download)

What I use (in case you were wondering from the images)

PoshGit – nice PowerShell module to allow me to see git status from the prompt(https://github.com/dahlbyk/posh-git)
PowerShell through command line – easier to see command colors in black background

What I will cover

I will list the commands with their explanations, then apply them on a sample application which I will build completely from scratch using .Net Core.

I will be 100% in the command line from creating the application to interacting with Git outside of using an editor to update files.

The .Net Core application that I will create will be a simple console application to keep things as light as possible and focus on Git rather the application itself.

Commands I will be using:

Git Config This command is to set your initial one-time configuration, I will use it to set my Identity (Name and Email). Note: You can set either global or project specific configuration, meaning the scope at which the configuration takes affect. I will apply it to the new project.

If you want to set the configuration globally you simply add the ‘—global’ switch.

Git Init

This initializes our new project directory to be a git repository.
Git Add

This command adds your files from your working directory to staging or to the git Index to be committed.
Git Status

This allow us to view the files which have been updated, added or deleted.
Git Commit

This command commits your changes ready to be pushed to your repository.

You add the -m switch to attach a message to your commit.

Tip: You can bypass the add command above and combine the add command to the commit command as one single command by adding the ‘-am’ switch –> I will demonstrate below.

Git Push

Pushes committed changes to your remote repository(GitHub in our case)
Git Checkout

Allows you to switch to a different branch, you can also add the ‘-b’ switch to create a new branch based off your current branch –> I will demonstrate below.
Git Diff

Finds differences between of files before and after changes.
Git Merge

Merges changes from one branch to another.
Git Stash

This command allows you to store away changes that you may not want to apply right away.
Git Clone

Clones from a remote repository to a local repository
Git Pull

Pull all the latest changes that may have been applied on your remote repository and applies them to your local repository.

Okay, let’s begin

First, we need to create our .Net Core Console application to work with.

Open up PowerShell, I will open it within the command prompt – by simply typing ‘PowerShell’ within the command prompt – and you will see below that PowerShell is listed on the top bar.

Next just browse to a directory in which you want to create the new project, in my case I will create in ‘c:\dev\github’.

Now create the application and name it ‘GitCommandLine’ by typing ->

‘dotnet new console -n GitCommandLine’.

You should see the below..

Now that we have our dotnet core console app created, now let’s start with Git.

First, Lets’ navigate in to our project directory we just created.

Git Init

Now we want to initialize our new project as a Git repository so we will want to use the ‘Git Init’ command like below.

Git Config

Now we want to set the config specifically the Identity (user name and email) to associate with our commits, I will use

User name: John Doe

Email address: john@johndoe.com

Note: The configuration scope is just this project so I excluded the –global switch.

Git Config –l

Let’s view the current configuration to confirm out setting for applied correctly by issuing the ‘git config –l command’

and it looks like they did as you can see near the bottom.

Git Status

Let’s call a Git status to see the current changes in our repository.

What this is telling us is that we have not specified any files to be included in the next commit.

Git Add

As mentioned above we can either call a Git add as a separate command or within the commit command, let’s call it separately for now.

Here we specify what is to be added to staging, we are adding everything with ‘git add .’ we could have specified individual files if we wanted to.

Now let’s rerun our git status after applying the git add command.

You will now see the output is different – stating we have new files ready to be committed.

Git Commit

Let’s go ahead and commit them, we will add a message to our commit let’s call it ‘Initial commit’

Git Push

Now we will want to push our local repository changes to our remote repository (GitHub).

FYI: I have already created a new repository in my GitHub account(https://github.com/andrewcahill/GitCommandLine).

In order to push I will need to specify my remote repository details, specifically where it is located so I use the below command.

‘git remote add origin https://github.com/andrewcahill/GitCommandLine.git’.

Now we will issue a push command with ‘git push -u origin master’.

Great, now we have our repository in GitHub.

Now let’s create a new development branch, make some changes in this new branch then push them to GitHub and then finally merge these changes to master.

Git Checkout

We will use the ‘-b’ switch to specify a new branch and call the new branch “development”

I will make a minor change to our Program.cs file and change the “Hello world!” output to “Hello GitHub!”

Quick tip: To view content of a file there is a PowerShell command ‘Get-Content’ which will allow you to do exactly that.

We can use this to view a before and after, like below..

And after the change..

Git Diff

We can use ‘Git diff’ to view the changes that will be applied on our next commit

Now let’s commit them, why not this time add our ‘add’ command within our commit command and add the message “Updated output”.

Now let’s push our changes to a new remote ‘development’ branch. In order to do this we will want to create our new remote branch by issuing the command ‘git push –set-upstream origin development’

Note this is only a one-time operation

Git Merge

Now let’s merge these development changes to master so both are in sync.

This is a two-step process first lets checkout or switch to our master branch then we will call merge – like below.

Now we can see the master branch has an update pending so let’s push this change to our remote repository.

That’s it, well at least from the point of view of the creator of the repository, what about the point of view of a contributor, they may want to make contributions to your repository. In this case the process is quite straight forward.

All we really need to do is clone down this repository, to a directory of our choosing, make the changes and push them back up.

Let’s do that..

I will create a new folder under my github directory called ‘GitCommandLineContributor’.

Git Clone

Now let’s navigate in to our new directory and clone using ‘git clone https://github.com/andrewcahill/GitCommandLine.git’.

Typically changes are done within either the development or feature branches so let’s switch to the development branch here, by issuing ‘git checkout development’.

Now as a contributor lets make another change to the Program.cs file –> we will change from “Hello GitHub” to “Hello from contributor”.

Okay, now let’s go ahead and commit and push as before.

Git Pull

Now go back to our owner viewpoint(c:\dev\github\gitcommandline) in my case, switch to development and pull down any changes that may have happened by contributors by issuing ‘git pull‘.

And we see after we perform a pull we can see new changes being applied.

Now let’s merge these contributor changes to master.

So we issue ‘git checkout master’ & ‘git merge development’ and finally ‘git push’ as before.

Git –Help

At any point you can run a ‘git –help’ command which lists all the available git commands available to run. As well as that you can run a ;git [command] –help; to bring up a webpage detailing further information on that specific command. For example ‘git commit –help’ –> pretty cool.

Git Stash

Finally one last command I wanted to show was the Git Stash command, as I have on more than one occasion forgotten to switch to the correct branch before making changes, for example accidentally making changes on master instead of development – this command really helps me out. As I usually try to make my commits small and often I had previously ended up simply copying my changes to the new branch –> a pain and error prone.

How I will show it is:

As an example we accidentally make a change to master that we wanted to make to development so we stash these changes, switch to the correct branch(development) and apply the changes to the development branch.

First make sure we are in the master branch by running ‘git status’

Lets again make a change to our Program.cs file with “Hello owner!” in our output.

Again let’s confirm our changes..

Now we realize this change should have been in development, so, lets stash these changes.

Checkout development and run ‘git stash apply’.

Now perform the same steps again(git commit with add – git push – git checkout master – git merge development & finally git push) as below..

Conclusion

That’s it a top to bottom of all my frequent Git commands, I hope somebody may find this overview useful.

Resources:

For further information on Git commands, explanations etc. check out: https://git-scm.com/

The post My Git Command Line Toolbelt appeared first on Avid Developer.

Adding Tiered Compilation in .Net Core 2.1

Andrew Cahill — Mon, 03 Sep 2018 09:03:56 +0000

One of the many great features of .Net Core is its performance, which it certainly excels at.

A few weeks back I seen a new performance feature, well preview feature in .Net Core 2.1 – called “Tiered compilation”.

What is it

Well you could say at its core in enables applications to not only start faster but also run faster. It has a lot to do with the JIT( J ust I n T ime) compilation process – first let me explain a little about JIT.

JIT

Leaving aside AOT compilation for now, The .Net CLR typically uses the JIT compiler to compile the MSIL against the CLR using CIL metadata to machine native code for the target machines architecture.

It performs this process either on startup with full assembly compilation when its loaded or on demand at runtime when a particular method is called – what happens here is basically that when a method is called first time it gets JIT compiled then on subsequent calls the calling code effectively just references the native/Jitted version of the method in memory.

Typically in JIT each method would be JIT compiled once either using aggressive optimization on startup sacrificing startup time but benefitting from steady state runtime performance, or simple optimization at the start sacrificing steady state performance throughout runtime. There was typically only a single choice.

Back to tiered compilation

With tiered compilation you can JIT compile the same method more than once to produce a more optimized version depending on its usage, it is hot swapped at runtime automatically for you making the decision for you.

On startup it performs a basic compilation(tier 0) but if the method gets used frequently the method is recompiled on the background thread to produce a more optimized version(tier 1) for you to improve the runtime performance. This is sometime termed called adaptive optimization.

This in effect produces both fast startup and fast steady state performance.

Another benefit of tiered compilations is faster steady state with applications that use precompiled images shipped with the framework code of .net core, with tiered compilation it can produce more optimized versions of the code at runtime.

How

In order to try it out for yourself, there are a few options

Assumes you have access to or are building the source code, then it is a trivial change to your csproj file by simply adding the MSBuild property :

like below

If the app was previously built and you have no access to the source code but you have access to the appsettings file simply add

to the configProperties.

For example:

Resources:

You can check out official Microsoft article here: Tiered Compilation
Check out the GitHub thread indicating performance comparisons of Tiered vs non-tiered apps : GitHub

Happy coding!

The post Adding Tiered Compilation in .Net Core 2.1 appeared first on Avid Developer.

Add versioning to an Asp.Net Core Web API

Andrew Cahill — Fri, 13 Jul 2018 05:11:40 +0000

There are occasions you would like or need to add versioning to your API in order to specify availability or non-availability of functionality.

There are a few options available such as Query String, Url based as well as Http header.

For my purposes I am going to talk about the Url based approach, as I have implemented this option in my MyReverie repo due to a matter of preference of being cleaner and more explicit.

First things first we will need to go ahead and install the Nuget package:

Microsoft.AspNetCore.Mvc.Versioning

Within the Startup class ConfigureServices method add the below:

services.AddApiVersioning(v =>
{
    v.ReportApiVersions = true;
    v.AssumeDefaultVersionWhenUnspecified = true;
    v.DefaultApiVersion = new ApiVersion(1, 0);
});

The above three properties are particularly useful in that they allow default versions to be set if none are explicitly set as well as providing header information on versioning that are supported.

ReportApiVersions : we are requesting the header to respond with supported API versions

AssumeDefaultVersionWhenUnspecified : set the default version if none are explicitly requested

DefaultApiVersion : Set the default version to 1.0 if default is used

To better illustrate the first property above see below. I used Postman to see the response header with supported versions listed, which I highlighted

Now all I have to do is simply add and update the controller attribute in my case

At the moment as I am still on version 1, I will add the attribute

[ApiVersion("1.0")]

and update the route attribute like so:

[Route("api/{v:apiVersion}/[controller]")]

So the result of the Controller class(GoalsController in my case) attributes should look like the below

Now let’s test our F5 and browse to our API endpoint with suffix : /api/1.0/goals, in my case below.

That’s it my API is now versioned v1.0

The post Add versioning to an Asp.Net Core Web API appeared first on Avid Developer.