DEV Community: Mac

The benefits of asynchronous communication

Mac — Tue, 31 Mar 2026 04:39:00 +0000

To me the main benefit of remote work (when done right) has always been the necessary focus on asynchronous communication. In these times when, for some reason, remote gets a bad rap and companies left and right implement new Return To Office (RTO) policies, the old synchronous ways seem to get a big revival. "People need to talk to each other", "it's just common sense". The idea is simple: put everyone together in an open-plan office and collaboration will naturally improve. Right?

Well, not quite, according to overwhelming evidence. Asynchronous communication is more demanding and requires discipline. But it leads to better focus, better decisions, and better knowledge sharing, whether remote or in the office.

Kill no flow
Share knowledge, include everyone
Have deeper interactions
Make history

Kill no flow

Constant interruptions lead to a higher workload, more stress, higher frustration, more time pressure, and effort. And they have a strong negative impact on work quality. It takes time and energy to get into a state of flow, and a very short interruption can ruin it all in an instant, requiring significant time to recover (also known as Carlson's law). Faced with only the threat of interruptions, many people will tackle easier, shorter tasks instead of delving into the most complex ones.

Asynchronous communication allows everyone to manage their most precious resource more effectively: time. You can check your messages when you are done with your current task: no-one expects you to answer immediately. In exchange for this mutual respect, everyone has to make sure they check their messages regularly. On their own terms.

Share knowledge, include everyone

Synchronous communication is either very compartmentalized or very inefficient (and it can be both, of course). As teams grow, communication complexity increases dramatically (as described by Metcalfe's law). In practice, this leads to a trade-off: either exclude people from conversations, or rely on more frequent all-hands meetings.

Most synchronous discussions have to happen in a silo: improvised meetings at the coffee machine, or even in the open space (commonly known as interruptions). You will be de facto excluded if you happen to be in another meeting at that time, at home on a day off or just out for a 5-minute break. More meetings will be needed to keep everyone on the same page.

Asynchronous communication can easily solve all those problems. If you work in the open by default, in public spaces accessible by anyone, then the whole team can be included at all times, if only to gain the knowledge that there is something going on. Of course not everyone will follow every discussion that happen in the public square, but everyone will be able to make their own choices of what to follow or not. Even if you take some days off, you will have the opportunity to catch up with what happened while you were away.

Have deeper interactions

Good async communication requires providing enough context upfront. The goal is to avoid unnecessary back-and-forth, since replies may not come immediately. As Sahil Lavingia, founder at Gumroad, wrote about async:

All communication is thoughtful. Because nothing is urgent (unless the site is down), comments are made after mindful processing and never in real-time. There's no drama.

One immediate benefit is the rubberducking effect: while it may cost me more time to write a comprehensible question than to interrupt the coworkers that happen to be next to me right now, in many cases I may not even need to send the message. The mere fact of synthesizing a proper context for my message can provide me with the clarity I need to answer my own question.

More importantly, effective interruptions would require that all participants were able to mobilize all their knowledge on the spot, on demand, unprepared. This is obviously rarely the case, and improvised discussions very often lead to some variation of "I'm not sure" or "Let me check". Asynchronous communication avoids all of this: I can take the time to come back with a comprehensive answer everytime.

Make history

Writing things down is what turns activity into history, and history into organizational learning. When communication is written and preserved, your organization gains the ability to learn from its past.

In many synchronous companies, most of the organizational knowledge lives in the heads of a few senior employees that happened to be present when decisions were made (aka tribal knowledge). But the older the knowledge, the more fuzzy it gets: it is just human nature.

Some companies may have the discipline to commit their discussions to writing, but this is always done after the facts, and usually in a hurry (think of meeting notes for instance). People can refer to those documents when they want to understand the context that led to a specific decision, which will be the first time most of them are thoroughly read and they will only find them obtuse and difficult to understand, often by lack of context.

Asynchronous communication relies on written communication that is meant to be read every time (please do not do voice messages 🙃), and ambiguities rarely make it to the final documents.

Async is not just a remote work practice, it is a better default for modern knowledge work. It requires effort and discipline, but the payoff is a calmer, more thoughtful, and an overall more effective way to collaborate.

How to resize the scrollbars in Firefox

Mac — Sat, 07 Feb 2026 06:39:00 +0000

What’s with the trend of tiny and hidden scrollbars? You can hear some people of all stripes, presumably old enough to remember a time when accessibility concerns mattered to the point that they were solved at the OS level complain about the size of scrollbars nowadays (wherever you can still find them): on Windows, on Linux, on MacOS, on Chrome, on Firefox...

People are supposedly only using scroll wheels or trackpads nowadays, so scrollbars are seen as a distraction. Maybe it's just my age showing but I still find scrollbars useful sometimes, if only as visual indicators of where I stand in a large article or a big document. And I usually find them more efficient when I only want a fine tweak of my position in a page or, to the contrary, when I want to get back quickly to the top of an infinite scrolling page.

Anyway if you think like me that you could use bigger scrollbars on Firefox, open the configuration editor (about:config) and set the following values:

widget.non-native-theme.scrollbar.size.override 16
widget.non-native-theme.scrollbar.style 4
widget.non-native-theme.win.scrollbar.use-system-size false

Style 4 represents Windows 10 scrollbars, which IMHO are the most adequate for most situations, but you can explore other options.

No need to restart your browser. 😊

GPG keys are no longer shared with development containers on Windows by default

Mac — Fri, 16 Feb 2024 14:51:00 +0000

If you are into the habit of signing your Git commits, the configuration on Windows has always been pretty straightforward: install Gpg4win, set up your keys in the graphical interface and you’re pretty much done. And sharing those keys inside a development container with Visual Studio Code was even easier: nothing to do!

Well, that was the case until version 4.2 of Gpg4win. I have been banging my head for days now on why this wouldn’t work on a new machine of mine… As it turns out, the way keys are stored by Gpg4win by default has changed: it now uses keyboxd and Visual Studio Code doesn’t know how to handle that. Yet?

The old behaviour can simply be restored by running:

gpg-disable-keyboxd.bat

Done. You may have to recreate your containers for the keys to be picked up in your container.

DoH must be set up for HTTPS navigation in private networks

Mac — Sun, 04 Feb 2024 09:08:00 +0000

I love computers. I always have. And as a professional developer I always was a little bit jealous of my fellow systems administrators who were in charge of dozens of servers and desktops (and sometimes much more). Active Directory was a misterious world where seemed to lie an unreachable and absolute power. So of course, as soon a I could I set up a decent homelab for myself: DIY servers, NAS, 10Gb network, Hyper-V VMs, Active Directory… I also plunged into the wild world of domotics, starting with Domoticz and slowly migrating to Home Assistant.

Home automation applications have special DNS requirements, though, because an application that is installed on your phone must resolve to a different address whether you’re at home in your private network or out in the wild. My internal needs are easily fulfilled by the DNS Server which is linked to my Active Directory. In parallel, I bought the same domain name on a DNS provider that redirects every request to my private router, which dispatches requests to the proper server according to the specific TCP/IP port of the application. Works alright enough.

But surely nowadays the least we require is an HTTPS connection, especially if we are going to send passwords over the Internet to applications that could entirely control our homes. And the management of TCP/IP ports become a tedious affair to manage when the number of applications grow. So the plan is first to set up a private Certificate Authority that can generate certificates for my applications: I should be the only one using those applications, so I can simply distribute my own root certificates to the devices that will use it (and Active Directory makes it very easy for Windows systems). One benefit over Let’s Encrypt for instance is that it will avoid publicizing the names I use for my applications (cf. Certificate Transparency). A small Linux VM can easily be set up for this purpose only, that provides a step-ca server. Easy peasy. Then I set up a reverse proxy application that can handle TLS negociations and requests forwarding based on the domain name of the request. Seems straightforward enough.

Except that it does not work. The reason being that modern browsers will use DNS over HTTPS (DoH) to resolve your addresses. At least it is my experience, though I could not find a definitive reference for this behaviour. So unless your private DNS server is able to handle DoH a public server that can will be used instead. As a result your internal clients will cease to be fed with your private adresses and will try to use the public adresses that are sent to them (which in that case should be the public address of your router). I learned this the hard way…

The solution here is to set up DoH internally , which my DNS Server is currently unable to handle. But I could set up another small Linux VM running dns-dist, that can handle DoH (with certificates generated by my private Certificate Authority) and then defer actual name resolution to my private DNS server. The new DNS server should then be used all over the private network. For good measure I also deployed this new DNS Server address to all my Windows clients using the Add-DnsClientDohServerAddress cmdlet in an Active Directory startup script. As I understand it, it will then be automatically used by every system that is in the same IP subnet as the new private DNS Server. It will be picked up by the browsers as well, so now internal clients can properly resolve private addresses!

apt commands fail (Connection timed out) in Debian based containers on Windows

Mac — Sat, 03 Feb 2024 17:23:00 +0000

The last few years have given me several opportunities to discover the Ruby language, and its ecosystem. I am very much a .NET guy, and I had wondered for a long time what the hype was all about. Now I can see how it inspired many modern platforms (like node.js or .NET ~~Core~~ ), but boy is the operations side of it hard! I am maintaining this very blog on GitHub Pages, which is based on Jekyll, and even a simple usage like this requires a development container. Especially as a Windows guy, although managing Ruby dependencies on Linux is a mess by itself.

Anyway, I am not creating those containers every day, but I have had a lot of trouble lately trying to create new ones, that happen to be based on Debian images (Buster and Bullseye), like ruby:2.7.8 for instance. Part of the container creation involves running apt-get update && apt-get upgrade to have an image that is already up to date. These commands would invariably fail, preventing me (or VS Code for that matter) to create the container. The errors showed many Connection timed out while trying to fetch packages from deb.debian.org or security.debian.org… And it happened with different Debian versions, on different (Windows) hosts.

Part of the debugging process was trying to reach mirrors of the Debian repositories, using netselect-apt. But the same thing happened with the mirrors. And the first step of using netselect-api was installing it, by downloading the package directly with wget (can’t use apt there 😁), thus proving that the container had proper access to Internet…

Long story short, it seems that under my conditions (Debian running in a container in Docker Desktop on Windows using WSL), and for some reason, the connections established by apt are quite unstable. The trick was to configure apt so that it retries many times before giving up: I added echo 'Acquire::Retries "100";' > /etc/apt/apt.conf.d/99custom as the first command in all my Dockerfile descriptions.

That’s it. I can create brand new Debian based development containers again. 🥳

Continuous Integration for .NET (Core)

Mac — Thu, 28 Jan 2021 19:30:00 +0000

A lot of work has been made in the last few years (almost 5 now) to revamp the whole .NET Platform and lead it confidently into this new decade. Going through .NET Core, .NET is now free, open-source and multiplatform. Sign of the times, the console addicts can now take advantage of the .NET CLI while everyone can enjoy the still evolving best parts of the eco-system: the C# language and MSBuild (amongst other things). So now is a good opportunity as ever to revisit my practices of Continuous Integration for the .NET Framework as well.

This might look at first as easy as a combination of various dotnet build, dotnet test, dotnet pack and/or dotnet publish commands. This might work for some (and if it does then this is very fine), but to me this comes too close to breaking my Build in 1 step rule and I think I will keep on basing my builds on… MSBuild:

all those dotnet commands have to be coordinated in some way, and I also want to be able to execute the build locally. If we are to create cross-platform scripts then our options become quite limited outside of MSBuild.
MSBuild has the perfect logging infrastructure that allows to have both an understandable console output and a complete file log that can prove very valuable when things go wrong. Oh, and the Structured Log Viewer is just an amazing piece of software that has no equivalent that I know of on other technologies.
most of the dotnet commands are just wrappers around MSBuild targets anyway…

A simple project

If you want to see these principles in action please go and check Vigicrues.Client, a .NET wrapper for the Vigicrues API which reports information about flooding hazards in France.

mcartoixa / Vigicrues.Client

.NET client for the Vigicrues API

What I need for Continuous Integration are:

A solution file (Vigicrues.Client.sln).
A build file (Vigicrues.Client.proj), written in MSBuild.
A script file (build.bat) that helps executing the build file locally.
A CI configuration file (appveyor.yml). The rest of the project is either code of infrastructure.

The solution

Solutions are (still) Visual Studio speak for a collection of related projects. You can load them (with Visual Studio), and you can also build them (with MSBuild).

In this context, solutions have 2 purposes:

they are an entry point for developers to edit the code. Specifically the Vigicrues.Client-dev.sln solution should be used for that.
they are an entry point for the build scripts to generate a deployable package. This is Vigicrues.Client.sln.

In the full .NET Framework we also had a solution dedicated to automated tests but this could not work here because of a bug in the test framework. More on that later.

The main point here is that developers can still use their usual toolkit to develop (Visual Studio in this case).

The build file

The architecture of the build is quite simple:

execute the MSBuild equivalent of the various dotnet commands on the Vigicrues.Client.sln solution.
add a sprinkle of execution of various external dependencies and tools (like the cloc utility) to make the whole thing more interesting.

The gist of the Vigicrues.Client.proj is very simple:

<Project DefaultTargets="Rebuild" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup>
    <Projects Include="Vigicrues.Client.sln" />
  </ItemGroup>

  <Import Project="$(MSBuildProjectDirectory)\build\Common.targets" />
</Project>

We define a list of solutions to act on, and there is only one in this case. The rest is imported from another MSBuild file (build\Common.targets) which is quite specific at this time but may evolve into a generic reusable build file over time. This is where define my own standard targets:

Clean: cleans the build.
- This is usually a simple matter of deleting the tmp\ folder, as every other target generates its outputs there.
Compile: compiles the specified solutions.
Test: compiles the tests and executes them.
- Also performs code coverage analysis and generates a human readable report about it.
Analyze: performs some analysis on the project.
- Right now it gathers statistics using the cloc utility.
Package: generates a deployable package (in the tmp\out\bin folder). In our case this will be a NuGet file (NuGet being the dependency manager of choice on the .NET platform).
Build: shortcut for the combination of Compile, Test and Analyze.
Rebuild: shortcut for the combination of Clean and Build.
Release: shortcut for the combination of Clean, Build and Package.

Clean

As planned the Clean target is quite simple:

<Target Name="Clean" DependsOnTargets="CleanDirectories" />
<Target Name="CleanDirectories">
  <RemoveDir Directories="tmp\" />
</Target>

There is actually a bit more to it to take bug #3485 into account but this is a detail for this article.

Compile

The Compile target consists simply of calling MSBuild on the target solutions (instead of dotnet build which actually does the same thing):

<Target Name="Compile" DependsOnTargets="CompileProject" />
<Target Name="CompileProject">
  <PropertyGroup>
    <_BaseOutputPath>tmp\bin\%(Projects.FileName)\</_BaseOutputPath>
    <_BaseIntermediateOutputPath>tmp\obj\bin\%(Projects.FileName)\</_BaseIntermediateOutputPath>
  </PropertyGroup>
  <MSBuild
    Projects="%(Projects.Identity)"
    RebaseOutputs="True"
    Properties="Configuration=%(Projects.Configuration);Platform=%(Projects.Platform);BaseOutputPath=$(_BaseOutputPath);BaseIntermediateOutputPath=$(_BaseIntermediateOutputPath);%(Projects.Properties)"
    Targets="Restore;Build"
  />
</Target>

The only trick here is to redirect the outputs (including intermediate files) into subfolders of the tmp\ folder. This is what made the Clean target so easy to write.

Test

At the core of the Test target is another call to MSBuild (instead of dotnet test), very much like above. Specificities include:

not using solutions here but finding projects which name end with .Tests.csproj. This is partially because of bug #411 which prevented the use of solutions in the execution of tests. It may have been fixed now, but in the meantime I got used to not having a dedicated solution for tests…
not redirecting intermediates, because of bug #3485 again.
adding custom properties to the build (like dotnet test would). For instance:
- I use xUnit for my tests, so I will configure the xUnit Test Logger.
- I can dynamically add another logger by using the %VSTEST_LOGGER% environment variable (cf. the CI configuration file).
- I activate code coverage collection (Coverlet is already a dependency of my tests).
lastly, generated XML reports are copied under the tmp</code> folder where every report is expected.

And this gives someting like:

<Target Name="Test" DependsOnTargets="TestProject" />
<ItemGroup>
  <TestProjects Include="*\*.Tests.csproj" />
</ItemGroup>
<Target Name="TestProject"
  Outputs="tmp\tst\results\%(TestProjects.Filename)\TestResults.xml"
>
  <ItemGroup>
    <_VsTestLoggers Include="xunit" />
    <_VsTestLoggers Condition="'$(VSTEST_LOGGER)' != ''" Include="$(VSTEST_LOGGER)" />
  </ItemGroup>
  <PropertyGroup>
    <_BaseOutputPath>tmp\tst\bin\%(TestProjects.Filename)\</_BaseOutputPath>
    <_VSTestResultsPath>tmp\tst\results\%(TestProjects.Filename)\</_VSTestResultsPath>
    <_VsTestLogger>@(_VsTestLoggers->'&quot;%(Identity)&quot;')</_VsTestLogger>
  </PropertyGroup>
  <ItemGroup>
    <_TestProperties Include="IsTestProject=True" />
    <_TestProperties Include="VSTestNoLogo=True" />
    <_TestProperties Include="VSTestNoBuild=False" />
    <_TestProperties Include="VSTestBlame=True" />
    <_TestProperties Include="VSTestVerbosity=normal" />
    <_TestProperties Include="VSTestResultsDirectory=$(_VSTestResultsPath)" />
    <_TestProperties Include="VSTestTestAdapterPath=$(MSBuildProjectDirectory)" />
    <_TestProperties Include="VSTestCollect=XPlat Code Coverage" />
    <_TestProperties Include="VSTestLogger=$(_VsTestLogger)" />
  </ItemGroup>

  <RemoveDir Directories="$(_VSTestResultsPath)" />
  <MSBuild
    Projects="%(TestProjects.Identity)"
    RebaseOutputs="True"
    Properties="Configuration=Release;BaseOutputPath=$(_BaseOutputPath);@(_TestProperties);%(Projects.Properties)"
    Targets="Restore;VSTest"
  />
  <Copy Condition="Exists('$(_VSTestResultsPath)TestResults.xml')" SourceFiles="$(_VSTestResultsPath)TestResults.xml" DestinationFiles="tmp\%(TestProjects.Filename)-xunit-results.xml" />
</Target>

Hey, but what about the actual code coverage? It is collected but not exploited yet: we will use ReportGenerator for this. This is a NuGet dependency that we can define and restore in the project file itself by defining the right properties:

<PropertyGroup>
  <RestoreGraphProjectInput>$(MSBuildProjectFullPath)</RestoreGraphProjectInput>
  <TargetFramework>netstandard2.1</TargetFramework>
  <MSBuildProjectExtensionsPath>tmp\obj\</MSBuildProjectExtensionsPath>
</PropertyGroup>
<ItemGroup>
  <PackageReference Include="ReportGenerator" Version="4.8.4" />
</ItemGroup>

<Import Project="$(MSBuildToolsPath)\NuGet.targets" />

<Target Name="Prepare" DependsOnTargets="Restore">
  <MakeDir Directories="$(TmpOutputPath)" />
</Target>

<Import Project="$(MSBuildProjectExtensionsPath)$(MSBuildProjectFile).nuget.g.props" />
<Import Project="$(MSBuildProjectExtensionsPath)$(MSBuildProjectFile).nuget.g.targets" />

Now we can use ReportGenerator on all the code coverage results and generate:

a nice HTML report for humans to consume.
a XML report, under the tmp\ directory along withy other reports.

<Target Name="GenerateTestReports"
  Returns="@(CoverageResults)"
>
  <ItemGroup>
    <CoverageResults Include="tmp\tst\results\**\coverage.cobertura.xml" />
  </ItemGroup>
</Target>

<Target Name="_GenerateTestReports"
  Condition="'@(CoverageResults)' != ''"
  AfterTargets="GenerateTestReports"
>
  <ReportGenerator ReportFiles="@(CoverageResults)" TargetDirectory="tmp\tst\" ReportTypes="HtmlInline;Cobertura" VerbosityLevel="Info" />
  <Move SourceFiles="tmp\tst\Cobertura.xml" DestinationFiles="tmp\tst\$(MSBuildProjectName)-cobertura-results.xml" />
</Target>

As a side note you may notice that the tests results are not part of the output due to poor integration of the Test Platform with MSBuild (cf. bug #680), but we are working on that.

Analyze

This target is just a matter of executing the cloc utility. The only trick is to execute the Perl script when not on Windows:

  <Target Name="Analyze" DependsOnTargets="CountLoc" />
  <Target Name="CountLoc">
    <PropertyGroup>
      <ClocResultsFile>tmp\cloc-results.xml</ClocResultsFile>
      <_ClocCommand Condition="'$(OS)'=='Windows_NT'">&quot;.tmp\cloc.exe&quot;</_ClocCommand>
      <_ClocCommand Condition="'$(_ClocCommand)'==''">perl &quot;.tmp/cloc.pl&quot;</_ClocCommand>
    </PropertyGroup>
    <Exec
      Command="$(_ClocCommand) &quot;$(InputPath)&quot; --exclude-dir=.tmp,.vs,.vscode,bin,build,doc,lib,obj,tmp,GeneratedCode --exclude-ext=csproj,dbmdl,proj,sln,sqlproj,suo,user --3 --quiet --progress-rate=0 --xml --report_file=&quot;$(ClocResultsFile)&quot;"
      YieldDuringToolExecution="True"
      IgnoreExitCode="True"
    />
  </Target>

Package

In the case of a library like we have, the Package target is just calling the MSBuild equivalent of dotnet pack:

<Target Name="Package" DependsOnTargets="Prepare;Project" />
<Target Name="PackageProject">
  <PropertyGroup>
    <_BaseOutputPath>tmp\pck\%(Projects.FileName)\</_BaseOutputPath>
    <_BaseIntermediateOutputPath>tmp\obj\bin\%(Projects.FileName)\</_BaseIntermediateOutputPath>
  </PropertyGroup>
  <ItemGroup>
    <_PackageProperties Include="PackageOutputPath=tmp\out\bin" />
  </ItemGroup>
  <MSBuild
    Projects="%(Projects.Identity)"
    RebaseOutputs="True"
    Properties="Configuration=%(Projects.Configuration);Platform=%(Projects.Platform);BaseOutputPath=$(_BaseOutputPath);BaseIntermediateOutputPath=$(_BaseIntermediateOutputPath);@(_PackageProperties);%(Projects.Properties)"
    Targets="Restore;Pack"
  />
</Target>

We will expect every final artefact to be generated in the tmp\out\bin folder.

The script file

There are actually 2 script files here:

build.bat on Windows.
build.sh on Linux and MacOS.

They both do the same thing on different platforms so I will detail only one of them. It is just a matter of interpreting command line parameters to create the right environment variables before executing the build:

dotnet.exe tool restore
dotnet.exe msbuild Vigicrue.Client.proj /nologo /t:Build /m /r /fl /flp:logfile=build.log;verbosity=%_VERBOSITY%;encoding=UTF-8 /nr:False /v:normal

One external dependency has to be installed prior to this execution though: the cloc utility. This is done in various stages:

version for this dependency is defined in the build\versions.env file.

_CLOC_VERSION=1.82

in the build\SetEnv.bat script versions are set as environment variables:

IF EXIST build\versions.env (
    FOR /F "eol=# tokens=1* delims==" %%i IN (build\versions.env) DO (
        SET "%%i=%%j"
        ECHO SET %%i=%%j
    )
    ECHO.
)

the right version of the tool is downloaded and extracted in the .tmp folder (if necessary):

IF NOT EXIST "%CD%\.tmp\cloc.exe" GOTO SETENV_CLOC
FOR /F %%i IN ('"%CD%\.tmp\cloc.exe" --version') DO (
    IF "%%i"=="%_CLOC_VERSION%" GOTO END
)
:SETENV_CLOC
powershell.exe -NoLogo -NonInteractive -ExecutionPolicy ByPass -Command "& { [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Invoke-WebRequest https://github.com/AlDanial/cloc/releases/download/$Env:_CLOC_VERSION/cloc-$Env:_CLOC_VERSION.exe -OutFile .tmp\cloc.exe; }"
:END

The build\SetEnv.bat is then simply called in the build.bat script file. The same architecture could be used for other tools that cannot be retrieved with NuGet.

The CI configuration file

I will use AppVeyor as a platform, but as usual the configuration will be very simple because all the complexity has been handled above. I could very simply switch to any other tool with minimal reconfiguration. The appveyor.yml can simply be:

version: 0.1.{build}.0
image: Visual Studio 2019

install:
  - cmd: CALL build\SetEnv.bat
  - cmd: dotnet tool restore

build_script:
  - cmd: dotnet msbuild Vigicrues.Client.proj /nologo /t:Release /m /r /l:"C:\Program Files\AppVeyor\BuildAgent\dotnetcore\Appveyor.MSBuildLogger.dll" /fl /flp:logfile=build.log;verbosity=diagnostic;encoding=UTF-8 /nr:False /v:normal

In pratice I will add a few tweaks though:

upload the coverage results to the Codecov platform:

  install:
    - cmd: dotnet tool update Codecov.Tool --version 1.12.4
  on_success:
    - cmd: dotnet tool run codecov -f "tmp\*-cobertura-results.xml"

use the AppVeyor test logger to automatically report test results to the platform (remember the %VSTEST_LOGGER% environment variable?):

  environment:
    VSTEST_LOGGER: Appveyor
  install:
    - cmd: dotnet add Vigicrues.Tests\Vigicrues.Tests.csproj package Appveyor.TestLogger --version 2.0.0

Going further

These scripts are still in early phase and they will evolve over time. In fact they might have already evolved at the time you read this post, but I guess I could still add:

the ability to handle the packaging of native applications, which is a simple matter of translating dotnet publish into MSBuild scripts.
the ability to handle the packaging of web applications which whould also involve dotnet publish and a small touch of Web Deploy (I love this tool).

When everything is stable enough maybe I could extract most of the files in the build\ directory into a proper NuGet package and reuse them over all my projects. dotnet make, anyone?

// Detect dark theme var iframe = document.getElementById('tweet-1219923246373425152-434'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1219923246373425152&theme=dark" }

How to host pgAdmin 4 on Windows Server

Mac — Sun, 03 Jan 2021 06:53:00 +0000

I just set out to create a PostgreSQL database server in my home lab, that I intend to use first with Home Assistant. And just because this is my home lab, it is hosted in a Windows Server 2019 VM in Hyper-V. 😁 I then foolishly installed pgAdmin 4 on my computer with the intention to remotely manage this new server… I remember pgAdmin III as such a slick application with an installer weighing less than 15MB. 4 years later the pgAdmin 4 installer is 10 times bigger , a Flask application that runs in the browser (these are the times we live in 🤷🏻‍♂️) once the server has started, which can take a while.

So I decided that this (otherwise fine) piece of bloatware belonged on a web server and not on my computer. This is not very difficult once you have got the recipe (and some experience on Windows of course).

Ingredients

The ingredients are:

a Windows Server machine. I used Windows Server 2019, but I am pretty sure the same applies to every version at least back to Windows Server 2008. Or Windows 8, for that matter.
an instance of IIS. This can be installed as a role inside Windows Server (or as as feature in Windows 10).
HttpPlatformHandler which is a IIS Module that can be used to manage any process on the server (we will be using Python in this case) and act as a reverse proxy.
pgAdmin 4:
- it can be installed anywhere, but %ProgramFiles% is usually where programs go.

Recipe

Create an application pool
Create a website
Open the firewall
Configure the website
Grant rights
Start the website

Create an application pool

As a best practice you will need a new Application Pool in IIS: this will allow us to isolate the application in a dedicated worker process with a dedicated identity. I named my pool PgAdmin:

Create a website

Then we need a new website:

select the Application Pool that we just created.
create a new folder under C:\inetpub called pgadmin.
configure your bindings as you wish: I configured my application to be accessible on port 3000. You will be able to change this later if necessary.
do not start the website right away (there is still work to do).

Open the firewall

Make sure your application is accessible by opening the Windows Firewall if necessary (on TCP port 3000 in my case).

Configure the website

Configure the website by adding the following configuration (cf. Configure Python web apps for IIS) in a file called web.config inside the pgadmin folder that you just created (you can use Notepad):

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <add name="PythonHandler" path="*" verb="*" modules="httpPlatformHandler" resourceType="Unspecified"/>
    </handlers>
    <httpPlatform
       processPath="%ProgramFiles%\pgAdmin 4\v4\runtime\python.exe"
       arguments="&quot;%ProgramFiles%\pgAdmin 4\v4\web\pgAdmin4.py&quot;"
       stdoutLogEnabled="true"
       stdoutLogFile="C:\inetpub\logs\pgadmin\pgAdmin4.log"
       startupTimeLimit="60"
       processesPerApplication="1">
      <environmentVariables>
        <environmentVariable name="PGADMIN_INT_PORT" value="%HTTP_PLATFORM_PORT%" />
        <environmentVariable name="PYTHONHOME" value="%ProgramFiles%\pgAdmin 4\v4\venv" />
        <environmentVariable name="SERVER_MODE" value="True" />
      </environmentVariables>
    </httpPlatform>
    <modules>
      <remove name="WebDAVModule" />
    </modules>
  </system.webServer>
</configuration>

It is pretty straightforward but notably:

the arguments attribute is quoted to account for the fact that the path contains spaces.
the process outputs logs inside a folder called C:\inetpub\logs\pgadmin that would have to be created beforehand.
pgAdmin notoriously handles a single process only.
HttpPlatformHandler will automatically create a process using Python and assign it a dedicated port using the HTTP_PLATFORM_PORT environment variable.

Grant rights

And now we simply have to give the proper rights to our application pool user named IIS APPPOOL\PgAdmin:

Read & Execute on the folder %ProgramFiles%\pgAdmin 4 (and subfolders).
Read & Execute on the folder C:\inetpub\pgadmin.
- Create a subfolder called pgAdmin and give Full Control to the user. This where the application will store its data.
Full Control on the logging folder (C:\inetpub\logs\pgadmin in my case).

Start the website

Now you can start the website. Enjoy!

Note that the application will recycle automatically every 29 hours by default, which means that the next request will take much longer while the process is restarted. Not such a bad thing by itself, but obviously this behaviour can be changed.

Also note that the logging folder should be cleaned up from time to time…

UPDATE 2021-05-29: the website configuration has been udpated for version 5 of pgAdmin... 4! Here is the configuration for version 4:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <system.webServer>
    <handlers>
      <add name="PythonHandler" path="*" verb="*" modules="httpPlatformHandler" resourceType="Unspecified"/>
    </handlers>
    <httpPlatform
       processPath="%ProgramFiles%\pgAdmin 4\v4\runtime\python.exe"
       arguments="&quot;%ProgramFiles%\pgAdmin 4\v4\web\pgAdmin4.py&quot;"
       stdoutLogEnabled="true"
       stdoutLogFile="C:\inetpub\logs\pgadmin\pgAdmin4.log"
       startupTimeLimit="60"
       processesPerApplication="1">
      <environmentVariables>
        <environmentVariable name="PGADMIN_INT_PORT" value="%HTTP_PLATFORM_PORT%" />
        <environmentVariable name="PYTHONHOME" value="%ProgramFiles%\pgAdmin 4\v4\venv" />
        <environmentVariable name="SERVER_MODE" value="True" />
      </environmentVariables>
    </httpPlatform>
    <modules>
      <remove name="WebDAVModule" />
    </modules>
  </system.webServer>
</configuration>

Continuous Integration for Java

Mac — Mon, 05 Oct 2020 05:19:00 +0000

From what I understand Gradle or even Maven are the way to go for your build in Java. They seem to rely heavily on conventions, which is good. But if you need (or want) to add to these conventions (like I do with my practices) it seems that the only 2 ways are to either build your own plugins or to execute additional Apache Ant tasks and targets inside your build. So it seems to me that the good old Apache Ant is still alive and well.

The project

As we will investigate in the next installment of this series I needed to create a build for a Salesforce project using the Salesforce CLI. No build tool is truly native to this environment, and Salesforce is very keen on demonstrating how simple builds can be achieved by batching command lines executions. I did not feel that this would allow me to create the kind of build that I wanted, and for reasons that I will detail when we get there I settled on using Apache Ant instead. In order to have proper control over the execution and the outputs I decided to encapsulate the Salesforce CLI commands into proper Ant tasks (instead of using the exec task). Hence the ant-sfdx project.

mcartoixa / ant-sfdx

Ant tasks that encapsulate the Salesforce DX CLI

The main elements of the project are:

The build file (build.xml).
The script file (build.bat) that helps executing the build file locally.
The CI configuration file (.travis.yml).

The build file

Given that the goal is to create Apache Ant tasks, it seems natural that I used it for this project as well, if only to acquire more experience with it. I also settled on using NetBeans, not because it is the best Java IDE around (I hope it is not) but because it provides an excellent support for Apache Ant and it can create a complete and extensible build based on Apache Ant.

As a matter of fact if I just import the build file provided by NetBeans in my own build.xml I automatically get all the targets I need to compile and test the project:

<import file="nbproject/build-impl.xml" />

My build provides the following targets (which should feel familiar by now):

clean: automatically provided by NetBeans.
- Extended to clean the tmp\ directory.
compile: automatically provided by NetBeans.
test: automatically provided by NetBeans.
- Extended to include JaCoCo for code coverage measurement.
package: creates a package for the plugin which consists of the JAR file, a POM for reference and deployment and a zip file for copy and paste deployment.
build: shortcut for the combination of compile, test and analyze.
rebuild: shortcut for the combination of clean and build.
release: shortcut for the combination of clean, build, and package.

The build provided by NetBeans contains empty targets that are meant to be overridden for extension. For instance the -post-clean target is the perfect extension point to delete the tmp\ directory:

<target name="-post-clean" depends="clean.tmp" />
<target name="clean.tmp">
  <delete dir="./tmp" includeemptydirs="true" quiet="true" failonerror="false" />
</target>

To be able to find extensions points you can try and read the whole included file (1700+ lines of XML!), or you can use NetBeans itself to navigate it:

I can also perform code analysis using PMD:

<target name="analyze.pmd">
  <taskdef name="pmd" classname="net.sourceforge.pmd.ant.PMDTask">
    <classpath>
      <fileset dir=".tmp/pmd-bin-6.21.0/lib">
        <include name="**/*.jar" />
      </fileset>
    </classpath>
  </taskdef>

  <pmd failOnRuleViolation="true" cacheLocation="tmp/obj/pmd.cache">
    <fileset dir="src/main">
      <include name="**/*.java" />
    </fileset>
    <sourceLanguage name="java" version="1.8"/>
    <ruleset>.ruleset.xml</ruleset>
    <formatter type="text" toConsole="true" />
    <formatter type="html" toFile="tmp/pmd-results.html" />
    <formatter type="xml" toFile="tmp/pmd-results.xml" />
  </pmd>
</target>

The package target has to be created entirely, but once you know what has to be done (cf. above) it is quite straightforward:

<target name="package" depends="jar,package.pom">
  <copy todir="tmp/out/bin/" preservelastmodified="true">
    <resources>
      <file file="tmp/bin/ant-sfdx.jar" />
      <file file="ivy.xml" />
    </resources>
  </copy>
  <zip destfile="tmp/out/bin/ant-sfdx.zip" compress="true" level="9" filesonly="true">
    <resources>
      <file file="tmp/bin/ant-sfdx.jar" />
      <file file="ivy.xml" />
      <file file="${ivy.runtime.classpath}" />
    </resources>
    <zipfileset dir="docs" prefix="docs" />
  </zip>
</target>

<target name="package.pom">
  <ivy:makepom ivyfile="ivy.xml" pomfile="tmp/out/bin/pom.xml" conf="default,compile">
    <mapping conf="default" scope="default" />
    <mapping conf="test" scope="test" />
    <mapping conf="compile" scope="compile" />
  </ivy:makepom>
</target>

As you may see here there are references to Apache Ivy: it is the dependency manager of choice when using Apache Ant. I found its learning curve to be somewhat steep but well integrated technologies prove powerful in the end. The dependencies for the project are described in an XML file (of course!) ivy.xml, and part of the build consists of retrieving these dependencies, updating the various CLASSPATHs associated with them and also updating the properties file that is the basis of the NetBeans project so that it remains up to date when dependencies change:

<target name="prepare.dependencies">
  <ivy:retrieve pattern="ivy/lib/[conf]/[artifact].[ext]" log="quiet" />

  <pathconvert property="ivy.compile.classpath" dirsep="/" pathsep=":">
    <path>
      <fileset dir="${basedir}/ivy/lib/compile" includes="**/*.jar" />
    </path>
    <map from="${basedir}${file.separator}" to="" />
  </pathconvert>
  <pathconvert property="ivy.test.classpath" dirsep="/" pathsep=":">
    <path>
      <fileset dir="${basedir}/ivy/lib/test" includes="**/*.jar" />
    </path>
    <map from="${basedir}${file.separator}" to="" />
  </pathconvert>
  <uptodate property="prepare.ivy.nbproject.isuptodate" srcfile="ivy.xml" targetfile="nbproject/project.properties" />
  <propertyfile file="nbproject/project.properties" unless:set="prepare.ivy.nbproject.isuptodate">
    <entry operation="=" key="ivy.compile.classpath" value="${ivy.compile.classpath}" />
    <entry operation="=" key="ivy.test.classpath" value="${ivy.test.classpath}" />
  </propertyfile>
</target>

The script file

At the core of the build.bat script file lies simply the execution of Apache Ant:

CALL "%ANT_HOME%\bin\ant.bat" -noclasspath -nouserlib -noinput -lib "ivy\lib\test" -Dverbosity=%VERBOSITY% -f %PROJECT% %TARGET%

But before getting there we need to initialize the environment for our build:

retrieving Apache Ivy dependencies that are necessary prior to lauching the build (including Apache Ivy itself).
retrieving dependencies that cannot be retrieved with Apache Ivy (like PMD, or Apache Ant for instance).
setting up the required environment variables (like %ANT_HOME%).

Apache Ivy dependencies

Everything Apache Ivy related is done right before executing Apache Ant:

first download Apache Ivy itself if necessary.
then execute it to retrieve the dependencies.

IF NOT EXIST ivy MKDIR ivy
PUSHD ivy
IF NOT EXIST ivy.jar (
    powershell.exe -NoLogo -NonInteractive -ExecutionPolicy ByPass -Command "& { [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Invoke-WebRequest https://repo1.maven.org/maven2/org/apache/ivy/ivy/$Env:_IVY_VERSION/ivy-$Env:_IVY_VERSION.jar -OutFile ivy.jar; }"
)
POPD
"%JAVA_HOME%\bin\java.exe" -jar ivy\ivy.jar -retrieve "ivy\lib\[conf]\[artifact].[ext]"

General dependencies

As for the rest of the initialization it happens inside the build\ directory. The versions for our dependencies are described in a build\versions.env file, so that those definitions can be reused accross scripts (and platforms):

_ANT_VERSION=1.9.14
_CLOC_VERSION=1.82
_IVY_VERSION=2.5.0
_PMD_VERSION=6.21.0

This file can easily be read as environment variables inside the build\SetEnv.bat file:

IF EXIST build\versions.env (
    FOR /F "eol=# tokens=1* delims==" %%i IN (build\versions.env) DO (
        SET "%%i=%%j"
        ECHO SET %%i=%%j
    )
    ECHO.
)

The shell equivalent is in the build/.bashrc file:

if [-f ./build/versions.env]; then
    # xargs does not support the -d option on BSD (MacOS X)
    export $(grep -a -v -e '^#' -e '^[[:space:]]*$' build/versions.env | tr '\n' '\0' | xargs -0 )
    grep -a -v -e '^#' -e '^[[:space:]]*$' build/versions.env | tr '\n' '\0' | xargs -0 printf "\$%s\n"
    echo
fi

The right version of Apache Ant can then easily be installed locally (inside the .tmp\ folder, by convention) and the proper environment variable be set:

SET ANT_HOME=%CD%\.tmp\apache-ant-%_ANT_VERSION%
IF NOT EXIST "%ANT_HOME%\bin\ant.bat" (
    IF NOT EXIST .tmp MKDIR .tmp
    powershell.exe -NoLogo -NonInteractive -ExecutionPolicy ByPass -Command "& { [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Invoke-WebRequest http://mirrors.ircam.fr/pub/apache//ant/binaries/apache-ant-$Env:_ANT_VERSION-bin.zip -OutFile .tmp\apache-ant-$Env:_ANT_VERSION-bin.zip; }"
    IF ERRORLEVEL 1 GOTO ERROR_ANT
    powershell.exe -NoLogo -NonInteractive -ExecutionPolicy ByPass -Command "Expand-Archive -Path .tmp\apache-ant-$Env:_ANT_VERSION-bin.zip -DestinationPath .tmp -Force"
    IF ERRORLEVEL 1 GOTO ERROR_ANT
)
ECHO SET ANT_HOME=%ANT_HOME%

A more traditional approach would have been to require everyone to install Apache Ant as a prerequisite but:

this adds a lot of burden to developers in the form of a lot of dependencies to install prior to developping proper.
part of the installation process is setting up a global environment variable (%ANT_HOME%) because there is no way to automatically detect these installation paths.
this makes the use of different versions (of Apache Ant for instance) in different projects, or even in different branches) very… tricky.
a CI platform is a particular developer that cannot perform manual installations, and so I would have to make sure that the right versions of the different dependencies are already available there. Or I would have to install them and write those scripts anyway…

The only real prerequisite for this project is thus Java 8. It has a proper installer and different versions can be installed on the same computer. All you have to do is use the registry (yes, I love the registry) to find where it has been installed and initialize the %JAVA_HOME% environment variable:

SET JAVA_HOME=
FOR /F "tokens=1,2*" %%i IN ('REG QUERY "HKLM\SOFTWARE\JavaSoft\Java Development Kit\1.8" /V JavaHome') DO (
    IF "%%i"=="JavaHome" (
        SET "JAVA_HOME=%%k"
    )
)
IF "%PROCESSOR_ARCHITECTURE%"=="AMD64" (
    FOR /F "tokens=1,2*" %%i IN ('REG QUERY "HKLM\SOFTWARE\Wow6432Node\JavaSoft\Java Development Kit\1.8" /V JavaHome') DO (
        IF "%%i"=="JavaHome" (
            SET "JAVA_HOME=%%k"
        )
    )
)

And so this was more work here than we had seen previously in the equivalent sections, but in the end developers can just clone the repository and execute the build locally right away. Which is exactly what a CI platform does by the way…

The CI configuration file

We solved every major problem previously so that this configuration should be a breeze, and I think it is. The configuration for Travis CI (yes, again) is simply:

install:
  - . build/.bashrc
  - if [! -d ivy]; then mkdir ivy; fi
  - if [! -f ivy/ivy.jar]; then wget -nv -O ivy/ivy.jar https://repo1.maven.org/maven2/org/apache/ivy/ivy/$_IVY_VERSION/ivy-$_IVY_VERSION.jar; fi
  - $JAVA_HOME/bin/java -jar ivy/ivy.jar -retrieve "ivy/lib/[conf]/[artifact].[ext]"
script:
  - $ANT_HOME/bin/ant -noclasspath -nouserlib -noinput -lib "ivy/lib/test" -logger org.apache.tools.ant.listener.AnsiColorLogger -f build.xml release

I also chose this project to try and create a configuration for Azure Pipelines, and here it is in the form of azure-pipeline.yml:

steps:
- task: BatchScript@1
  inputs:
    filename: build\SetEnv.bat
    arguments: /useCurrentJavaHome
    modifyEnvironment: True
    workingFolder: $(Build.Repository.LocalPath)
- script: |
    IF NOT EXIST ivy MKDIR ivy
    PUSHD ivy
    IF NOT EXIST ivy.jar (
        powershell.exe -NoLogo -NonInteractive -ExecutionPolicy ByPass -Command "& { [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Invoke-WebRequest https://repo1.maven.org/maven2/org/apache/ivy/ivy/$Env:_IVY_VERSION/ivy-$Env:_IVY_VERSION.jar -OutFile ivy.jar; }"
        IF ERRORLEVEL 1 GOTO END_ERROR
    )
    POPD
    "%JAVA_HOME%\bin\java.exe" -jar ivy\ivy.jar -retrieve "ivy\lib\[conf]\[artifact].[ext]"

- task: Ant@1
  inputs:
    buildFile: 'build.xml'
    options: -noclasspath -nouserlib -noinput -lib ivy/lib/test -Dverbosity=verbose
    targets: release
    publishJUnitResults: true
    testResultsFiles: '$(Build.Repository.LocalPath)\tmp\obj\test\results\**\TEST-*.xml'
    antHomeDirectory: '$(ANT_HOME)'
    jdkVersionOption: 1.8

As promised, this shows I am not tied to a particular CI platform: I cannot imagine the pain it would have been to try a new CI platform if all the logic for the build had been located in these configuration files. It could even have been much shorter had I decided to include the Apache Ivy related commands in the build\SetEnv.bat part. I am sure I must have had my reasons not to at the time…

Just as a side note, I had to create and use a special /useCurrentJavaHome argument to the build\SetEnv.bat so that it would not override the %JAVA_HOME% environment variable: registry installation detection would not work on Azure Pipelines. No wonder we need tools like Docker everywhere now when the registry is gone…

Ant as a build tool

Apache Ant has a bad reputation and I can understand why. I think its main failures lie in the fact that it was the first of the next-gen build tools that promised:

tight integration with a specific ecosystem (Java in this case).
multi-platform description (and execution).

And as it was the first it was inevitably the worst. Especially when it comes to understanding the core concepts, the types to understand and use are far too numerous and too complex (I mean: FileLists and FileSets?…). And after all this time you would think something really should have been done about this (that would have required some amount of breaking changes along the way). But these concepts are key to understand, and cheap “solutions” like ant-contrib (I don’t even want to link to this project) only help you make a mess of your builds.

On the other hand, a lot of people seem to resent Apache Ant because of The Angle Bracket Tax. But like any language it is the developer’s responsibility to remember that he or she writes code for his or her fellow human colleagues. And I will take XML any day over JSON (or YAML), for both power and expressiveness, thank you very much.

As I said in the beginning of this post, the Java world seems to rely heavily on conventions now with tools like Maven or Gradle, and this is very fine. But if you need to get further than those conventions Apache Ant will still take you a long way, that is usually more convenient than writing plugins for these tools.

We will meet Apache Ant again soon.

Continuous Integration for PHP

Mac — Mon, 28 Sep 2020 05:59:00 +0000

It has been a very long time since I have used PHP in any capacity for web development. And I am not planning doing so again any time soon (sorry). But I have had the opportunity to apply my practices on a very small, though still useful scale in PHP. From what I could gather it is indeed very possible to properly and continuously integrate PHP projects, provided you use the right tools for the job: a build tool and a dependency manager (optional but always recommended).

A simple project
A more complete project

A simple project

Disclaimer: I do not condone illegal behaviour… But for research purposes I have created a custom search module for Synology Download Manager, and as it happens this requires PHP. Hence the synology-dlm-rarbg project. No Laravel or Symfony involved here obviously but I want to believe that the concepts will remain the same on a more consequent project.

mcartoixa / synology-dlm-rarbg

RarBG search module for Synology Download Manager

The main elements of the project are:

The build file (build.xml).
The script file (build.bat) that helps executing the build file locally.
The CI configuration file (.travis.yml).

The build file

Given my lack of recent experience with the technology I was a bit worried at first about the state of software development in PHP. But it turned out surprisingly pretty decent: you can find a fine dependency manager in Composer and a very decent build tool in Phing. It is based on the old timer Apache Ant and so probably suffers from the same drawbacks (that I will cover in a subsequent post). But like the original: as long as you respect the tool (which involves understanding it…) you can go pretty far with it.

So Phing it is, and the build.xml file defines the following targets (pretty much consistent with what we have already seen in .NET, node.js and Ruby):

clean: cleans the build (the tmp\ directory).
build: runs PHPLint on the source code and copies the source code and the dependencies in the tmp\out\bin folder.
- If I had to do it today I would probably isolate the first action in an analyze target. The second action would do well in the package target. My own consistency evolves over time…
test: executes the tests using PHPUnit.
package: creates a package for the search module (which is an archive of the source code and its dependencies).
rebuild: shortcut for the combination of clean and build.
release: shortcut for the combination of clean, build, and package.

Before showing some code it appears that I may have over engineered this build a bit: I created a second build file dlm\build.xml that is called by the first. Obviously I cannot clearly remember why right now but I think it had to do with the possibility of packaging multiple search modules in the same project. Anyway, how do you call another build file then? Simple:

<target name="test" depends="prepare.build">
  <phing phingfile="build.xml" dir="./dlm" target="test" inheritAll="true" haltonfailure="true" />
</target>

The test target must execute PHPUnit on the tests files, and it needs to have the dependencies properly downloaded before that (using Composer):

<target name="test" depends="test.prepare">
  <phpunit printsummary="true" bootstrap="vendor/autoload.php" haltonerror="true" haltonfailure="true">
    <formatter type="plain" usefile="false" />
    <batchtest>
      <fileset dir="./tests">
        <include name="**/*Test*.php"/>
      </fileset>
    </batchtest>
  </phpunit>
</target>

<target name="test.prepare">
  <composer command="update" composer="bin/composer.phar">
    <arg value="-q" />
    <arg value="-n" />
  </composer>
</target>

The key in this project is to be able to package the search module in the format that is expected by the Download Manager, which is explained in a PDF document downloadable somewhere on the Synology website. I would never remember how, but now it has been described in Phing it seems pretty straightforward:

<target name="package" depends="package.prepare,build">
  <tar destfile="tmp/out/bin/mcartoixa_rarbg.dlm" compression="gzip">
    <fileset dir="tmp/bin/dlm">
      <include name=" **/**" />
      <exclude name="composer.*" />
    </fileset>
  </tar>
</target>

The script file

The gist of the script file build.bat is simply to install the dependencies, including Phing, and then run the build:

CALL .\bin\composer.bat install -q -n
CALL .\vendor\bin\phing.bat -f %PROJECT% %TARGET%

As you can see Composer is itself stored in the repository as a PHAR archive. I could have chosen to download it instead. The key here is that the repository is ready for development: nothing to configure.

The CI configuration file

For whatever reason I have again chosen Travis CI as my Continuous Integration platform, which configuration is simply:

install:
  - php bin/composer.phar install -n

script:
  - vendor/bin/phing -f build.xml prepare.version release

A more complete project

Nothing much to be found here (as usual?), but I think proper foundations have been laid out above. As with every technology it is essential to delve into the build tool, understand it and use it to its full potential. For instance, if Phing is anything like Apache Ant (and it looks like it is), understanding the core structures (like FileLists of FileSets) is very important.

Also, I would tend to install every dependency locally instead of globally (as part of the build, including frameworks like Laravel for instance). It makes it easier to handle the %PATH% consistently for everyone (including your Continuous Integration platform) and it makes it easier to use different versions on different projects (or even branches).

Continuous Integration for Ruby

Mac — Mon, 21 Sep 2020 07:57:00 +0000

I have had to deal with a few projects in Ruby recently and I have tried to adapt my practices to this new environment. On a personal level this means less cognitive load when I have to come back to these projects months later, even more so because Ruby is far from my area of expertise. As it happens I have had quite some trouble implementing Continuous Integration because of (seemingly) conflicting conventions. But I have tried anyway:

A simple project
A more complete project

A simple project

My website (which includes my blog) is in fact based on Ruby, as it is a Jekyll project hosted on GitHub Pages (for now). Granted, it is not a proper Ruby project (as Ruby is not required on the target server for instance) but I find it is a good introduction to the concepts and the tools involved.

mcartoixa / mcartoixa.github.io

The main elements of the project are:

The build file (Rakefile).
The script file (build.sh) that helps executing the build file locally.
The CI configuration file (.travis.yml).

The build file

I don’t think there is much choice here in the build tool to use in a Ruby environment, so Rake it is. Here is the whole file:

PROJECT_NAME ||= 'www.mcartoixa.me'
PROJECT_VERSION ||= '0.0.0'

import 'build/common.rake'

Alright, I cheated. I like to gather everything build related in a build\ folder and I did that here. But what we loose in straightforwardness we gain in consistency, because in a Rails project, where many tasks and targets are imported, I find this actually brings more readability to the project.

So, what’s in there? The build file defines the following targets (ie tasks in Rake speak), overall consistent with what we have seen in other technologies:

build:clean: cleans the build (the tmp\ directory).
build:compile: creates the static version of the website (using Jekyll), ready for deployment (in the tmp\obj\bin\ folder).
build:test: tests the project.
- I want to check the links in my website using HTMLProofer. I could use a Jekyll plugin for this, but testing links proves rather time consuming and error prone (more on that later) and I would rather keep it out of my development workflow.
build:analyze: checks deprecation and configuration issues (using Jekyll).
build:package: creates a deployable archive of the static version of the website (in the tmp\out\bin\ folder).
- This is of little use right now as everything is handled by GitHub pages, including deployment, but this is partly a proof of concept. And I like the freedom of knowing I could host my website elsewhere at any moment.
build:build: shortcut for the combination of build:compile, build:test and build:analyze.
build:rebuild: shortcut for the combination of build:clean and build:build.
build:release: shortcut for the combination of build:clean, build:build, and build:package.

What’s remarkable here is the use of namespaces in the tasks names. Again, this proves useful (and consistent) in a Rails project. And so the description of the build:compile task is simply:

namespace 'build' do
  task 'compile' do
    sh 'jekyll', 'build', '-dtmp/obj/bin/', 'JEKYLL_ENV=production', '--strict_front_matter'
  end
end

The build:test task is slightly more complicated, but is in fact mainly HTMLProofer configuration:

require 'html-proofer'
namespace 'build' do
  task 'test' => %w[compile]
  task 'test' do
    options = {
      :allow_hash_href => true,
      :assume_extension => true,
      :check_favicon => true,
      :check_opengraph => true,
      :file_ignore => [
        /\/blog\/software-craftsmanship\/20[01]\d\//, # Do not check old blog posts
        /\/sections\//
      ],
      :root_dir => 'tmp/obj/bin/',
      :url_ignore => [
        'http://html5up.net', 'https://chrisbobbe.github.io/', # Included by the template
        'https://www.facebook.com/mathieu.cartoixa', # 404 only when checked...
        'https://www.linkedin.com/in/cartoixa/', 'https://www.linkedin.com/in/cartoixa/?trk=profile-badge' # 999 only when checked
      ]
    }
    HTMLProofer.check_directory('tmp/obj/bin/', options).run
  end
end

And then the build:package task requires an external package for the creation of the archive: the default PackageTask does not seem to support well the archiving of files outside the root of the project. This gives something like:

require 'zip'
namespace 'build' do
  namespace 'package' do
    PACKAGE_FILE = 'tmp/out/bin/www.mcartoixa.me-0.0.0.zip'
    file PACKAGE_FILE => %w[build:compile]
    file PACKAGE_FILE do
      FileUtils.mkdir_p('tmp/out/bin/')
      Zip::File.open(PACKAGE_FILE, Zip::File::CREATE) do |zf|
        Rake::FileList.new('tmp/obj/bin/**/*').each do |f|
          zf.add(f.delete_prefix('tmp/obj/bin/'), f)
        end
      end
    end

    task 'build' => PACKAGE_FILE
  end
  task 'package' => %w[package:build]
end

Please note here that one of the main features of a build tool is the ability to describe dependencies between targets. This is demonstrated here:

the build:package task depends on build:package:build.
the build:package:build task depends on the rule to build tmp/out/bin/www.cartoixa.me-0.0.0.zip.
the rule to build tmp/out/bin/www.mcartoixa.me-0.0.0.zip depends on build:compile (the static website needs to have been created before attempting to package it).This might seem a bit confusing to the newcomer but it is in fact very powerful (when used sensibly).

The last thing I would like to add here is that HTMLProofer is a fine tool but:

I still have to find a way for it not to complain when I just added a new blog post that it in fact does not exist (yet).
sometimes websites will fail because… reasons, and it should not prevent my own build from succeeding. I guess I should make sure that it only triggers warnings instead of errors.

The script file

Having managed to run an instance of Redmine on Windows Server 2012 and IIS, I figured Ruby was a cross-platform technology. In reality I have since experienced that many dependencies that require native compilation are poorly (or not at all) maintained: running a Rails application with MariaDB and Redis connections is a nightmare (when possible at all).

Disclaimer aside (no build.bat then), the gist of the build.sh file is simply (using Bundler here):

if bundle check > /dev/null; then
  bundle clean
else
  bundle install --standalone --clean
fi

bundle exec rake build:${TASK}

The CI configuration file

We need a CI platform that runs MacOS or Linux, so Travis CI will do just fine. Once again, having dealt with all the difficulties in the build file, the .travis.yml is just:

before_install:
- gem install bundler
install:
- bundle install --standalone --clean --jobs=3 --retry=3
script:
- bundle exec rake build:release

A more complete project

This is where the trouble really begins. Having tried to apply the same principles to a more complex Rails project I came against difficulties that seem hard to overcome. This may be because of my lack of knowledge of the environment itself, but it might also be because of conflicting conventions. I have the sense for instance that everything is done to allow the developer to achieve everything (and anything) from the command line. This might mean many things like:

you don’t have to work in another directory, and in fact you should not. It seems quite a hassle (compared to Javascript for instance) to handle a copy of the source code elsewhere where you retrieve only production dependencies for packaging purposes.
some tools have memory. Bundler will remember your current environment (development, production) for instance.
worse, the so called Deployment Mode will explicitely screw up your development environment.

Hard to run your build locally with these features!… This is not intended as a rant though, and if you have some insight on how to reconcile both worlds: please share.

Though I cannot share a real and complete project here I can show Rakefile elements that can be of some use:

how to run Ruby to check your source files syntax (this goes in the build:compile target):

SOURCE_FILES = Rake::FileList.new('.ruby-*', 'config.ru', 'Gemfile*', 'Rakefile', 'app/ **/*', 'bin/** /*', 'config/ **/*', 'db/** /*', 'lib/ **/*', 'public/** /*', 'vendor/**/*') do |fl|
  fl.exclude(/^config\/app_parameters.yml$/)
  fl.exclude(/\.log$/)
end
RUBY_FILES = Rake::FileList.new(SOURCE_FILES.dup.to_a.select do |path|
  path.ends_with?('.rb')
end) do |fl|
  fl.exclude(/^vendor\//)
end

namespace 'build' do
  rule '.rb.log' => [
    proc { |tn| tn.gsub(/\.rb\.log$/, '.rb').sub(/tmp\/obj\//, '') }
  ] do |t|
    FileUtils.mkdir_p(File.dirname(t.name))
    ruby "-wc #{t.source} > #{t.name}"
  end
  task 'compile' => RUBY_FILES.gsub(/\.rb$/, '.rb.log').sub(/^/, 'tmp/obj/')
end

We define rules for *.rb.log files that will be generated by running Ruby on the corresponding *.rb file and storing the output. The task we define then depends on the generation of those files.

how to run Reek on your Ruby files (this creates a reek target):

require 'reek/rake/task'
Reek::Rake::Task.new do |t|
  t.config_file = '.reek.yml'
  t.source_files = RUBY_FILES
  t.reek_opts = '-s --force-exclusion --no-progress'
end

how to run Rubocop on your Ruby files (this creates a rubocop target):

require 'rubocop/rake_task'
RuboCop::RakeTask.new(:rubocop) do |t|
  t.formatters = ['clang', ['html', '-o', 'tmp/rubocop-results.html']]
  t.patterns = RUBY_FILES
end

As I wrote before, should you choose to use static analysis tools such as Reek and/or Rubocop (and you should), you should do it as soon as possible in the life of your project in order to avoid to have hundreds of warnings to correct at once…

Packaging proves the more challenging because when you are generating assets as part of the build (like minified Javascript files for instance), then you have to create file lists after those files have been generated (target assets:precompile in a Rails project). This gives something like:

require 'rake/packagetask'
PACKAGED_FILES = Rake::FileList.new(SOURCE_FILES.dup) do |fl|
  fl.exclude(/^app\/assets\//)
end
namespace 'build' do
  namespace 'package' do
    pt = Rake::PackageTask.new('my_application', '0.0.0') do |t|
      t.need_tar_gz = true
      t.package_dir = 'tmp/obj/bin/'
      t.package_files = PACKAGED_FILES
    end

    file pt.package_dir_path + '/public/assets' => %w[assets:precompile]
    file pt.package_dir_path + '/public/assets' do
      pt.package_files = pt.package_files + Rake::FileList.new('public/assets/**/*')
    end

    task 'build' => pt.package_dir_path + '/public/assets'
    task 'build' => %w[package]
    task 'build' do
      FileUtils.mkdir_p('tmp/out/bin/')
      FileUtils.mv(File.join(pt.package_dir, pt.tar_gz_file), 'tmp/out/bin/')
    end
  end
  task 'package' => %w[package:build]
end

The whole build kind of works when specifying specific command lines in the CI configuration (again this cannot be part of the local build):

bundle install --standalone=test --deployment
RAILS_ENV=test bundle exec rake build:release

Not ideal as we will be packaging with test dependencies as well (things like capybara or RSpec Rails). But hey: work in progress.

Continuous Integration for node.js

Mac — Mon, 14 Sep 2020 05:42:00 +0000

Applying the same principles to node.js is interesting because this is an entirely different world from .NET (for the best and for the worst):

Javascript is interpreted: there is no compiler and no inherent notion of packaging.
the platform is very fragmented: even the package manager seems to be up for grabs (stigmergy, anyone?).

And so in this this installment we will have a peak at how to implement Continuous Integration in a way that is both consistent with what we have seen before and specific to this platform:

A simple project
A more complete project

A simple project

leaflet-binglayer was a plugin for Leaflet (“an open-source Javascript library for mobile-friendly interactive maps”). Its goal was essentially to integrate Bing Maps layers in a way that did not consume too many transactions. This is a browser library, and not a proper node.js project, but the principles and the toolkits are remarkably the same.

mcartoixa / leaflet-binglayer

Yet another Leaflet plugin that handles Bing maps layers

The main elements of the project are:

The build file (gulpfile.js).
The script file (build.bat) that helps executing the build file locally.
The CI configuration file (.travis.yml).

The build file

I have chosen gulp.js for the build description, mainly because it is the only build tool that allows it to be pure Javascript. It also hardly knows how to do anything by itself, which is always a good thing in a build tool I believe. Other alternatives I know of seem lacking in at least one regard:

grunt.js is a fine tool but the description is mainly JSON, which was only designed as a data interchange format and notoriously lacks comments…
webpack is a very powerful module bundler that has so many features that it can sometimes masquerade as a build tool. But once again the description is only JSON (did I mention that it lacked comments?) and it makes complex scenarii convoluted and sometimes impossible. I like to keep it as a module bundler, which it does best, as part of a larger build described elsewhere.

Key elements of the build system are:

you have access to native node.js functions, which allow for simple (as well as very complex) configurations:

gulp.task('clean', function () {
  const del = require('del')
  return del(['tmp/'])
})

you have access to a large library of plugins to configure common tools and scenarii (eslint in this case):

const plugins = require('gulp-load-plugins')({ lazy: true })
gulp.task('analysis-eslint', function () {
  return gulp.src('src/**/*.js')
    .pipe(plugins.eslint())
    .pipe(plugins.eslint.format())
    .pipe(plugins.eslint.failAfterError())
})

Consistency reduces my cognitive load, thus the targets defined in the build file (gulpfile.js) are more or less the same as in .NET:

clean: cleans the build (the tmp\ directory).
compile: “compiles” (ie minifies) the source code using UglifyJS.
analysis: performs static analysis on the project with eslint, and then gathers statistics using the cloc utility.
package: simply copies the minified files to the tmp\out\bin\ folder.
build: shortcut for the combination of analysis and compile.
rebuild: shortcut for the combination of clean and build.
release: shortcut for the combination of clean, build, and package.

Automated testing is hard in this configuration (a plugin based on the visual integration of an external API) and so there is no test target (what are your excuses?).

And then as a convenience, for ease of use and discovery of our system, these targets are referenced in the main package.json file, which will then be used as the main entry point:

{
  "scripts": {
    "analysis": "gulp analysis",
    "clean": "gulp clean",
    "compile": "gulp compile",
    "package": "gulp package",
    "build": "gulp build",
    "rebuild": "gulp rebuild",
    "release": "gulp release"
  }
}

The script file

Nothing new here, as the whole build has been described above and the script is just here to allow us to easily build locally:

CALL npm.cmd install --no-package-lock --no-shrinkwrap --loglevel info --cache .tmp\npm-cache
CALL npm.cmd run-script %TARGET% --loglevel %VERBOSITY%

Well, almost nothing:

First of all as our technology is cross platform and as our scripts are very simple, it is easy to make our build work on Windows and MacOS for instance. Just add a simple build.sh file (not forgetting the execute mode permissions of course):

npm install --no-package-lock --no-shrinkwrap --loglevel info --cache .tmp/npm-cache
npm run-script $_TARGET --loglevel $_VERBOSITY

Some external dependencies have to be installed prior to the build. For instance I like to gather statistics on my projects using the cloc utility, which is much easier installed locally (this is a simple download) outside of the proper build.
- By convention local installations go to the .tmp\ directory (notice the dot).
- In the build\ folder (by convention) I have a SetEnv.bat script that will be called by other build scripts on Windows that can initialize the environment for the build (detect installation paths using the mighty Windows registry, initialize environement variables, warn of missing dependencies…) and (locally) install missing ones:

IF NOT EXIST "%CD%\.tmp\cloc.exe" (
    IF NOT EXIST .tmp MKDIR .tmp
    powershell.exe -NoLogo -NonInteractive -ExecutionPolicy ByPass -Command "& { [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12; Invoke-WebRequest https://github.com/AlDanial/cloc/releases/download/v$Env:_CLOC_VERSION/cloc-$Env:_CLOC_VERSION.exe -OutFile .tmp\cloc.exe; }"
)

You can do the same for other platforms in a .bashrc file (not forgetting the execute mode permissions):

if [! -d .tmp]; then mkdir .tmp; fi
if [! -f $(pwd)/.tmp/cloc.pl]; then
    wget -nv --show-progress -O .tmp/cloc.pl https://github.com/AlDanial/cloc/releases/download/v$_CLOC_VERSION/cloc-$_CLOC_VERSION.pl
fi

Handling your external dependencies this way as much as possible can significantly reduce the need for documentation (in the pre-requisite section) and cut down the time needed for a new developer to jump in your project. I know it saved me much time when I got back to this project 2 years later…

The CI configuration file

Having already handled all the difficulties our configuration file could hardly be more simple, which is the goal. For instance using Travis CI, the gist of the .travis.yml is only:

install:
  - . build/.bashrc
  - npm install --no-package-lock --no-shrinkwrap

script:
  - npm run-script release --loglevel notice

A more complete project

Well, I do not think I have much to show you here. But every useful concept has been touched in the above (not so simple then) project. Simply know that I have been able to integrate this pipeline with success in other kinds of projects:

Chrome extensions, which required JSON transformation of configuration files depending on the environment (to set up OAuth credentials for instance).
proper node.js projects (backend and frontend). The main trick there is to be able to create a deployable package: you will want to execute npm install --production in a temporary folder, along with a copy of your backend source files, to be sure to package only the dependencies that are relevant to production. I leave it as an exercise for you, but remember you have the whole power of Javascript at your disposal (instead of YAML or JSON), so this should be fairly achievable (I did that using grunt.js a looong time ago).

Continuous Integration for the .NET Framework

Mac — Mon, 07 Sep 2020 05:59:00 +0000

I realize writing this post that I have been practicing .NET development (C# in particular) since 2005 (15 years!). It is natural for my Continuous Integration practices to have been heavily influenced by this platform during all this time. The advent of .NET Core will be the occasion to revisit these, and also the subject of another blog post. This post shows what those years of maturation led to on the (soon legacy) .NET Framework platform.

A simple project
A more complete project

A simple project

For a starter, let’s look at how the principles apply to a relatively simple (and largely unfinished) .NET Framework project: meet NetMonkey, a .NET wrapper for the MailChimp API (it was version 3.0 at the time). What you need are:

A solution file (NetMonkey.sln).
A build file (NetMonkey.proj), written in MSBuild.
A script file (build.bat) that helps executing the build file locally.
A CI configuration file (appveyor.yml).The rest of the project is either code of infrastructure.

mcartoixa / NetMonkey

A .NET wrapper for the MailChimp API v3.0.

The solution

Solutions are Visual Studio speak for a collection of related projects. You can load them (with Visual Studio), and you can also build them (with MSBuild).

In this context, solutions have 2 purposes:

they are an entry point for developers to edit the code.
they are an entry point for the build scripts to generate a package. This is why there are actually 2 solutions in the project:
- The main solution, NetMonkey.sln is used to generate the library.
- The second solution NetMonkey.Tests.sln also contains the associated unit tests. The main project is shared with the previous solution, so this is the one the developer will be encouraged to develop against (being more complete).

The main point here is that developers can still use their usual toolkit to develop (Visual Studio in this case).

The build file

This is the crux of the build, written in MSBuild. I might also have considered Cake, or even psake but for me nothing beats MSBuild. It is an acquired taste though, and it most certainly deserves its own dedicated blog post. Meanwhile the keypoints are:

In .NET Framework the whole build system is based upon MSBuild, and project files are proper, editable MSBuild files (though notably solutions are not). In fact Visual Studio could be thought of as a visual editor for MSBuild projects. Using MSBuild means tighter integration with the .NET Framework platform.
The logging capabilities of MSBuild are just amazing (yes, really).
I don’t mind XML (yes, I’m that old).
There were no real alternatives 15 years ago anyway; NAnt was a step back IMHO. More on that another time…

The gist of this file is deceptively simple:

<Project xmlns="http://schemas.microsoft.com/developer/msbuild/2003" DefaultTargets="Rebuild" ToolsVersion="14.0">
  <ItemGroup>
    <Projects Include="NetMonkey.sln" />
  </ItemGroup>

  <Import Project="$(MSBuildProjectDirectory)\packages\Isogeo.Build.*\tools\build\Isogeo.Common.targets" />
</Project>

It just says that the main solution is the NetMonkey.sln file. As I adhere to my own conventions (as I easily tend to) I was able to abstract the essential part of the build in a library that can be shared accross many projects. The drawback is that after 15 years of trying to handle every situation (like web applications and native applications for instance), the common library tends to become heavy and difficult to get into (see for yourself https://github.com/isogeo/Isogeo.Build/blob/531d173efc326afceb013a6ff841e58ffcdaff25/files/build/Isogeo.Common.targets). But with a simple definition like the above it provides the following build targets:

Clean: cleans the build.
- This is usually a simple matter of deleting the tmp\ folder, as every other target generates its outputs there.
Compile: compiles the specified solutions.
Test: tests the project.
- More specifically, if a solution exists that has the same name as the provided solution but with a .Tests.sln suffix (like NetMonkey.Tests.sln in this case) it compiles it and executes the tests.
- It also uses OpenCover to check code coverage.
Analysis: performs static analysis on the project.
- If FxCop is detected it performs a legacy analysis.
- If a SonarQube configuration file is present it performs an analysis.
- It gathers statistics using the cloc utility.
Document: generates documentation (in the tmp\out\bin folder) for the project using SHFB.
Package: generates a deployable package (in the tmp\out\bin folder). Depending on the kind of solution being provided this can be:
- A zip file.
- A NuGet file for libraries (NuGet being the dependency manager of choice on the .NET platform).
- A Web Deploy package for web applications.
Build: shortcut for the combination of Compile and Test.
Rebuild: shortcut for the combination of Clean and Build.
Release: shortcut for the combination of Clean, Build, Document, Package and Analysis.

Providing the same structure reduces my cognitive load when going from project to project: I always know where to start. Also note that most of the tools used generate reports (int the tmp\ folder), that can be sent to various platforms of choice (like AppVeyor or CodeCov…).

The script file

The script file allows for easy local execution of the build file. What may not be easy is:

MSBuild is usually not in your %PATH%. I consider this a feature: it allows you to have multiple versions installed and dynamically choose the version you want at runtime, thanks to the Windows Registry.
You can add advanced MSBuild arguments to improve your build. For instance a complete log of the build can be generated for you to inspect in order to debug anything that might be wrong.
You could need to check hard dependencies (like the .NET Framework itself!)

Here is the core of the batch file (cf. https://github.com/mcartoixa/NetMonkey/blob/master/build.bat):

PUSHD .nuget
NuGet.exe restore "packages.config" -PackagesDirectory ..\packages
POPD
msbuild.exe %PROJECT% /nologo /t:%TARGET% /m:%NUMBER_OF_PROCESSORS% /p:GenerateDocumentation="%GENERATE_DOCUMENTATION%" /fl /flp:logfile=build.log;verbosity=%VERBOSITY%;encoding=UTF-8 /nr:False

Anyway, now you can test the build locally: build.bat.

The CI configuration file

A looong time ago I used CruiseControl.NET to build my projects, and it served me well. Nowadays there are many options in the cloud that are much more practical, like AppVeyor for instance which has provided Windows and .NET integration for many years now. It was inevitable that I made the switch at some point.

But as my build is not tied to any CI platform, the transition was very simple. It keeps my YAML very simple (check for yourself https://github.com/mcartoixa/NetMonkey/blob/master/appveyor.yml), and I could easily switch again, to Azure DevOps for instance.

The configuration is the equivalent of build.bat for the CI platform. It only adds the handling of packages (artifacts in AppVeyor speak) and releases. All the packages being output in a single folder (tmp\ou\bin\ by convention), the configuration is still very simple:

artifacts:
  - path: tmp\out\bin\*.nupkg
    type: NuGet
  - path: tmp\out\bin\*.zip

So it is mainly about releases.

A more complete project

Meet GeoSIK, a set of libraries that were destined to ease the development of OGC Web Services in .NET. It provided 11 libraries that were used to either implement those services or integrate them with external geospatial libraries (like ProjNet of Sql Server Spatial Data types).

mcartoixa / GeoSIK

GeoSIK is a set of libraries that help develop OGC Web Services in .NET.

Even though this project is much more complex, its structure is the same. And if you understood the structure of the simple project above you should not have too many problems delving into this one now. The only major change is that to keep things simple it was decided to have all the libraries built by 1 single solution (instead of the 11 mandated by the principle 1 package, 1 solution). This required a specific packaging system, and so the GeoSIK.proj (https://github.com/mcartoixa/GeoSIK/blob/master/GeoSik.proj) is a bit more complex.

To me this shows that although it relies heavily on conventions, this build system is still quite adaptable. In fact the whole system can be adapted to other platforms.