DEV Community: MeStrak

Pop culture design patterns: creational

MeStrak — Sun, 17 Oct 2021 18:28:44 +0000

During a recent interview I asked the candidate about their understanding of design patterns. I was impressed not only with their honesty, telling me they only had basic knowledge, but also with their description of the Singleton pattern:

It's like Highlander … there can be only one.

Since then I discovered that there are already some memes floating around with the same joke, but it made me chuckle anyway. It got me thinking that perhaps there is something in the idea of using pop culture references to remember the fundamental goals of design patterns.

This series is my attempt to choose some good references. It's not to be taken too seriously, and I won't provide in depth explanations of each pattern as there are already amazing sites for this like https://refactoring.guru.

Let's try creational patterns first. Here goes …

Singleton: Highlander - there can be only one

Image sourced from: https://www.vintagemovieposters.co.uk/

Singleton ensures that only one instance of the class which implements it can exist. Global access to that object can be used within state management applications (think Redux).

If you haven't seen The Highlander, you're missing out. I love it, even though it's 80s cheese. Immortals battle it out to be the only immortal, because just like the Singleton pattern 'THERE CAN BE ONLY ONE!'.

Other similarities between Singleton and The Highlander:

They're both classics, and both much criticised. I'm not afraid to say that I love both of them - I can still remember my first time implementing a Singleton.
In The Highlander, after beheading another immortal, all knowledge that person has obtained during their lifetime is transferred to the surviving immortal in a process called The Quickening. One of the criticisms of Singleton is that it can be used as a global store for everything, resulting in separate components of an application knowing too much about each other.

Builder: Five Guys

Image sourced from: https://www.timeout.com/

The goal of the Builder pattern is to construct complex objects, and allow you to create variations on those objects using the same code.

A frequently used analogy for this pattern is assembling a meal in a fast food restaurant. My somewhat limited imagination allowed me to think of Five Guys. Think of making a burger as a complex task - implement the Builder pattern to call:

AddPickles()
AddBacon()
AddCheese()
AddKetchup()

Obviously if those are your only choices of Five Guys burger toppings something has gone wrong, but hopefully you get the idea.

Builder also has an optional implementation of a Director, further abstracting the implementation and simplifying things for the client. This is like your Five Guys cashier who takes your order for a burger all the way, and a crazy milkshake with 11 mixins, and then sends all required instructions to the builder team who assemble and deliver your meal.

Prototype: Futurama S6E17 - Benderama

Image sourced from: https://www.assignmentx.com/

The Prototype pattern is used to remove complexity from copying objects. Objects that implement the Prototype pattern know how to clone themselves, so code that needs to make a copy does not need to know the complexity of the class.

In Benderama, Bender makes clones of himself. Unfortunately each clone is smaller than the previous one and when they reach molecular level they nearly destroy the world. Obviously knowing that detail this analogy falls down entirely.

Factory Method: The Pepsi Challenge

Image sourced from https://abcnews.go.com/

Factory Method provides a way for a class to create similar objects, but to leave the individual subclasses to use their own logic. This is done by defining an interface implemented by all subclasses so individual objects can be created using common methods, but without knowing the details of what's inside.

First of all, in case you don't know what it is, the Pepsi Challenge is a marketing campaign from the 1980s where shoppers would do a blind tasting of Coke and Pepsi and say which one they preferred.

Why is Factory Method similar to the Pepsi Challenge? Think of an abstract class called SoftDrink which provides a creational method.

CreateDrink()

And an interface implemented by every soft drink providing the following common methods:

FillBottle()
OpenBottle()
PourDrink()
DrinkCola()

This is Factory Method! The creational logic of each drink is the secret recipe closely guarded by Coke or Pepsi. When the consumer takes the challenge, they call the DrinkCola() function but they do not know if they are doing that for Coke or Pepsi, thanks to the Factory Method which hides that complexity (in the form of a big yellow box).

Using this approach, Pepsi could add as many drinks as they want to into the challenge. They don't really need to do that though, as instead through shameless capitalism the two companies have successfully wiped out or bought out most other competition.

One drawback of the Factory Method is that requires implementation of many subclasses. This can make quite a simple concept harder to understand than intended when actually implemented, just like my attempt to use the Pepsi Challenge to explain this pattern.

Abstract Factory: IKEA

Image sourced from https://about.ikea.com/

You may guess from the name that this one adds a layer of abstraction to the Factory Method. It is used for creating groups of related objects (a family). The Abstract Factory class holds a bunch of abstract creation methods which know how to create instances of the objects. Concrete factories will then be called to actually create the object.

Think of each family like a set of matching furniture from Ikea. The furniture family might contain a chair, sofa and a table. Different style chairs would have the same chair interface, the create() method would implement the style of that chair's given family. The factory for a given family will call the specific create() methods to create each furniture type with the correct style matching the family.

I'm sorry to say that I had a lack of imagination for this one, and the chosen analogy is the same as several other examples online. One of those is the excellently written article on https://refactoring.guru/ which has a great explanation. I recommend that you spend some time reading that, and hopefully the Ikea link will start to stick.

And there you have it, hopefully some memorable references for Singleton, Builder, Prototype Factory and Abstract Factory creational design patterns.

Look out for further posts in this series where I will try to do the same for structural and behavioural patterns.

Fin.

Testing secure APIs by mocking JWT and JWKS

MeStrak — Mon, 04 Jan 2021 12:12:11 +0000

Recently, I've had some time to get back to my pet project Catkin. I'm working on gradually improving the testing which was sorely neglected when I created the initial prototype app.

When implementing end to end API tests I quickly ran into the issue of a missing authentication token as there is no logged-in user. As I'm using Auth0 to keep things nice and simple for my user login implementation, I have no easy way to login a user from an endpoint directly on the backend which is the usual approach.

In this article I'll explain how I solved that problem.

I use the Jest for running my tests. In writing this I'm assuming that you have the basic framework up and running already so that you can run tests against your API. The full setup of jest is not covered.

The Catkin user authentication process

First let's look at how users log in to Catkin. In the Catkin login flow the following happens:

The application frontend connects directly to Auth0 to get a JWT token.
The token is then added to the authorisation header of each request from the frontend to the backend API.
Upon receiving a request, the backend validates that the token was generated by Auth0 and is valid for Catkin. This is done by the Auth0 JWKS endpoint.
If the token is valid, the requested query/mutation is executed. If not, then a 401 Unauthorized code is returned.

Quick definitions

Just in case you're not familiar with the terms, two fundamental things to know are:

JWT: JSON Web Token - a secure token signed by the authentication provider using a secret key. This contains the details of the authenticated user and can be used to securely store other information such as user security roles. Read more.
JWKS: JSON Web Key Set is a list of the public keys which can be used to verify the JWT. They are stored by the authentication provider and used in step 3 of the process described above. For Auth0 the JWKS is always found at https://your_auth_domain.xx.auth0.com/.well-known/jwks.json Read more.

For the artists among you

Here's a picture ...

Image sourced from https://auth0.com/docs/architecture-scenarios/web-app-sso/part-1.

And here's another one. Simpler. Better. But you have to imagine that instead of REST it says GraphQL 😉.

Image sourced from https://hceris.com/setting-up-auth0-with-terraform/.

With that covered, it's now time to think about how we can test our API with this additional layer of complexity.

Testing approach

I need to test:

That Catkin the Catkin GraphQL API returns the correct query results/performs the expected mutation.
That the security applied to the API works.

With the authentication flow that, is in place any unauthenticated user will be rejected. This obviously makes testing the API a little more difficult, as tests must run as an authenticated user.

The two most obvious approaches to test the secured API are:

Connect to Auth0 during test execution to get a token.
Mock a JWKS endpoint and use that for testing.(A JWKS endpoint is the thing that actually validates that the JWT is legitimate).

I would prefer to avoid option one, even though the Auth0 free tier would be enough to support my testing needs. Option two is cleaner, and my chosen approach which I will cover below. It means that if anybody else wants to use the Catkin code they would not be tied in to using only Auth0 or having an external connection available.

Implementation

Now that we know the theory and have decided the approach, let's have a go at implementing it.

Mocking the JWT and JWKS

To fully mock the authentication process, we need to achieve the following:

Create a JWT without depending on Auth0.
Allow the backend to verify the JWT without connecting to Auth0.

To do both things, we can use a lovely little library called mock-jwks which was created for exactly this use case.

Mock-jwks works by intercepting calls to Auth0 (or actually any OAuth service) using nock. Nock helps us to perform isolated testing of modules which make HTTP requests by intercepting those requests before they are sent to the external service and allowing us to act on them. Once the request to the JWKS endpoint has been intercepted, mock-jwks can then validate (or not) the JWT which is being passed to it.

First, install the libraries:

yarn add mock-jwks nock --dev

Now in our tests we can create a mock Auth0 endpoint with the following code:

const jwks = createJWKSMock('https://catkin-dev.eu.auth0.com/');
jwks.start();

Then generate a token as below. For the Auth0 token you should specify the reserved claims audience (aud) and issuer (iss) as you have set up in your environment variables. The https://catkin.dev/permissions is specific to Catkin and an example of how you can use custom data in Auth0 which will be added to your token:

const token = jwks.token({
    aud: "https://catkin.dev",
    iss: `https://catkin-dev.eu.auth0.com/`,
    'https://catkin.dev/permissions': [
        {
            "group": "*",
            "role": "admin"
        }
    ],
});

The token can then be added to any request header:

it('Creates an item when user is logged in', async () => {
  const res = await request(global.app.getHttpServer())
    .post('/graphql')
    // add the token to the request header
    .set('Authorization', 'Bearer ' + global.validAuthToken)
    .send({
      operationName: null,
      query: createItemQuery,
    })
  const data = res.body.data.createItem;
  expect(data.title).toBe(item.title);
});

Now whenever your backend tries to check something with Auth0, mock-jwks will intercept the request using nock, and do the check instead. No external connection is required.

Likewise, we can also test that our endpoint rejects unauthenticated users by omitting the Authorization header:

 it('Throws an error when API is called with no token', async () => {
    const res = await request(global.app.getHttpServer())
      .post('/graphql')
      // send the request without the auth token
      .send({
        query: CREATE_ITEM_GQL,
        variables: {
          createItem: item,
        },
      });

    expect(res.body.errors).toBeTruthy;
    expect(res.body.errors[0].extensions.exception.status)
    .toBe(401);
  });

Finally, at the end of the tests, or if we want to break the auth service for further testing, simply stop the JWKS server.

jwks.stop();

Cleaning up the code

The basic test is now in place but the implementation is a bit messy. To help with re-use of the code, let's implement a helper file which contains all code for setting up the JWKS mock, generating tokens, etc. Auth service settings should also not be hard-coded; they will instead be passed to this helper function allowing us to provide incorrect details in the token to simulate an invalid token.

auth.helper.ts

import createJWKSMock, { JWKSMock } from 'mock-jwks';

export function startAuthServer(jwksServer: string): JWKSMock {
  const jwks = createJWKSMock(jwksServer);
  jwks.start();
  return jwks;
}

export function getToken(
  jwks: JWKSMock,
  authDomain: string,
  authAudience: string): string {
  const token = jwks.token({
    aud: [`${authAudience}`, `${authDomain}/userinfo`],
    iss: `${authDomain}/`,
    'https://catkin.dev/permissions': [
      {
        group: '*',
        role: 'admin',
      },
    ],
    sub: 'testprovider|12345678',
  });
  return token;
}

export function stopAuthServer(jwks: JWKSMock) {
  jwks.stop();
}

These functions are then called from my global setup.ts file beforeAll() and afterAll functions, providing a global JWKS endpoint and JWT that can easily be reused in all tests. Take a look at the full setup here: https://github.com/MeStrak/catkin.

Wrap up

As the objective of Catkin is to provide a hosted environment for several organisations, the security must be rock solid. Thanks to mock-jwks it was straightforward to mock the whole authentication process allowing the API to be fully tested, including fail cases for unauthenticated users.

I now have a simple framework in place allowing me to quickly write tests simulating authenticated or unauthenticated users.

The next step will be to simulate authenticating as users with different roles to check that granular security levels work correctly.

Thanks for reading! I deliberately kept this fairly brief to provide an overview. I hope that even at this high level the article is still useful. As always, I'm happy to answer any questions you may have.

Fin.

Semantic release with Python, Poetry & GitHub Actions 🚀

MeStrak — Sun, 20 Dec 2020 23:05:34 +0000

I'm planning to add a few features to Dr. Sven thanks to some interest from my colleagues. Before doing so, I needed to set up some proper release management as I will be introducing some breaking changes as well as automated deployment.

To manage this automatic versioning, the obvious choice is semantic versioning/semantic release which automatically increments a version number based on commit conventions.

Here's how ...

What I want to achieve

Automatic version number increment of releases.
Version numbers based on conventional commit format (Semantic Versioning).
Automatically create a release object on GitHub.
Only allow merging to main when tests and linting are passing.
Perform a release following a merge to the main branch.

TL;DR

If you don't want to read the detail you should be able to implement this in under 10 minutes.

Install this GitHub bot to enforce semantic PRs: https://github.com/apps/semantic-pull-requests.
Set the same version number in pyproject.toml and __init__.py. For example:

./pyproject.toml
```
[tool.poetry]
name = "your-project"
version = "0.0.1"
```
src_folder/__init__.py
```
__version__ = '0.0.1'
```

Add Semantic Release section to pyproject.toml:

[tool.semantic_release]
version_variable = [
    "src_folder/__init__.py:__version__",
    "pyproject.toml:version"
]
branch = "main"
upload_to_pypi = false
upload_to_release = true
build_command = "pip install poetry && poetry build"

Create the file .github/workflows/ci.yml within your repo, and copy the contents of this file into it to set up your CI action: https://github.com/MeStrak/dr-sven/blob/main/.github/workflows/ci.yml.

Done! Read on if you would like a little more explanation.

Background

Before diving into the code, let's do a light review of the principles behind Semantic Versioning to understand how the version numbers are generated.

Conventional Commits

This is a specification designed to ensure that commit messages are human-readable. Developers using this convention use specific commit prefix text such as fix: for commits which fix a bug, and feat: for commits which add a new feature. The most common convention is the convention developed by the Angular team which includes the following keywords:

build:, chore:, ci:, docs:, style:, refactor:, perf:, test:.

A scope can also be added in parentheses. For example: feat(blog): add code parsing.

BREAKING CHANGE can also be added into the commit text to indicate that the commit is not backwards compatible.

These commit conventions are then used in the Semantic Versioning specification to calculate the version number.

Have a look at the full specification here.

Also, here are two tools worth investigating to help enforce this convention in all commit messages:

Semantic Versioning

Semantic Versioning, or SemVer is a specification defining a clear, logical version numbering system for an API. In this case, I'm actually using it to version an application with no external API, but the specification still holds very well because I have a requirement to indicate whether a new release is backwards compatible with configuration files.

I would recommend reading the full specification, but here is an excellent summary taken directly from the specification:

Given a version number MAJOR.MINOR.PATCH, increment the:

MAJOR version when you make incompatible API changes,

MINOR version when you add functionality in a backwards compatible manner, and

PATCH version when you make backwards compatible bug fixes.

Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format.

Take a look at the full specification here.

Various packages existing to calculate a SemVer version number based on conventional commit messages. We'll be using Semantic Release to do this, and perform a release at the same time.

Semantic Release

Semantic Release is a tool which automatically sets a version number in your repo, tags the code with the version number and creates a release (for example publishing it to NPM, or PyPi).

This is done using the contents of Conventional Commit style messages to generate a version number. For example, fix: will bump the PATCH version, feat: will bump the minor version, and a message containing BREAKING CHANGE will bump the major version to indicate that the release is not fully backwards compatible with the previous version.

The original (as far as I know) semantic-release package was created for node projects and can be found here.

We'll be using the Python implementation of that project, Python Semantic Release.

Implementation

Now that we have covered the basic concepts, let's look at how to implement them. We need to do the following things, as listed in the TL;DR section but this time with a bit more explanation:

Ensure Conventional Commits are used.
Configure Semantic Release in pyproject.toml.
Configure a basic quality CI step using GitHub Actions.
Add a release step using Python Semantic Release.

Ensure Conventional Commits are used

The use of Conventional Commits is central to our whole versioning strategy. If it's not used then nothing will work. To enforce use of this type of commit on PRs I'll be using a GitHub bot which is easy-peasy to install: https://github.com/apps/semantic-pull-requests.

Just install that bot for your GitHub project, and you'll automatically see a new check for any new PR created on your project. If the PR doesn't contain commits which comply with the Conventional Commit specification then the check will fail.

As a later enhancement, I will add a pre-commit hook to the repo to check the format of local commit messages because it can be pretty annoying for developers to discover issues like this only when a PR is raised.

The first step is done! Easy right?

Configure Semantic Release in pyproject.toml

Semantic Release can be configured in pyproject.toml according to your requirements. The main things to note here:

As we're using Poetry the version variable is managed in both __init.py__ and pyproject.toml.
Branch is set to main, as we're using the GitHub flow branching strategy. We want our releases to be performed using code on the main branch, which is where all code is merged to for production deployment. If you use a different strategy just change the name.
upload_to_pypi has been set to false because Dr. Sven is not published on PyPi, so this is irrelevant for me.

[tool.semantic_release]
version_variable = [
    "dr_sven/__init__.py:__version__",
    "pyproject.toml:version"
]
branch = "main"
upload_to_pypi = false
upload_to_release = true
build_command = "pip install poetry && poetry build"

Configure a basic quality CI step using GitHub Actions

If you're reading this, I assume you're already aware that GitHub Actions is GitHub's CICD workflow solution. Workflows are configured in YAML files stored in the .github/workflows folder in your repo.

I'm using the nice and simple GitHub flow branching strategy, where all bugfixes and features have their own separate branch, and when complete each branch is merged to main and deployed.

The Dr. Sven CI pipeline will consist of 2 main stages, known as jobs:

Quality: runs linting and automated tests on the code, preventing merging if any of these steps fail.
Release: runs following the successful quality job when code has been merged to the main branch.

The full workflow file can be found here. Although the structure and syntax are simple, let's break it down to make sure we understand each part.

The header defines that the CI workflow will run whenever there is a direct push to main, or a pull request to merge another branch to main.

name: CI
on:
  push:
    branches: [ main ]
  pull_request:
    branches: [ main ]

The quality job will use an ubuntu runner instance. The runner is the name of the application which runs GitHub Actions jobs, such as unit tests in almost the same way as you would run them on your development PC.

The checkout action will checkout the code so that it is available for you to use in the executing runner.
setup-python will install Python on the runner.

jobs:
  Quality:
    runs-on: ubuntu-latest
    steps:
    - uses: actions/checkout@v2
    - uses: actions/setup-python@v2
      with:
        python-version: 3.8

Next we install Poetry, check the version (not required but sometimes useful to see in the logs), and then install the dependencies using Poetry.

    - name: Install Python Poetry
      uses: abatilo/actions-poetry@v2.1.0
      with:
        poetry-version: 1.1.2
    - name: Configure poetry
      shell: bash
      run: python -m poetry config virtualenvs.in-project true
    - name: View poetry version
      run: poetry --version
    - name: Install dependencies
      run: |
        python -m poetry install

Finally, run Flake8 linting and pytest unit tests on the Poetry virtualenv. Other linting or unit test frameworks could also be used here.

    - name: Lint with flake8
      run: |
        # stop the build if there are Python syntax errors or undefined names
        python -m poetry run flake8 . --exclude .venv --count --select=E9,F63,F7,F82 --show-source --statistics
        # exit-zero treats all errors as warnings. The GitHub editor is 127 chars wide
        python -m poetry run flake8 . --exclude .venv --count --exit-zero --max-complexity=10 --max-line-length=127 --statistics
    - name: Test with pytest
      run: |
        python -m poetry run python -m pytest -v tests

That's the quality step ready, simple 🤓.

Add a release step using Python Semantic Release

Now that our tests have run and passed the quality stage, it's time to release our new version.

The release job will only run after a successful quality stage. We also check the following:

This is a push to main. This prevents a release from being performed on PRs before they are merged.
The commit message doesn't contain chore(release), the message used by Semantic Release when a release is performed. Without this we would have an endless loop of releases, each release triggered by the previous release.

  Release:
    needs: Quality
    # https://github.community/t/how-do-i-specify-job-dependency-running-in-another-workflow/16482
    if: github.event_name == 'push' && github.ref == 'refs/heads/main' && !contains(github.event.head_commit.message, 'chore(release):')

Now install Python on the runner, checkout the code, then install and run python-semantic-release publish, which will:

Bump the version number in both version files.
Tag the code with that version.
Create a GitHub release object.
Commit the updated files to the main branch.

    runs-on: ubuntu-latest
    steps:
      - uses: actions/setup-python@v2
        with:
          python-version: 3.8
      - name: Checkout code
        uses: actions/checkout@v2
      - name: Semantic Release
        run: |
          pip install python-semantic-release
          git config user.name github-actions
          git config user.email github-actions@github.com
          semantic-release publish

Following this simple step, our releases are now created and the workflow is complete. It could even be simplified further by using the Python Semantic Release GitHub Action.

Final result

That's it!

We can now see all Actions workflow runs from the GitHub actions page. Looking at a specific run we can see the successful quality and release steps:

A release has been created, and assets can be downloaded for that specific version:

Of course some tweaks can always be made to improve the workflow, but now we have a nice lightweight CI workflow in place to support automated testing and release as I develop Dr. Sven further.

Possible Improvements

As GitLab say: everything is in draft. In keeping with that, of course there are things that I can and will improve in this CI workflow. Here are some of them:

Lock the main branch: currently I can commit directly to the main branch, which is bad practice. As I'm currently the only project contributor it's OK for now, but I need to lock the branch and configure Semantic Release to use a specific token with permission to push to that branch when updating the version numbers.
CommitLint within the CI: currently I've used the Semantic Commit bot which is great to allow you to see that your PR contains compliant commits from the PR page. However, technically, I could bypass this and commit something without compliant commit messages directly to main, skipping the release. This can be prevented by adding a CommitLint step within the pipeline (also locking the main branch will help).
Integration testing within the CI: currently only unit tests are executed. With this testing strategy, all integration testing must be performed manually on my dev machine or in AWS. Not very DevOpsy, only really acceptable in my simple, single developer environment.
Pre-commit hook to ensure Conventional Commits are used: prevent any commits from being made unless the developer is using the specified commit style. This prevents developer frustration by avoiding the annoying discovery at PR time that you didn't follow the project conventions.

Thanks for reading! Please leave a comment, I'm happy to answer any questions you may have.

Fin.

Guidebook: fostering open source in a large corporate environment (part 2 of 2)

MeStrak — Sun, 20 Dec 2020 13:03:08 +0000

In part one we looked at some of the simple steps that you can take to help your enterprise support open source projects. Now let's look at some options to give more by investing company resources.

Sponsoring an open source project

Depending on your company, this one may be even easier than contributing to code, support and documentation. Many open source projects ask for sponsorship to help the core team dedicate more time to working on the project. There are now several platforms available for you to easily sponsor the projects including GitHub Sponsors and Open Collective.

How does it work?

Typically, projects who are serious about sponsorship offer several levels, from basic to advanced with different advantages. Quite often these are called Gold, Silver and Bronze level memberships.

An example from one of my favourite projects, NestJS https://nestjs.com/:

Backers ($2 USD per month): Support us with a monthly donation and help us continue our activities.
Bronze ($100 USD per month): Become a bronze sponsor and get your logo on our README on GitHub as well as a small logo on the official page with a link to your site.
Silver ($250 USD per month): Become a silver sponsor and get your logo on our README on GitHub as well as on the official page with a link to your site.
Gold $500 USD per month: Become a gold sponsor and get your logo on our README on GitHub as well as on the official page and the official documentation with a link to your site.
Base ($1000 USD per month): Become a base sponsor and get your logo on our README on GitHub as well as on the official page and the official documentation with a link to your site (as a main project sponsor).

Some projects also offer priority support for higher level sponsorships, and some even have a specific corporate support model, for example Vuetify (another one of my favourites!) https://vuetifyjs.com/en/introduction/enterprise/.

Make your motivation clear

My personal motivation for sponsoring projects, is that I want things to be 'fair'. I believe that if large companies are making use of open source software, it shouldn't be completely free. They have the means, and therefore a social responsibility (as well as a mutual interest) to help that project continue to evolve.

Whenever I suggest sponsoring a new project, I'm very transparent about my personal motivation, but I'm also careful to put together a well-structured case about why it is the right business decision. Depending on whom you discuss this with, each of these points will be more or less important.

Choose your projects strategically

We touched on this in part one - you might not be in a position to offer sponsorship to many projects. Establishing a standard tech stack certainly helps with this by limiting the number of open source projects you use, allowing you to focus sponsorship on only a few. Here are some other things you might consider:

How hard will it be to switch to another technology if the project is no longer maintained?
How well-supported is the project by other sponsors? If it already has heavy investment then perhaps your sponsorship will count more elsewhere

Invest time to educate your finance/procurement teams about open source, and the sponsorship model

Your procurement team is likely to be highly trained in the art of negotiation. The reality is that this skill isn't usually required in the sponsorship model - you can decide what you pay and in most cases (unless there is a premium support model) it doesn't make any difference to the final cost. Often the sponsorship cost of the project is also relatively low compared to the type of licence agreement or service contract that negotiations are usually designed for. Make sure you spend time with your contacts in this team to explain the philosophy and make sure they are on-board with your idea.

Financials

Not everybody will have the same motivation, and this is a very important part to prepare properly. As I mentioned you will most likely need to discuss this with finance or procurement colleagues used to an entirely different licensing model. The idea of paying what you want may be an alien concept, and you need to show them why it is a good idea. Also, if this is the start of your sponsorship journey, it's likely that you'll want to sponsor additional projects in the future.

Do your calculations including the following:

Ongoing costs of continued sponsorship.
Estimate how much you would pay in licence fees and maintenance costs if you were purchasing a commercial product to do the same thing. Often there isn't actually an equivalent commercial product. In that case just get whatever costs you can for alternative options if you chose to buy instead of build your project.
List of benefits (improved support, project continuity, in some cases publicity, ...).
Compare sponsorship cost to developer daily rate, and estimate how much development cost time is saved by using the open source solution.
List of future expenditure as you sponsor additional projects.
Have everything projected as monthly and yearly costs.

Be patient

You may, from time to time, become frustrated with this process of trying to get financial approval for something which seems low cost. Don't let this de-motivate you. Remind yourself that what you're doing is for the good of the project community and your company, as well as being personally rewarding.

Make a big thing about the community support

Assuming you followed my advice about choosing well-supported projects with an active community for the most critical components of your tech stack, you should be very confident in advertising this fact when discussing with procurement teams. That active open source community is essentially replacing a paid maintenance and support plan. Again, as with licence fees be ready with estimated savings for maintenance and support agreements.

Logo or not?

Often procurement teams will have a clause built into the contract about whether a vendor can use your company logo on their website to show that you are a customer. The example sponsorship model I showed focuses heavily on showing your company logo in different places of the project documentation depending on how much you decide to sponsor them.

If your company logo is shown, it's a great feeling of pride, and can be a badge of honour for the developers using that technology.

If you're not sure what your company rules are about showing the logo, don't let it delay your sponsorship - just ask the project owners not to display your logo as you still have some things to check internally.

How to pay

This one may seem frustrating, but if you've made it this far you're nearly there. This is the final red tape to cut through!

GitHub sponsors and Open Collective are wired up to take payment via credit card or PayPal. Some large companies don't like paying in this way, accepting only payments by PO - in this case reach out to the project owner and ask them to create an invoice. If your company only allows you to purchase from a list of pre-approved vendors they probably have a partner that you can use who will handle payments like this for a small fee.

The cherry on the cake

Congratulations! You've successfully navigated the internal red tape, educated your colleagues and have just sponsored your first open source project. If you've made a significant contribution, why not reach out to the project sponsor and ask them to record a short video to say thanks to your team? Hopefully they are grateful for your support and will be happy to do this for you. Trust me - this is a great way to announce the sponsorship internally and will be a great motivation to your developers.

Launching an open source project

If you have an internal project which you think could be useful for others, then this option is for you. It generally requires the highest effort, but you can start with a small library if you think that you have something which is useful to share. You'll be publicly releasing something under your company name and depending on the company there are likely a few official stamps that you'll need before releasing it.

List the benefits

Start by making a bullet pointed list of the benefits of releasing your project. You may or may not present this as a slide to people involved in the approval. You should do it even if you won't present it, because it helps you to ensure you have a succinct explanation of what you aim to do and why. Here are some which come to mind:

Motivate the team by working together on a project visible to the outside world.
Attract talent when recruiting thanks to your visible open source activities.
Great publicity for your company.
Depending on your project making it open source may help to improve architecture because of the thought process required to ensure the design will be useful for the community (I had this experience with whispr: https://github.com/Sanofi-IADC/whispr/).

Licence review by legal team

Larger corporations most likely have a legal team who should be involved in the approval process for releasing a project. Check if your contact has experience with open source software. If not, spend some time explaining the concept to them and why you want to launch a project. Once you have educated them in the open source philosophy, have them review and approve the specific open source licence that you intend to use to release your project.

Communications/marketing team

You might also have an internal communications or marketing team. Again, spend some time with them to educate them about open source and explain your project. You might need their blessing to release a project stamped with your company name, and if all goes well you might be able to get their help with publicising the launch of your project either internally or externally.

Wrap-up

That's it! In the first part of this series I tried to give you some suggestions on how to get started with open source in your enterprise. In this second part we looked at how to go about sponsoring an open source project, and the importance of educating your non techy colleagues to help you do this. Finally we looked at launching your own project. Large or small I know that you and your team will find this incredibly satisfying.

I hope that this series has given you some ideas and inspiration for promoting open source within your enterprise. Thanks for reading!

Guidebook: fostering open source in a large corporate environment (part 1 of 2)

MeStrak — Sun, 11 Oct 2020 11:48:29 +0000

In this series I'll present a guide on how to help your company use open source software responsibly by fully investing in it and contributing to the open source community. I believe that all companies using open source for software development should be responsible for this.

I write this following my very positive personal and professional experience in launching a significant open source initiative within a large enterprise and I hope to inspire others to do the same.

You should read on if …

you work in a large company and want to educate non technical colleagues about open source
you want to promote the use of open source in your company
you want to help your company contribute to the open source community

We'll look at the following topics. In each one I'll suggest how you can deal with some challenges you may experience in a larger corporate environment which is not familiar with the open source philosophy.

Part 1:

Why use open source?: a quick overview of the benefits of using open source. Useful for your elevator pitch when you need to explain it internally.
Open source licensing: licensing basics you should be able to explain clearly to your colleagues.
Choosing an open source tech stack: some simple things to consider when choosing your stack.
Contributing to the community: a look at the easiest way to contribute to open source projects. This is the easiest and fastest way to provide support without worrying about internal approval.

Part 2:

Sponsoring open source projects: sponsoring a project financially to help main contributors dedicate more time to the project, or to help with general project expenses.
Launching an open source project: creating your own open source project by releasing something that has been developed internally under an open source licence.

Introduction to Open Source

Why use open source?

With a quick Google search you'll find a plethora of articles professing the goodness of open source. Here are a few of the benefits that appeal the most to me, and which speak clearly within a corporate environment.

Better together

Thanks to the open sourcing of some amazing projects such as Express.js, Vue.js and Go to name only a few, all developers have free access to some amazing tools. Instead of writing functionality from scratch they can accelerate development by building on something great which is already open source.

Working this way results in better tools available for all as they are improved together by a community. It also promotes inclusivity as the cost barrier is removed for accessing these tools and frameworks.

Community security

All the source code is available to anyone who wants to look at it, including security flaws in the code and vulnerable dependencies. While at first sight this may seem like a security risk, it is actually very beneficial:

Code is visible to everyone, meaning it can be reviewed and issues fixed by many people.
Thanks to these community peer reviews, open source projects must fix their vulnerabilities. There can be no hidden or ignored vulnerabilities, something which is possible in closed source software.

Empowered developers

You'll see this as a common theme throughout this article. Fully embracing open source and encouraging your developers to contribute to the project will be a great motivation for your development team. They have the chance to work on not only your project, but also to make a difference to something bigger. This applies to developers on external contracts too.

Open source licences

You need to know about this because if you want to promote open source development within your organisation, you may need to educate others about different licence types.

When choosing a licence for a new project it's really important to read the detail to know which one suits your project best.
When using open source code you must check the licence to make sure it's compatible with your intended use.

Many licencing models are available for open source projects, which impact how the code can be used and modified by others. Well known licences include MIT, Mozilla Public Licence, BSD and GNU GPLv3. Licences are generally categorised as 'permissive' or 'copyleft', depending on how the source code can be used and whether those modifying the code for their own use must make the changes visible:

Permissive: allows a great deal of freedom with no obligation to contribute code back to the community, or to share modified code. The most well known highly permissive licence is probably the MIT licence.
Copyleft: obligates people improving or adapting the code in an open source project to share their improvements with the community so they can be integrated back into the project.

As I will explain, reviewing the licence is also an important part of choosing your tech stack. There are also tools available to scan your code within to check open source licence compliance. This can be very useful to demonstrate to internal security and legal teams that you're compliant.

Choosing a licence is also a very important part of creating an open source project, because depending on the success of the project it can make a big difference to how people use and contribute to it. In cases where somebody intends to modify your open source code to make a proprietary product, the licence could be a deciding factor in whether to use your code.

More about the different licence types here: https://choosealicense.com/.

Choosing an open source tech stack

When choosing a tech stack, developers and architects have a huge number of open source projects to choose from. Here are a few tips to help you do this in a controlled way.

Standardise your tech stack to focus sponsorship

It's not uncommon for major parts of a tech stack to be based on developer preference. With multiple product teams working in parallel this can lead to many similar but different frameworks being used in your landscape. Assuming you intend to sponsor key parts of your stack, you should aim to standardise those similar parts. For example where you use a Node.JS backend, aim to use the same framework across products, the same goes for your UI components. Doing this will allow you to focus your sponsorship on fewer technologies and give a meaningful contribution to these projects. It also facilitates cross-collaboration between teams as developers will be familiar with the stack used on multiple products.

Mitigate the risk for critical components

You must be careful when choosing technologies for critical components in your stack. You can either make sure you choose a well-supported open source project ensuring continued development and maintenance from the community, or acknowledge and accept the risk if a project is less active but provides a unique feature that you would like to take advantage of.

Some of the metrics that you can use to measure an open source project's success:

Issue turnover: too many open issues can signify a project which is not well maintained, but if no issues are opened at all this may be an indicator that very few people are using the project
Number of weekly downloads: package repositories such as NPM give a simple insight into the frequency that open source packages are downloaded
Contributions to master: Similar to issue turnover, check how frequently commits are made to master (by people, not by bots) - this is a good indication of whether a project is actively developed or not

Check the licence is compatible with your intended use

If you plan to make modifications to the code in parts of your tech stack which you would be uncomfortable sharing with the rest of the world, you should be looking for permissive licenses such as MIT or BSD. If you make changes to the core code on a GNU licensed project for example, you must open source those changes under the same licence. In most cases you'll probably find that there's no issue, but check anyway!

Contributing back to the community

If you got this far, you're probably convinced about the value of open source, both the technological and philosophical aspects. Now that you have your open source tech stack in place, let's look at the easiest way for you to contribute back to the open source world.

Many open source projects have only a few core contributors. These contributors keep the project alive. They do all major development, documentation and maintenance. The simplest and easiest way that you can contribute to open source is to help some of the projects in your tech stack with these activities.

Documentation updates - just do it!

It's easy to get hung up on corporate approval for things like this but in reality this isn't always necessary. If someone from your team finds a mistake in some documentation, it's not necessary to advertise that the pull request you're raising is on behalf of your company. Most likely the developer already has a GitHub account and there's no reason not to just raise a PR for the updated documentation. You'll need the documentation internally anyway, so the only overhead is a few minutes of preparing a well formatted pull request according to the project guidelines. Have a developer update the documentation, and it's done. The dev will feel empowered, you'll feel great having made the first small step on your corporate open source journey, and you've helped out the project core team who have developed this amazing piece of technology for you, for free!

Helping out with support requests

Many open source projects run support either through GitHub issues or a free chat tool like Discord. Your developers are working with the project every day and undoubtedly have valuable knowledge they can share with others. Every time your team ask for help, or file a bug report they're essentially taking advantage of a free maintenance agreement where a member of the community (frequently a core team member) will help you out.

This is another very simple one - ask your dev team to browse the issues channel of one of the projects you're using a couple of times a week. Let them know they are free to answer questions raised by other project users. You're not trying to create a fully fledged support team, you're contributing to a community. Imagine what would happen if everybody did the same.

Bug fixes and enhancements

This is the next step and does require a little more effort from your development team. You'll find most open source communities very welcoming of anyone submitting useful PRs for their project. Your developers will feel great having contributed a fix, and will improve their knowledge of the technology by developing on its codebase. My recommendation to start on this one is when an issue is identified by your team:

Ask your developers to review it together and see if they have an idea of how it can be resolved
You can then raise an issue through GitHub (or whatever the official bugs channel of the project is), and suggest a fix for that issue, making it clear you would like to work on it.
Assuming the proposed approached is accepted, then go ahead and fix it!

Wrap-up

In this first article we've looked at some arguments for using open source within large enterprises, considerations when choosing an an open source tech stack and how to take your first steps into contributing to an open source project. I've tried to make as many suggestions as I can to keep it simple and avoid corporate red tape. Make sure you follow your company's internal rules, but don't be afraid to challenge them if you need to - sometimes it's the only way to make progress.

In part two we'll look at how you can use this information to educate colleagues within your company about open source to help you sponsor some open source projects, and potentially release your own company open source project.

Introducing Dr. Sven: open source data health checker

MeStrak — Wed, 19 Aug 2020 13:51:10 +0000

I just created a new project on GitHub that I want to share with you all: https://github.com/MeStrak/dr-sven.

What follows is a brief description of Dr. Sven adapted from the project readme, and a demo of what it can do.

What is Dr. Sven?

Dr Sven is a very simple data health checker which performs a checkup on your data to give an indication of whether it's in good shape or not.

The doctor comes in the form of an AWS Lambda function written in Python. It scans a table in an AWS Athena database based on a set of simple data rules defined by you to check that the data is in good shape. When I say simple I mean it - right now the only rule that exists is a check that every day within a given time range contains at least the specified number of records. Specific dates or days can be ignored to avoid spam.

Dr. Sven practises KISS medicine

No the doc doesn't break the hypocratic oath, it just prescribes to the ideology that simple is better.

Motivation for creating Dr. Sven

Dr. Sven is designed with a data lake type situation in mind where data is transferred from one data source to another repository. Most data ingestion monitoring tools I've seen focus on whether individual processes have run without error at the expected time. In some complex scenarios they also try to check that elements of the source data match the finally processed data which can be very difficult to do.

Imagine this situation ...

all dashboards are green 😄
logs contain no errors 😁
all appears well with the world 🚀

... then a pesky human looks at the data and says

"but there are no records from Sunday, that's not right" 😰

"there's something weird - it looks like there's a hole in this data from 1 month ago" 😱.

This project aims to reduce dependence on that human by providing a naive yet effective way to detect data issues. Specify some data rules defining what you expect to see, then let our kind doctor check if the processed data meets the criteria.

Demo!

Let's take a look at a simple example of using Dr. Sven using the AWS Athena getting started dataset. We'll create a checkup for the following:

Check every weekend contains at least a few records
Check that every week day contains lots of records
Specify a few dates to ignore for each of those rules, because we already know that there are some exceptions to the rule

Prerequisites if you want to play along

Installation instructions are in the project readme, and it's better to leave them there for now as I improve them. Here I'll show what Dr. Sven can do once installed.
We'll be using the dataset in the AWS Athena getting started guide: https://docs.aws.amazon.com/athena/latest/ug/getting-started.html.

Configuring Dr. Sven

Each dataset you want to check must have a specific configuration file. A configuration file can contain multiple rules as long as they are for the same dataset.

First let's look at the configuration file (this example is contained in the project on GitHub), then I'll explain the parts which aren't immediately obvious.

Datasource.query

This is the basis of the dataset which will be inspected
- It must return data with 2 columns: date, and count, where count is the number of records on that day
- The query text must contain '{start_date}' and '{end_date}' which will be replaced by Dr. Sven based on the date range you specify.
Date must be in the formay YYYY-mm-dd

Rules

This is the list of the rules which will be executed on the data
Currently only [[rules.min_records]] is implemented, which checks that the data contains the required number of records
Ignore dates is a list of dates which should be excluded from the rule, usually because there is a good reason why the data doesn't on that day meet the requirements of the rule
Ignore days is a list of days, using the name of the day which should be excluded. For example if every Sunday there is system maintenance so there will be no records, add 'Sunday' to this list. You can also use the friendly names 'Weekends' instead of ['Saturday', 'Sunday'] and 'Weekdays' instead of ['Monday', 'Tuesday', ..., 'Friday']
In this configuration we exclude 'Weekdays' to define a rule only for the weekend, and 'Weekends' to define a rule only applicable Monday - Friday

[general]
title = "Example config for Dr. Sven using AWS Athena getting started dataset"
output_location = "lovely-bucket"
output_region = "eu-west-1"

[datasource]
query = "SELECT date, COUNT(*) as count FROM cloudfront_logs WHERE date BETWEEN date '{start_date}' AND date '{end_date}' GROUP BY date"
database = "mydatabase"
start_date = 2014-07-01
end_date = 2014-08-08

[rules]
  [[rules.min_records]]
  name = "Weekends must have a few records"
  ignore_dates = [2014-07-12, 2014-07-13]
  ignore_days = ["Weekdays"]
  min_records = 10
  explanation = "The logs should always have some activity at weekends, just not as much as in the week"

  [[rules.min_records]]
  name = "Weekdays must have many records"
  ignore_dates = [2014-07-06, 2014-07-07, 2014-07-06, 2014-07-25]
  ignore_days = ["Weekends"]
  min_records = 100
  explanation = "There will always be a lot of log activity during the week unless there is some downtime"

Running Dr. Sven

Now we launch our dr-sven lambda with this cloudwatch event (replacing bucket name with the bucket you put the config file in specified).

 {
  "s3": {
    "bucket": {
      "name": "my-bucket"
    },
    "object": {
      "key": "aws-athena-example.toml"
    }
  }
}

Checking the results

Dr. Sven will output two files with the timestamp, checkup ID and the

Summary: .md file containing a summary of the results for each rule (number of rows checked, passed, failed and ignored)
Results: .csv file with a line per date for each failed rule

The dataset that we tested actually only contains data for two dates. We configured Dr. Sven to check data over from 2014-07-01 to 2014-08-08, so we're expecting a lot of failures.

Summary.md

Contains a summary of all rules that were checked, importantly telling you how many rules failed which you can then check in results.csv.

Weekends must have a few records

The logs should always have some activity at weekends, just not as much as in the week
Total dates checked: 39
Ignored: 31
Passed: 1
Failed: 7

Weekdays must have many records

There will always be a lot of log activity during the week unless there is some downtime
Total dates checked: 39
Ignored: 12
Passed: 1
Failed: 26

Results.csv

The results .csv output lists every issue found, along with the name of the rule taken from the config file. I've truncated the output below because it was too long - the real output found 32 issues, and correctly excluded ignored dates and days.

	count	day_of_week	symptom	failed_rule
06/07/2014	0	Sunday	Expected at least 10 records but found 0	Weekends must have a few records
19/07/2014	0	Saturday	Expected at least 10 records but found 0	Weekends must have a few records
20/07/2014	0	Sunday	Expected at least 10 records but found 0	Weekends must have a few records
(…)
04/08/2014	0	Monday	Expected at least 100 records but found 0	Weekdays must have many records
06/08/2014	0	Wednesday	Expected at least 100 records but found 0	Weekdays must have many records
07/08/2014	0	Thursday	Expected at least 100 records but found 0	Weekdays must have many records
08/08/2014	0	Friday	Expected at least 100 records but found 0	Weekdays must have many records

Wrap-up

So there you have it. Like I said Dr. Sven is a really simple way to check that your data obeys some basic rules. I don't know if it's useful to others but I'm very eager to hear your feedback!

Ongoing notes on things I did to get WSL2 to work

MeStrak — Tue, 14 Jul 2020 11:55:45 +0000

In the interests of productivity, and many issues trying to get dev stuff to work on my Windows workstation I started to use WSL 2 (I skipped WSL 1).

This post will be used for ongoing notes of things that I had to fix during the setup. It might not be complete as I'm starting my notes after the fact but I'll do my best to remember everything.

All the fixes are actually simple, but hopefully it will still be useful to save some time for people facing the same issues.

NPM/Yarn install not working

Unfortunately I don't have a log of the exact error messages from this time, but when trying to install these package managers I got an error message (something related to not being able to connect - I couldn't even ping anything external).

The fix 🚀

DNS settings had not been configured, even pinging external sites didn't work.

Create a file: /etc/wsl.conf.
Put the following lines in the file in order to ensure the your DNS changes do not get blown away
[network] generateResolvConf = false
In a cmd window, run wsl --shutdown
Restart WSL2
Create a file: /etc/resolv.conf. If it exists, delete and replace existing one with this new file (sudo rm /etc/resolv.conf to delete, and if you get a warning about an existing swap file then D to delete it).
Put the following line in the file
nameserver 8.8.8.8 (or use your DNS server instead of 8.8.8.8 which is a Google DNS server)
In a cmd window, run wsl --shutdown
Restart WSL2

Credit: this fix is copied directly from this gist by @coltenkrauter

error: linker `cc` not found when compiling rust code

When running rustc I came across the following error, basically because gcc wasn't installed.

$ rustc fart.rs
error: linker `cc` not found
  |
  = note: No such file or directory (os error 2)

error: aborting due to previous error

The fix 🚀

$ sudo apt update
$ sudo apt install build-essential

I did actually have more errors than this which I've written up separately, see the 404 not found error below

Package installation fails with 404 errors

Annoying errors like the one below to show when I was installing some new packages (actually after trying to install build-essential).

$ sudo apt install build-essential
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following additional packages will be installed:
  cpp cpp-9 dpkg-dev fakeroot g++ g++-9 gcc gcc-9 gcc-9-base libalgorithm-diff-perl libalgorithm-diff-xs-perl
  libalgorithm-merge-perl libasan5 libatomic1 libc-dev-bin libc6-dev libcc1-0 libcrypt-dev libdpkg-perl libfakeroot
  libfile-fcntllock-perl libgcc-9-dev libisl22 libitm1 liblsan0 libmpc3 libquadmath0 libstdc++-9-dev libtsan0
  libubsan1 linux-libc-dev make
Suggested packages:
  cpp-doc gcc-9-locales debian-keyring g++-multilib g++-9-multilib gcc-9-doc gcc-multilib autoconf automake libtool
  flex bison gdb gcc-doc gcc-9-multilib glibc-doc bzr libstdc++-9-doc make-doc
The following NEW packages will be installed:
  build-essential cpp cpp-9 dpkg-dev fakeroot g++ g++-9 gcc gcc-9 gcc-9-base libalgorithm-diff-perl
  libalgorithm-diff-xs-perl libalgorithm-merge-perl libasan5 libatomic1 libc-dev-bin libc6-dev libcc1-0 libcrypt-dev
  libdpkg-perl libfakeroot libfile-fcntllock-perl libgcc-9-dev libisl22 libitm1 liblsan0 libmpc3 libquadmath0
  libstdc++-9-dev libtsan0 libubsan1 linux-libc-dev make
0 upgraded, 33 newly installed, 0 to remove and 33 not upgraded.
Need to get 1096 kB/35.2 MB of archives.
After this operation, 157 MB of additional disk space will be used.
Do you want to continue? [Y/n] y
Ign:1 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 linux-libc-dev amd64 5.4.0-31.35
Err:1 http://security.ubuntu.com/ubuntu focal-updates/main amd64 linux-libc-dev amd64 5.4.0-31.35
  404  Not Found [IP: 91.189.88.142 80]
E: Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/l/linux/linux-libc-dev_5.4.0-31.35_amd64.deb  404  Not Found [IP: 91.189.88.142 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

The important bit:

Failed to fetch http://security.ubuntu.com/ubuntu/pool/main/l/linux/linux-libc-dev_5.4.0-31.35_amd64.deb  404  Not Found [IP: 91.189.88.142 80]

Running with --fix-missing as suggested in the error message and a few blog posts gave the same error.

sudo apt-get update didn't work either:

$ sudo apt-get update --fix-missing && apt-get install gcc
Hit:1 https://dl.yarnpkg.com/debian stable InRelease
Hit:2 http://archive.ubuntu.com/ubuntu focal InRelease
Hit:3 https://deb.nodesource.com/node_12.x focal InRelease
Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [107 kB]
Get:5 http://archive.ubuntu.com/ubuntu focal-updates InRelease [111 kB]
Get:6 http://archive.ubuntu.com/ubuntu focal-backports InRelease [98.3 kB]
Reading package lists... Done
E: Release file for http://security.ubuntu.com/ubuntu/dists/focal-security/InRelease is not valid yet (invalid for another 10d 6h 3min 42s). Updates for this repository will not be applied.
E: Release file for http://archive.ubuntu.com/ubuntu/dists/focal-updates/InRelease is not valid yet (invalid for another 10d 6h 3min 50s). Updates for this repository will not be applied.
E: Release file for http://archive.ubuntu.com/ubuntu/dists/focal-backports/InRelease is not valid yet (invalid for another 10d 6h 4min 14s). Updates for this repository will not be applied.

The fix 🚀

The issue was that my clock was out of sync with the Windows host (I don't know why).

$ sudo hwclock --hctosys

After this $ sudo apt update ran happily, as did $ sudo apt install build-essential.

Credit: I found this solution here

Can't add python poetry auto completions for bash

Got the error bash: /etc/bash_completion.d/poetry.bash-completion: Permission denied when running the following command:

$ poetry completions bash > /etc/bash_completion.d/poetry.bash-completion

At the time of writing (10 Aug 2020), according to the output of $ poetry help completions this should be the command to install bash completions for poetry.

The fix 🚀

Run this command instead!

$ poetry completions bash | sudo tee /etc/bash_completion.d/poetry.bash-completion

Credit: I found this solution here

DEV Community: MeStrak

Pop culture design patterns: creational

Singleton: Highlander - there can be only one

Builder: Five Guys

Prototype: Futurama S6E17 - Benderama

Factory Method: The Pepsi Challenge

Abstract Factory: IKEA

Testing secure APIs by mocking JWT and JWKS

The Catkin user authentication process

Quick definitions

For the artists among you

Testing approach

Implementation

Mocking the JWT and JWKS

Cleaning up the code

Wrap up

Semantic release with Python, Poetry & GitHub Actions 🚀

What I want to achieve

TL;DR

Background

Conventional Commits

Semantic Versioning

Semantic Release

Implementation

Ensure Conventional Commits are used

Configure Semantic Release in pyproject.toml

Configure a basic quality CI step using GitHub Actions

Add a release step using Python Semantic Release

Final result

Possible Improvements

Guidebook: fostering open source in a large corporate environment (part 2 of 2)

Sponsoring an open source project

How does it work?

Make your motivation clear

Choose your projects strategically

Invest time to educate your finance/procurement teams about open source, and the sponsorship model

Financials

Be patient

Make a big thing about the community support

Logo or not?

How to pay

The cherry on the cake

Launching an open source project

List the benefits

Licence review by legal team

Communications/marketing team

Wrap-up

Guidebook: fostering open source in a large corporate environment (part 1 of 2)

Contents

Introduction to Open Source

Why use open source?

Better together

Community security

Empowered developers

Open source licences

Choosing an open source tech stack

Standardise your tech stack to focus sponsorship

Mitigate the risk for critical components

Check the licence is compatible with your intended use

Contributing back to the community

Documentation updates - just do it!

Helping out with support requests

Bug fixes and enhancements

Wrap-up

Introducing Dr. Sven: open source data health checker

What is Dr. Sven?

Motivation for creating Dr. Sven

Demo!

Prerequisites if you want to play along

Configuring Dr. Sven

Datasource.query

Rules

Running Dr. Sven

Checking the results

Summary.md

Weekends must have a few records

Weekdays must have many records

Results.csv

Wrap-up

Ongoing notes on things I did to get WSL2 to work

error: linker `cc` not found when compiling rust code