DEV Community: Andreas Frömer

Using monorepo-builder outside of github and gitlab

Andreas Frömer — Mon, 16 May 2022 08:07:19 +0000

At work we are managing multiple repositories. Also repositories where the context is exactly the same. For example: We are developing an application providing multiple endpoint to our clients. We also developing an api-sdk for our clients to communicate with our apis.

Right now, this is all done in two separate repositories.
Which is absolutely fine. But I kind of don't like it anymore.

The problem with separate repositories for a single context

What happens if you change an API and you have to adjust the contract for it?

clone API-SDK repository
clone API
make changes to your API
make changes to your API-SDK
require dev-branch (or local link) from API-SDK into API
write tests for your API with the changed contract
tag API-SDK with new version
release API changes

As you can see, these are a lot of steps you have to do while working with the API.

So, what can we do about it?

Monorepo to the rescue

If you don't know what a monorepo is, or why you should care about. Check out the blog post from @tomasvotruba All You Always Wanted to Know About Monorepo But Were Afraid To Ask

Adding the SDK into the repository of the API makes thing easier than ever. You can now work directly on the SDK in the latest version, write your tests immediately from inside your API.

If everything works, just tag it and publish your package.

Easy right? Right?

How to publish packages from your monorepo

When it comes to the part of actually publishing packages from inside your monorepo you have to get creative with a lot of git commands, or other existing tools like symplify/monorepo-builder from @tomasvotruba

In the latest version, he added the support for github and gitlab, so you can just include this as a workflow action and you are done for. This is awesome!

There is a but! Using the github action in my company was not a real option since we are using an on-site bitbucket server. Damn it!

Gone are the dreams of easy monorepo splitting :(

Docker to the rescue

If you don't know, Github actions is working with docker in the background. So, actually I could just use the docker image and run it myself, right?

Yes I Can!

But what about the direct reference to Github and Gitlab for the splitter?

Well, lucky for us, we can just set some environment variables from bitbucket and everything works as expected :)

The only "hack" we need to do, is to set the GITLAB_CI=bitbucket environment so that we don't need to deal with the prefix INPUT_ from Github.

Running the docker image

So how can I run the splitter using docker?
Well, here is a little bash script to do it.

docker run \
  -e GITLAB_CI=bitbucket \
  -e PAT="user:accesstoken" \
  -e PACKAGE_DIRECTORY=packages/test-package \
  -e REPOSITORY_ORGANIZATION=org \
  -e REPOSITORY_NAME=test-package \
  -e BRANCH=master \
  -e REPOSITORY_HOST=bitbucket.host.de/scm \
  -e CI_COMMIT_SHA=$COMMIT_SHA \
  -e USER_NAME="User Name" \
  -e USER_EMAIL=user.name@email.de \
  -e TAG=$TAG \
  --workdir=/bitbucket/workspace \
  -v $(pwd):"/bitbucket/workspace" \
  --rm \
  symplify2/monorepo-split:latest

Gitlab allows the login using only accesstoken. But we can just pass the username:accesstoken as PAT for the splitter.

Also the REPOSITORY_HOST is not only the host, but contains also a part of the basepath to the repositories.

One important change is the --workdir and -v option. The workdir has to be set, as we need our sources inside the running container and the can't override the sources from the splitter itself. So we need another folder where we link our current working directory into it.

And thats it. It works perfectly :)
Thanks @tomasvotruba for this amazing tool <3

Delay between retries for Supervisor

Andreas Frömer — Thu, 04 Nov 2021 13:36:13 +0000

Having process managing tools on current systems is a must have. The most common used, or at least the tools I have been using, are systemd and supervisor.

While both do basically the same, supervisor is lacking one crucial ability: The ability to have a delay between process restarts

While this is not always necessary, in some cases it will prevent your process to get into FATAL state which stops it altogether.

But why isn't there such an option?

The solution already exists

The problems is not that it is impossible to add such a feature, but rather that the maintainers need to merge and maintain it.

There already exists a feature request from 2014 and the solving pull request from 2015 which, to this time of writing, haven't been merged.

I have to make clear: I am not blaming the maintainers for not merging it. It is simply a matter of time from the maintainers.

How can you still add a delay

To still add a delay you need to get somewhat creative.
While searching for a solution I found the feature request for supervisor (see above), this mentions a solution to use a sleep X after your command for your supervisor process.

[program:www]
command=bash -c "<path to your script>; sleep X"

While this might work, the problem is that supervisor can not gracefully stop this process using a SIGTERM signal. The SIGTERM will only hit your bash command, but not your actual script.

So what can you do?
While reading further into the feature request if found a pull request to symfony/messenger component, here the author added a script which actually is capable forwarding the SIGTERM signal into the child process.

You can find the PR here: https://github.com/symfony/symfony-docs/pull/13597

Extracted from this PR, the scripts looks as follows:

#!/bin/bash

# Supervisor sends TERM to services when stopped.
# This wrapper has to pass the signal to it's child.
# Note that we send TERM (graceful) instead of KILL (immediate).
_term() {
    kill -TERM "$child" 2>/dev/null
    exit 1
}

trap _term SIGTERM

# Execute console.php with whatever arguments were specified to this script
"$@" &
child=$!
wait "$child"
rc=$?

# Delay to prevent supervisor from restarting too fast on failure
sleep 30

# Return with the exit code of the wrapped process
exit $rc

So the "only" thing you need to change is to prepend this wrapper in front to you previous command:

[program:www]
/path/to/wrapper <path to your script>

This way every SIGTERM will passed into your script and will exit as soon as the child is terminated. If you child process will die with some exit code larger 0, it will sleep for 30 seconds before exiting. After that period, supervisor will restart it again.

Credits goes to the authors of both pull requests as they have done the work already. I am just a messenger :)

Making a string shorter without losing its meaning

Andreas Frömer — Fri, 10 Jul 2020 14:12:47 +0000

Today I was confronted with a small but meaningful problem writing some piece of software. I reached some kind of threshold where the string does not fit, so I would have to shorten it.

I could remove all characters of each word keeping only the first three
Remove all vowels

But how do I make sure to keep the meaning of the string so you could still understand what's written 🤔

Small overview

To give you a small outline of what I was doing - I am importing huge amounts of data of unknown structure. So I am doing a scan over all data sets trying to figure out the structure, flatten it, and use it as MySQL column names.

Here is an example structure:

{
   "payload":{
      "product":{
         "details":{
            "aru":{
               "children":[
                  {
                     "aru":{
                        "products":[
                           {
                              "aru":{
                                 "street":"",
                                 "streetnumber":"",
                                 "productKey":"",
                                 "price":"",
                                 "tariffName":"",
                                 "insurancePolicyNumber":""
                              }
                           }
                        ]
                     }
                  }
               ]
            }
         }
      }
   }
}

You might already see an upcoming problem here 😉
And don't ask about the structure - it is what it is.

The limitations

Using innodb the limitation for column names is 64 characters. At first glance, and with multiple scans, this was going to be close to maximum but still worked.

Hence comes the problem

Of course, the limitations were reached while going into production as there was new data coming which exceeded it.

I am now presented with a string of 69 characters of length. Knowing that there will be a string coming exceeding it even further there needed to be some kind of reduction in characters.

Possible solutions

Given the following strings here are some solutions I have considered.

payloadProductDetailsAruChildren3AruProducts0AruStreet
payloadProductDetailsAruChildren3AruProducts0AruStreetnumber
payloadProductDetailsAruChildren3AruProducts0AruProductKey           
payloadProductDetailsAruChildren3AruProducts0AruPrice                
payloadProductDetailsAruChildren3AruProducts0AruTariffName           
payloadProductDetailsAruChildren3AruProducts0AruInsurancePolicyNumber

Only keep first 3 characters

Shortening the string by keeping the first 3 characters was the first and a very fast solution. Which ended in not even a solution but making it even worse:

payProDetAruChi3AruPro0AruStr
payProDetAruChi3AruPro0AruStr
payProDetAruChi3AruPro0AruProKey           
payProDetAruChi3AruPro0AruPri                
payProDetAruChi3AruPro0AruTarNam           
payProDetAruChi3AruPro0AruInsPolNum

You see the problem, we ended up with two equal column names which would be a disaster since we would be losing data.

Remove all vowels

A colleague of mine came up with the idea of removing all lower case vowels.

pyldPrdctDtlsArChldrn3ArPrdcts0ArStrt
pyldPrdctDtlsArChldrn3ArPrdcts0ArStrtnmbr
pyldPrdctDtlsArChldrn3ArPrdcts0ArPrdctKy           
pyldPrdctDtlsArChldrn3ArPrdcts0ArPrc                
pyldPrdctDtlsArChldrn3ArPrdcts0ArTrffNm           
pyldPrdctDtlsArChldrn3ArPrdcts0ArInsrncPlcyNmbr

Which looks better and we don't have a conflicting column name. But still, some column names seem hard to read. For example pyldPrdctDtlsArChldrn3ArPrdcts0ArPrc. Thinking of Price at the end is hard to nearly impossible.

Also, the product name Aru is shortened. That is not a big problem but still 😉

The Significance of Letter Position in Word Recognition

This is the title of a Ph.D. thesis written by Graham Rawlinson back in 1979.

In a short summary: Every word having the first and last character in place might miss or have shuffled characters in between and you might still understand the word.

With this in mind, I improved the "vowel solution" to:

Only remove lowercase vowels which are not followed by an uppercase character or beginning or end of the string.

The final solution

Using regex substitution I shortened the string so that it should be understandable for someone working in the same field.

(?<!^)[aeiou](?!([A-Z]|$))

The result is better understandable and still with no naming conflicts:

pyldPrdctDtlsAruChldrn3AruPrdcts0AruStrt
pyldPrdctDtlsAruChldrn3AruPrdcts0AruStrtnmbr
pyldPrdctDtlsAruChldrn3AruPrdcts0AruPrdctKy           
pyldPrdctDtlsAruChldrn3AruPrdcts0AruPrce                
pyldPrdctDtlsAruChldrn3AruPrdcts0AruTrffNme           
pyldPrdctDtlsAruChldrn3AruPrdcts0AruInsrncePlcyNmbr

There is still the possibility to end up with identical consonants after removing the vowels. But this is considered a lower probability then simply removing all vowels.

One word could be:

Color => Clr
Clear => Clr

If you liked the article leave me a comment/like. Do you have any other suggestions? Let me know!

You can follow me on twitter with @icanhazstring

Update: 2020-07-14
Changed the final solution regular expression to avoid removing lowercase vowels at the beginning of the string

Add a required argument to every symfony/console command

Andreas Frömer — Wed, 03 Jun 2020 13:44:40 +0000

I would assume that everyone has fiddled around with symfony/console right? If not, you should probably check it out here. It is pretty easy to use and understand.

One thing I had problems today is: How do you add a required argument for every command that is added to the console application?

I came up with 3 solutions:

Adding a BaseCommand class which every command added has to extend from
Add a required argument to the Console\Application
Add a required argument for each Console\Command added to the Console\Application

TLDR;

Like President Schwarzenegger used to say in the Simpson Movie: "I pick number three".
This way for me is most robust change as you don't need to alter any command and every new command that will be added, will receive the required argument.

Just as a side note: I used to pick the first approach :)

Adding a `BaseCommand` class which every command added has to `extend` from

This is pretty straight forward. You create a new BaseCommand class and add your required argument into the configure() method.

<?php

use Symfony\Component\Console\Command\Command;

class BaseCommand extends Command 
{
    public function configure(): void
    {
        $this->addArgument(
            'name',
            InputArgument::REQUIRED,
            'description'
       );
    }
}

But, what this would require is you have to call parent::configure() in every command so that the argument is there, if not. Well its not.

Add a required argument to the `Console\Application`

This was the second approach I found while searching for a solution.
Sadly, every solution I found was adding a new optional argument to the definition, which is not what I needed, you might need to "fiddle" around with the setArguments() of the InputDefinition of an application.

You can do this inside you bootstrap file, whery you load the application.

$app = new Application();
$appDefinition = $app->getDefinition();

$appDefininition->setArguments(
    array_merge([$requiredArgument], $appDefinition->getArguments())
);

But this is kind of out of scope where you want to define it.
So you can create your custom application class and do it there:

<?php

use Symfony\Component\Console\Application;
use Symfony\Component\Console\Input\InputDefinition;

class CustomApplication extends Application
{
    protected function getDefaultInputDefinition(): InputDefinition
    {
        $definition = parent::getDefaultInputDefinition();

        $definition->setArguments(
            array_merge([$requiredArgument], $definition->getArguments())
        );

        return $definition;
    }
}

Downside on this: You have a require argument for every command of your application even with the HelpCommand or ListCommand. So you can't run you application by simply running bin/console (if that is you entrypoint). You would need to pass the required argument everytime.

Add a required argument for each `Console\Command` added to the `Console\Application`

The solution I choose, was to add this argument for each command added to the application.

Some of you might think: Wait a second, what abount HelpCommand and ListCommand you mentioned before?

Yes you are right, every command, except commands from symfony/console namespace :)

So this is my solution: Creating my CustomApplication and overwriting the add(Command $command) adding my required argument to every added command to my application.

<?php

use Symfony\Component\Console\Application;
use Symfony\Component\Console\Command\Command;

class ConsoleApplication extends Application
{
    public function add(Command $command)
    {
        if (strpos(get_class($command), 'Symfony\Component\Console') === 0) {
            return parent::add($command);
        }

        $commandDefinition = $command->getDefinition();
        $commandDefinition->setArguments(
            array_merge([$requiredArgument], $commandDefinition->getArguments())
        );

        return parent::add($command);
    }
}

There you have it. Every command will get your required argument. Also this will be the first argument in every command. So even if you have an optional argument configured, there won't be a problem with that.

Hope that will help some of you solving the same problem I had :)
If you have another clever solution, let me know!

Local testing for GitHub Actions (on MacOS)

Andreas Frömer — Sat, 23 May 2020 07:31:03 +0000

The beginnings

Back in October 2018, when GitHub Actions were announced, things got pretty serious pretty fast. Everyone was jumping on the train and were replacing their old CI/CD integration with the new workflow GitHub now provided.

From my memory, the development of these workflows was a bit painful to say the least. You would have to add the changes to your workflow into your repository, push them and wait for the actions to complete.

Hopefully.

If not, do it again until everything worked out. The only thing I had done this far was a simple checkout, running some tests and that was pretty much it. Nothing fancy going one there.

Recently I was reminded by @localheinz in a tweet that some of the actions for GitHub made a huge improvement.

Later today

Now one and a half years later having an actively used repository wit composer-unused, I decided to check things again and improve the workflows. But - the testing. There has to be a way to test your workflow on your local machine. Right?

Yes there is! nektos/act to the rescue.

This tool, written in Go, is pretty easy to install and navigate. I will give you a small overview how to install and run it on MacOS (other OS are also supported, see the documentation)

Get the tool running

Requirements

nektos/act is using Docker to run your workflow on your machine. So first things first: setting up Docker using homebrew



$ brew install docker docker-machine

You will also need to install a provider for Docker. For this example we are using Virtualbox. You can install it a Homebrew Cask recipe:



$ brew cask install virtualbox
-> password

You might see the following message when installing Virtualbox:

You need to head over into the settings and accept the software installation from Oracle America Inc..

Next up, set up the default docker-machine with virtualbox driver and start it.



$ docker-machine create --driver virtualbox default
$ eval "$(docker-machine env default)"

Install nektos/act

Installation instruction taken from the docs using homebrew:



$ brew install nektos/tap/act

Run your workflow

To run your workflow on your machine, just navigate to your repository folder and run.



$ act

This will run your default job inside your workflow. Thats it!

But wait! Thats all in the docs, so why this post?
Well - I encountered a problem along the way:



::error::ENOENT: no such file or directory, open '/opt/hostedtoolcache/linux.sh'

Solving the issue

The problem above has nothing to do with nektos/act itself. But rather with the default Docker image it uses. To solve this, you can run act -P <docker-image> to use another default image.

Turns out shivammathur/setup-php was the solution.



$ act -P ubuntu-latest=shivammathur/node:latest

But typing this command all over again is - well - annoying.

To improve this, you can create a ~/.actrc file and put the command line arguments there so you only have to run act inside your repository.



// ~/.actrc
-P ubuntu-latest=shivammathur/node:latest

Now everything is working as expected and I can improve my workflow to my needs while testing it locally on my machine first!

Disclaimer

nektos/act is a little over a year old now, so there are still some problems with certain workflows and OS setups. I have only encountered the problem mentioned above.

If you encounter any other problem, don't hesitate and head over to the issue tracker of nektos/act and describe your problem. This way we all can improve this awesome tool even further!