DEV Community: Acter

Beware of the DashMap deadlock

Benjamin Kampmann — Fri, 29 Mar 2024 23:00:00 +0000

Rust is famously build for the multi-threaded-processor world. From its core ownership-enforcement-model up to the type-based Sync + Send-types, all is around allowing the compiler to ensure memory safety and consistency across thread boundaries. And though the std also has collections (like HashMap and BTreeSet), Atomics and Locks, once you start building real programs with Rust, probably some tokio for async-support as well, these are not always sufficient.

DashMap for the Win

No wonder that you can pick from a handful of libraries helping you achieve this feat, and quite a feat that is, with DashMap being among the most popular with a whopping 52million downloads on crates.io at the time of this writing:

52million downloads, years since publication, updated just a few months ago. That looks like a reasonably sound library. So, you start playing with it and find that its API is convenient, seems to work across awaits and async and all the things you've ever dreamed of. So you implement it as the main caching layer for the transient state machine of models within the core business logic of your application.

Deadlocks? Really?

I is only a long while later, until the first reports come in. Sparse at first and unclear in its origin, but sometimes, it seems, your state machine processing doesn't process the events coming in. Or, better - as you learn when digging into it - their futures never resolve.

You see, already known since at least December 2022 is that you can use DashMap in a way that can cause deadlocks and without the compiler detecting them.

A primer on dead locks

If you have no further knowledge about locks and have never heard of deadlocks, let me give you a minimal rough cut of the problem (overly simplified): Sometimes you have memory that is accessed by multiple threads, but clearly if both write at the same time this can cause problems. Thus, the concept of "locks" was created: small pieces around the memory that you need to have hold of first before you can write to that memory. While it is locked no other thread can write to it and thus have to wait for their turn. Ensuring they all write-one-at-a-time and not in between one another.

Now, how ever long you hold that lock is your prerogative and there are several problems with holding a lock very long. For example: what if your thread panics while you hold the lock? This in rust is usually referred to as a "poisoned lock", you might have seen that Error in the std, and how to deal with that depends on the specific code.

In this case, we are looking into a so called dead-lock situation. This can even be cause by a single thread easily: when you hold the lock and your code, running on the same thread, for whatever reason, tries to acquire the same lock while still holding the lock. This stops the execution as the thread is waiting on the lock it itself is holding and thus preventing from being released.

This type of scenario can be and in the std-cases is detected by the rust compiler (yay), but not in the case of DashMap. As DashMap actively allows for locks to be held over await-points (that is kinda its jam ... that it allows the user to do that), it isn't possible for the compiler to figure out that this might lead to a dead lock.

How to avoid that problem

The best advice is the one given from [Alice in her post from January 2022] already:

It is especially important to follow the advice about never locking it in async code when using dashmap for this reason.

While this is good advice, this isn't really mentioned on the docs of DashMap and considering there is nothing detectable wrong with the examples showing the problem when looking at the code this is quite the foot gun.

However, the code in question in our case doesn't even hold any locks over await-points, yet it seems to deadlock in some race condition scenarios.

Then you only find out about it after some long debugging and researching the github issues of that dependency. Taking all that into account, and then that there is no real way for you to create tests or otherwise automatically ensure it isn't reintroduced by any further update the code ... I consider this pretty harmful.

What to do about it?

Well, supposedly, this is fixed the in next big iteration of DashMap, which is said to have async support by getting rid of locks entirely, but with the issue open since 2021 and most of the ideas of how to avoid the locks being discounted for now, there is no telling when this come - if ever. What I have seen most people do referencing that issue, and what we also ended up doing is: replace or at least remove DashMap from the code base.

In our case we replaced it with the up and coming scc, which uses a different locking concept and has the additional benefit of being faster. Others have opted for cachemap2 or replaced it with the std lock & hashmap: there at least the compiler will tell you if you accidentally created a dead-lock-scenario.

No disrespect

I am not writing this post to shit on the authors of DashMap, nor its contributors or maintainers. Building a async-safe lock-free-ish collection is a hard task. One that I wouldn't even really want to attempt myself. I still personally don't even understand why this deadlocks internally myself, nor would I consider trying to patch it either - considering that they haven't done it yet makes leads me to believe this isn't an easy thing to do. As such I don't think anyone should be mad about them either, call them names or do any of the other nasty things the internet can do to people that lost its favor.

I am raising this issue because this is a pretty widely spread library, probably the most popular for the concurrent hashmaps and this is a severe problem that you should know about when using it. That's why I spent a significant amount of this post explaining the core problem and how to avoid it. So, if you use DashMap and want to continue using it, you know what to look out for now.

Edit 2024-04-01: Edited for clarity, based on corresponding feedback and removed a misleading quote.

Adding a new Ghost via docker-compose to your traefik setup

Benjamin Kampmann — Fri, 02 Feb 2024 19:31:30 +0000

Sometimes the easiest and quickest way to try (or even deploy) a new service is by using the recommended docker-compose-setup that they often have as an example. But if you have an existing infrastructure, like we do with the great mother of all self-hosting ansible playbooks, this isn't always easy to integrate. In particular when that infrastructure is managed and started and stopped independently from the additional docker-compose you intend to add. Lucky, who is running their out-most proxy using traefik, because with just a few extra labels your docker-compose becomes available TLS-certs included.

Fortunately for us the MASH-playbook uses traefik and so adding a Ghost setup for testing was quick and easy. Let's look at the docker-compose (for our fictional blog.example.org-address) and then we'll explain some of the specific aspects to address:

version: '3.1'

services:

  ghost:
    image: ghost:5-alpine
    restart: unless-stopped
    environment:
      # see https://ghost.org/docs/config/#configuration-options
      database__client: mysql
      database__connection__host: db
      database__connection__user: root
      database__connection__password: SOME_PRIVATE_ROOT_PASSWORD
      database__connection__database: ghost
      # this url value is just an example, and is likely wrong for your environment!
      url: https://blog.example.org
    volumes:
      - ./data/ghost:/var/lib/ghost/content
    networks:
      default:
      traefik:
        aliases:
         - blog-example-org

    labels:
      - traefik.enable=true
      - traefik.docker.network=traefik
      - traefik.http.routers.blog-example-org.rule=Host(`blog.example.org`)
      - traefik.http.services.blog-example-org.loadbalancer.server.port=2368
      - traefik.http.routers.blog-example-org.entrypoints=web-secure
      - traefik.http.routers.blog-example-org.tls=true
      - traefik.http.routers.blog-example-org.tls.certResolver=default
      - traefik.http.routers.blog-example-org.service=blog-example-org
  db:
    image: mysql:8.0
    restart: unless-stopped
    networks:
     - default
    environment:
      MYSQL_ROOT_PASSWORD: SOME_PRIVATE_ROOT_PASSWORD
      MYSQL_DATABASE: ghost
    volumes:
      - ./data/db:/var/lib/mysql


networks:
  traefik:
    external: true
  default:
    external: false

Alright, there's a few things here that have changes compared to the default example from ghost. We will be ignoring the specific Ghost and MySQL changes as they aren't that relevant but are only included for completeness.

The networks

First and foremost, we have the additional networks-section a the bottom of the configuration with two networks: default which we will use for this specific service and the other that is bridging to the traefik-service, which is marked as external: true telling docker to use the existing set up network. This must the the network the dockerized traefik is using. In the case of MASH this is just called traefik as well.

Secondly we need to the networks-section to both our services, where any internal service is only on the default network and the exposed service must also be on the traefik-network. Here we also give it some specific DNS name within that network for traefik to route the traffic to.

the `traefik` labels

When traefik is set up to use docker-labels, which is the case in our MASH setup, we can just label our service with a view fields and the traefik service will automatically recognize and configure the routing appropriately. Let's go through them one by one:

traefik.enable=true: to configure traefik to route this one. Depending on your setup this might not be needed
traefik.docker.network=traefik: the network traefik is on
traefik.http.routers.blog-example-org.rule=Host(\blog.example.org): the actual hostname we want this service to be available under between the final ticks. Note that we are creating a custom traefik-router for this called blog-example-org, all the following configuration is also using that router prefix:
traefik.http.services.blog-example-org.loadbalancer.server.port=2368: the port on this service the traffic should be routed to
traefik.http.routers.blog-example-org.entrypoints=web-secure: if the traefik has multiple outsid eendpoints, which ones to serve - in the MASH case we want this to be available at https, which is named web-secure in our setup.
traefik.http.routers.blog-example-org.tls=true: to enable TLS for this router
traefik.http.routers.blog-example-org.tls.certResolver=default: use the default DNS cert resolving functionality. In MASH this means we are using lets-encrypt certification
traefik.http.routers.blog-example-org.service=blog-example-org: the service to route the traffic to. The value is the dns-alias we gave in the network configuration before.

up and go

And that's about it. Assuming the DNS name already resolves to your server and your traefik is already running just doing a docker compose up -d and a short time later (if it needs to fetch the certificates for the first time), the service will be routed through and be available at blog.example.org. Neat!

Six niche tips for shipping Flutter MacOS builds

Benjamin Kampmann — Thu, 11 Jan 2024 11:00:00 +0000

Ever since we started shipping Acter to the Apple iOS AppStore, we wanted to have it on the Apple MacOS Store as well. With us building it on Rust and Flutter this should have been quite an easy feat as both have native support for MacOS. Yet actually shipping it was multiple months of try and error—with the last month spent on just a tiny problem caused by the Github Actions Runners. these are six niche tips we wished someone had told us before, that would have saved us months of work.

Nightly builds as the baseline

For testing and internal distribution we had nightly builds in Acter for a few months already. They would automatically be created every night at 3am (hence the name) if changes had been found on the main branch. Our build here consists of two parts: the internal Rust library and then we follow with a simple flutter build $target. So obviously, we have created a nice Github Actions Matrix to reuse as much as possible. I am not going into too much detail here and the latest action setup probably already changed when you read this, but I have linked the specific sections for record. the

1. MacOS is iOS but different—and Google won’t tell you

For the release build of iOS to work, we needed to sign the app. As a matter of fact, the flutter build won’t really work if you don’t have the necessary signatures set up. For iOS nightly we use an Ad-Hoc setup with a few pre-configured internal devices, for the release we used the distribution profiles, both stored as environment secrets as is so commonly used in many tutorials. For MacOS signatures aren’t necessary to build and distribute the App - presumably because MacOS App development pre-dates signed builds. Thus our nightly builds didn’t have any setup for that yet.

Another important difference to note is on the flutter side. While flutters build ipa offer the options --export-options-plist=PATH allowing you to specify certain plist information overrides for that specific build, no such option exists in flutter build macos. Meaning that all the configuration setup inside the macos-folder is and must be used as-is. That is a bit annoying as it means we can’t easily make a local release build without the signatures now but that’s what it is.

One annoying side-effect of Flutter being a lot more popular for building mobile apps is that when you try to Google for information regarding the apple setup needed you’ll almost exclusively find questions and problems for iOS. They then recommend stuff like the export-options-command or other obscure settings you are supposed to change via xcode but that doesn’t actually do anything in the desktop version or doesn’t even exist. Google really doesn’t help you when you get stuck with your Flutter MacOS build.

2. Switching from github environment secrets to git-crypt for signatures and profiles

One thing we wished we had done earlier was switching from storing signatures and provisioning profiles in Github secret environment variables to using git-crypt`. Many tutorials and setups out there recommend using the Github secrets to store, well, secret information like the provisioning profiles and the secrets from the keychain and then have some companion script that puts that into the local Github Action build. That is all good and dandy if you only do that setup once and rarely change it. But I always found it kinda annoying that despite no hint in the Git history a build might fail or pass. Once you go beyond just managing a single profile the scripts are then often falling apart and the increasing number of environment variables becomes very confusing and it is super easy to mess up in converting them into the right base64 because it was soo long ago you did it last.

Rather than storing profiles and the keystore and similar file-based secrets in the secret environment variables we switched to using git-crypt a git extension you can configure that transparently encrypts a subset of files before committing them to the repo. That makes it super easy and simple to update them and still keep the files available. Rather than extracting each secret from the environment into a file we just install git-crypt and have the main password as the action secret that we then use to decrypt the files:

- name: Unlock git-crypt
  if: matrix.with_apple_cert
  run: |
    brew install git-crypt
    echo "$" | base64 --decode > .github/assets/git-crypt-key
    git-crypt unlock .github/assets/git-crypt-key
    echo "Files found:"
    git-crypt status -e

Once we have the files decrypted, we use the commonly used script to import the keychain from the now decrypted file. Technically we wouldn’t even need the extra password for that file, but it also doesn’t hurt:

# Install the Apple certificate and provisioning profile
- name: Install the Apple certificates
  if: matrix.with_apple_cert
  env:
    P12_PASSWORD: ${{ secrets.BUILD_CERTS_P12_PASSWORD }}
    KEYCHAIN_PASSWORD: ${{ secrets.KEYCHAIN_PASSWORD }}
  run: |
    echo "starting in $RUNNER_TEMP"
    # create variables
    CERTIFICATE_PATH=".github/assets/build_certificates.p12"
    KEYCHAIN_PATH="$RUNNER_TEMP/app-signing.keychain-db"
    echo "vars set"
    # import certificate and provisioning profile from secrets
    # create temporary keychain
    echo "creating keychain"
    security create-keychain -p "$KEYCHAIN_PASSWORD" "$KEYCHAIN_PATH"
    echo "setting keychain"
    security set-keychain-settings -lut 21600 "$KEYCHAIN_PATH"
    echo "unlocking keychain"
    security unlock-keychain -p "$KEYCHAIN_PASSWORD" "$KEYCHAIN_PATH"
    # import certificate to keychain
    echo "importing certificate"
    security import "$CERTIFICATE_PATH" -P "$P12_PASSWORD" -A -t cert -f pkcs12 -k "$KEYCHAIN_PATH"
    echo "listing keychains"
    security list-keychain -d user -s "$KEYCHAIN_PATH"

And finally we just take all the now decrypted provisioning_profiles files and copy them where they need to be. All files for all builds in the git repo. Sweet.

# Install the Apple certificate and provisioning profile
- name: Install the Apple certificates
  if: matrix.with_apple_cert
  env:
    P12_PASSWORD: ${{ secrets.BUILD_CERTS_P12_PASSWORD }}
    KEYCHAIN_PASSWORD: ${{ secrets.KEYCHAIN_PASSWORD }}
  run: |
    echo "starting in $RUNNER_TEMP"
    # create variables
    CERTIFICATE_PATH=".github/assets/build_certificates.p12"
    KEYCHAIN_PATH="$RUNNER_TEMP/app-signing.keychain-db"
    echo "vars set"
    # import certificate and provisioning profile from secrets
    # create temporary keychain
    echo "creating keychain"
    security create-keychain -p "$KEYCHAIN_PASSWORD" "$KEYCHAIN_PATH"
    echo "setting keychain"
    security set-keychain-settings -lut 21600 "$KEYCHAIN_PATH"
    echo "unlocking keychain"
    security unlock-keychain -p "$KEYCHAIN_PASSWORD" "$KEYCHAIN_PATH"
    # import certificate to keychain
    echo "importing certificate"
    security import "$CERTIFICATE_PATH" -P "$P12_PASSWORD" -A -t cert -f pkcs12 -k "$KEYCHAIN_PATH"
    echo "listing keychains"
    security list-keychain -d user -s "$KEYCHAIN_PATH"

Finally don’t forget to clean all that up, regardless of whether the build failed or succeeded!

- name: Clean up keychain and provisioning profile
  if: ${{ always() }}
  run: |
    security delete-keychain $RUNNER_TEMP/app-signing.keychain-db
    rm ~/Library/MobileDevice/Provisioning\ Profiles/*
    rm .github/assets/git-crypt-key

This makes it super easy to update all that data. Got a new provisioning profile? Just put it into that folder. Update to the keystore? Just export the p12-file with the same password again. No base64 conversion, no copying into the Github Secrets - just git commit && push. *chefskiss*.

3. Github Search is the hidden champion

One of Githubs most underrated features is its search. It being the biggest crowd source code knowledge base in the world, including the largest source for all their own configuration files (which the workflow-yamls are one of) their search can be truly amazing. Not everyone who found the hack to make something happen will blog about it or write a stack overflow—this post almost didn’t make it either. But it if works there is a high chance they commit it and it ends up in the Github repo, discoverable via the search.

Similar as Google, Github’s search has many advanced options. For us looking for alternative ways of doing the Flutter build within the Actions, adding the path:.github/workflow flutter macos was the key to unlocking a treasure of knowledge. Mind you that even though code is committed doesn’t necessarily mean it runs, though. But it is how we first found out about the git-crypt idea! And that’s also how we found out about the final upload pattern we ended up using.

Seriously, if you are ever stuck on some Github Action configuration that others probably already attempted try the Github search. Google doesn’t even know about a fraction of it and with the advanced search you can make the queries very specific to your problem.

4. MacOS apps need all their binaries signed

One particularly nasty difference between the iOS and MacOS flutter build is that the latter doesn’t really manage the signing properly for you. Singing the Mac App is different than the iOS one, too: While on iOS you create an ipa-file (effectively a Zip-File) which is then signed as a whole (oversimplified), the “Mac App” is actually a directory with the extension .app. You can’t effectively “sign” directories. Instead the people at Apple decided that what you must do is sign each binary within that app directory and provide these signatures in the directory. This is hidden in the docs somewhere but if you tried to Google for this information, you will only find iOS fixes (see No 1). So I am telling you know.

For most cases that is fairly irrelevant but as we had a bunch of binaries, our own included. We found a script that iterates through the final app and signs each binary with the provided credentials, which we then added to the regular Xcode shell-script build process for release builds. That means that at the end of flutter build macos, we now have an Acter.app directory with all the proper signatures included. Yay.

5. Build code ages quickly

One problem you’ll be facing with the Github Search as well as the Google search answers regardless is that the infrastructure you are building with and against constantly changes. For us, there were several tutorials out there recommending ways of packaging or uploading the app that were outdated to simply not supported anymore for the latest version (this was even worse for building the Windows App). Trying to figure out which is the latest recommended and thus hopefully the longest-lasting code you could write is a tedious and annoying process. Very often you don’t know this isn’t supported anymore until you installed and tried the command. But there is a few tricks to keep in mind, when you find a novel approach you might want to try out: you can check the official docs and see if it is still supported, if it is on Github you can see when it was last run, for StackOverflow and many blogs you can quickly gather whether this is a new or rather old idea. Unfortunately in this space, old often means less likely too still work…

For us the latest—at the time of writing—and recommended way to package and submit the Flutter MacOS app to the Apple Mac Store is:

build the release version of the app with flutter build macos; make sure all binaries are signed (see above)
use productbuild to create a modern .pkg and have it signed: productbuild --component Acter.app /Applications --sign "$APPLE_SIGN_CERTNAME" Acter.pkg
then use altool to upload the .pkg to the Apple Mac AppStore using a private_key credential (which we stored with git-crypt, of course):

      - name: Upload to App Store
     env:
       APPLE_API_KEY_BASE64: ${{ secrets.APPLE_API_KEY_BASE64 }}
       APPLE_API_KEY_ID: ${{ secrets.APPLE_API_KEY_ID }}
       APPLE_ISSUER_ID: ${{ secrets.APPLE_ISSUER_ID }}
     run: |
       mkdir private_keys
       echo -n "$APPLE_API_KEY_BASE64" | base64 --decode --output "private_keys/AuthKey_$APPLE_API_KEY_ID.p8"
       ls -ltas private_keys
       xcrun altool --upload-app --type macos --file acter-macosx-${{ needs.tags.outputs.tag }}.pkg \
           --bundle-id global.acter.a3 \
           --apiKey "$APPLE_API_KEY_ID" \
           --apiIssuer "$APPLE_ISSUER_ID"
     shell: bash

6. Github artifacts are not a proper package mechanism: measure twice

With that we are all set and everything should work. Yet Apple kept rejecting our app. But only after the upload in the post-processing on the server side, a few hours later we’d receive an email saying something along the lines of:

      - name: Upload to App Store
     env:
       APPLE_API_KEY_BASE64: ${{ secrets.APPLE_API_KEY_BASE64 }}
       APPLE_API_KEY_ID: ${{ secrets.APPLE_API_KEY_ID }}
       APPLE_ISSUER_ID: ${{ secrets.APPLE_ISSUER_ID }}
     run: |
       mkdir private_keys
       echo -n "$APPLE_API_KEY_BASE64" | base64 --decode --output "private_keys/AuthKey_$APPLE_API_KEY_ID.p8"
       ls -ltas private_keys
       xcrun altool --upload-app --type macos --file acter-macosx-${{ needs.tags.outputs.tag }}.pkg \
           --bundle-id global.acter.a3 \
           --apiKey "$APPLE_API_KEY_ID" \
           --apiIssuer "$APPLE_ISSUER_ID"
     shell: bash

The attentive reader might have already noticed the actual problem already. As we had gotten several similar looking emails (especially the top part) from before when the signatures were not properly set up for each binary, we assumed it was something wrong with that part again. Weirdly enough, when doing the entire process manually (rather than via Github Action) it all worked fine and Apple didn’t reject our submission. Weird. So we downloaded the latest Acter.app from the build artifacts to try to see if we could sign and submit that. The download was larger than usual (220mb rather than the usual 140mb we saw for most builds before) but we didn’t really think much about it. Indeed, trying to package and upload this version Apple rejected it again.

So, we look into the insides of the Acter.app build by the Github Action: it is just a folder after all (even though MacOS finder hides it under the right click -> Open Contents). Right away we noticed something odd: all binaries for the frameworks appeared to be in there twice: once under $framework/Versions/Current and once as $framework/Versions/A. That sure explained why it would be about twice the size. Interestingly our nightly builds didn’t show this behavior: there Current was a symlink to A for each as—we’d expect it to be. So although the nightly build system was the baseline we started with, we must have altered something along the way.

Then it hit us: the main difference is that in the nightly job packages the .app-Folder as a tar.bz directly and submits it to the Github release from the build job, while in the publishing action we store the folder as a Github Artifact that a second job after downloads and submits to the store.

Why does that matter? The Github Artefact is stored as a ZIP, too (to save disk space), after all. Yes, but for zipping by default is that symbolic links are resolved. As we are zipping the entire folder the symlinks that is usually Versions/Current -> Versions/A is resolved, meaning the files are stored twice. Yet the signature is only stored once and only for Versions/A (not for Versions/Current). So when we download that zipped version, we have an .app-folder with each framework version stored twice yet only a signature for one (and the file having about twice the size). Looking at the error messages sent by Apple, the last batch of errors even gives a hint to that problem.

Finding that issue, one and off, took us a month. Yet the fix was small and trivial: we moved the productbuild to create a .pkg-file from the publishing job into the build-job and store that .pkg-file as the artefact. Problem solved.

These are just a few things we wished we had known before. Do you have any additional tips for that—apparently niche—Flutter MacOS build systems you wished someone had told you before? Let us know below!