DEV Community: Vandebron

Replacing App Center with GitHub Actions

John Fisher — Sat, 22 Feb 2025 13:03:27 +0000

Why GHA?

This seems like a lot of work... Why not go with an off-the-shelf solution from something like Bitrise or Codemagic? Fair question but not the main point of this post. To look into our reasoning check out the Benefits section!

Some notes before we get started

We use this workflow to build two apps, one of which can be white-labeled, so we have additional app and white-label-release inputs which makes things a bit trickier. You might not need that so feel free to trim stuff down and make it your own! But for us, we why we have the env-variable-prep-android.sh which normalizes the variable names used for secrets so those secrets and build file names, etc. can be easily used. If you just have a single app you probably don't need this script.
The code below is only for the build process. Though the Architecture Decision Record (ADR) considered how this would affect future decisions about artifact upload automation and rolling out releases for internal testing, nothing about that is automated here.
There's a lot of code here. We removed version numbers to make sure we're not giving out too much information. If you see something like @vx.x.x, you'll have to fill those in with the versions that are needed for your app/pipeline.

Implementation

Part 1 - Basic Setup

Add the files below. Nothing in this setup should effect App Center but it is good to check in your changes to a branch and test those against the regular App Center build flow. The main point of the steps in this section is to just get a basic action, with inputs, set up and running. Full implementation is further down. The env-variable-prep-android.sh and env-variable-prep-ios.sh files created variables based off the input parameters passed in so they can be used in a reusable and consistent manner later on. The env-file-prep.sh file is responsible for creating a .env file with values from whichever environment you're using. Though it's nice to add the Fastlane and Gymfile files, those won't be used until Step 4 when the rest of the pipeline is fleshed out.

Android

file: .github/workflows/mobile-apps-build-android.yaml

  name: Mobile App Build for Android
  run-name: "Build: Android, ${{ inputs.app }}, ${{ inputs.environment }}, white-label-release=${{ inputs.white-label-release }}"
  on:
    workflow_dispatch:
      inputs:
        app:
          description: 'App'
          required: true
          type: choice
          options:
            - my-first-app
            - my-second-app
        environment:
          description: 'Environment'
          required: true
          type: choice
          options:
            - test
            - acceptance
            - production
        white-label-release:
          description: 'White-labeled Release?'
          type: boolean
          default: false
  jobs:
    build:
      name: Build app for android
      runs-on: ubuntu-latest
      environment: ${{ inputs.environment }}
      steps:
        - name: Checkout ${{ github.repository }}
          uses: actions/checkout@vx.x.x
        - name: Echo Input
          env:
            APP: ${{ inputs.app }}
            WHITE_LABEL_RELEASE: ${{ inputs.white-label-release }}
            ENVIRONMENT: ${{ inputs.environment }}
          run: |
            echo "APP=$APP"
            echo "WHITE_LABEL_RELEASE=$WHITE_LABEL_RELEASE"
            echo "ENVIRONMENT=$ENVIRONMENT"

file: env-variable-prep-android.sh

  #!/bin/bash
  ARTIFACT_NAME="mobile-my-first-app-${ENVIRONMENT}-release"
  FASTLANE_ANDROID_FLAVOR="${ENVIRONMENT}"
  {
    echo "artifact-name=${ARTIFACT_NAME}"
    echo "fastlane-android-flavor=${FASTLANE_ANDROID_FLAVOR}"
  } | tee -a "$GITHUB_OUTPUT"
  # Secrets should not get sent out over tee command (which also prints it to console)
  {
    echo "android-key-alias=my-first-app"
    echo "android-keystore-file=../keystores/my-first-app.jks"
    echo "android-keystore-pass=${ANDROID_KEY_PASSWORD}"
    echo "android-store-pass=${ANDROID_STORE_PASSWORD}"
  } >> "$GITHUB_OUTPUT"

file: ./fastlane/Fastfile

  opt_out_usage
  platform :android do
    desc 'Build app for Android'
      lane :build do
        build_android_app(
          task: 'bundle',
          flavor: ENV['FASTLANE_ANDROID_FLAVOR'],
          build_type: 'Release',
          project_dir: 'android/',
          print_command: true,
          print_command_output: true,
        )
      end
  end

iOS

file: .github/workflows/mobile-apps-build-ios.yaml

  ---
  name: Mobile App Build for iOS
  run-name: "Build: iOS, ${{ inputs.app }}, ${{ inputs.environment }}, white-label-release=${{ inputs.white-label-release }}"
  on:
    workflow_dispatch:
      inputs:
        app:
          description: 'App'
          required: true
          type: choice
          options:
            - my-first-app
            - my-second-app
        environment:
          description: 'Environment'
          required: true
          type: choice
          options:
            - test
            - acceptance
            - production
        white-label-release:
          description: 'White-labeled Release?'
          type: boolean
          default: false
  jobs:
    build:
      name: Build app for iOS
      runs-on: macos-latest
      environment: ${{ inputs.environment }}
      steps:
        - name: Checkout ${{ github.repository }}
          uses: actions/checkout@vx.x.x
        - name: Echo Input
          env:
            APP: ${{ inputs.app }}
            WHITE_LABEL_RELEASE: ${{ inputs.white-label-release }}
            ENVIRONMENT: ${{ inputs.environment }}
          run: |
            echo "APP=$APP"
            echo "WHITE_LABEL_RELEASE=$WHITE_LABEL_RELEASE"
            echo "ENVIRONMENT=$ENVIRONMENT"

file: env-variable-prep-ios.sh

  #!/bin/bash
  # Secrets should not get sent out over tee command (which also prints it to console)
  {
    echo "ios-mobile-provisioning-profile=${IOS_MOBILE_PROVISIONING_PROFILE}"
  } >> "$GITHUB_OUTPUT"

file: ./fastlane/Gymfile

  scheme(ENV['SCHEME'])
  workspace("ios/my-first-app.xcworkspace")
  export_options("ios/exportOptions.plist")
  output_directory("ios/build")
  output_name(ENV['SCHEME'])

Both Android & iOS

file: env-file-prep.sh

  #!/usr/bin/env bash
  #####################################################
  # Creates an .env file for use in react-native-config
  # This script should be run from the /mobile folder
  #####################################################
  if [ "${ENVIRONMENT}" != "test" ] && [ "${ENVIRONMENT}" != "acceptance" ] && [ "${ENVIRONMENT}" != "production" ]; then
      echo "ENVIRONMENT is not set, using 'production' as default."
      ENVIRONMENT=production
  fi
  echo "Copying 'my-first-app/.env.${ENVIRONMENT}' to 'my-first-app/.env'..."
  cp "my-first-app/.env.${ENVIRONMENT}" "my-first-app/.env"
  echo "Success!"

Part 2 - Testing your Workflow Locally

WARNING: GitHub Actions doesn't let you run a workflow_dispatch action until it has been merged into the main branch so you'll want to get the basic setup above in place before you start tinkering with application logic or things that could effect App Center. Also note that while act is helpful for getting the basics in place, it's unlikely you'll be able to test the complete process locally because, at least for us a) Android pipeline crashes halfway through the Build App step with an error of Gradle build daemon disappeared unexpectedly (it may have been killed or may have crashed), and b) iOS tries to install a fresh copy of Xcode

Follow installation instructions on their User Guide

Create a ~/.actrc file that looks like this

  --container-architecture linux/amd64
  --secret GITHUB_TOKEN=$GITHUB_TOKEN

Get your $GITHUB_TOKEN env variable in place
1. Install gh, the GitHub cli
2. Modify shells (zsh shown below)
```
  export GITHUB_TOKEN="$(gh auth token)"
```
3. Close and re-open your terminal. Test to make sure you can see your GITHUB_TOKEN envrionment variable
Set up a temporary local file for secrets. WARNING: Once you’re done, remember to delete the file so it’s not hanging around on your system!! (or just don’t create it in the first place unless you really need it)

file: .secrets
```
  ANDROID_STORE_PASSWORD="someSecr3ts"
  ANDROID_KEY_PASSWORD="YouWantQuotesBecause*s_etc.WillScrewYouUp"
  MAPBOX_READ_TOKEN=shhhhh.Its.asecret
```

Set up a temporary local file for your workflow trigger inputs.

file: app-input.json

  {
    "action": "workflow_dispatch",
    "inputs": {
      "app": "my-first-app",
      "environment": "test",
      "white-label-release": false
    }
  }

Run your workflow like this. As far as I know, you can only run the iOS workflow locally if you're on a mac. To do that you'll need to provide an additional parameter: -P macos-latest=-self-hosted

Android

  act \
  --workflows .github/workflows/mobile-apps-build-android.yaml \
  --eventpath="${HOME}/Documents/IT/GitHub Actions/Mobile Apps/app-input.json" \
  --secret-file="${HOME}/Documents/IT/GitHub Actions/Mobile Apps/.secrets"

iOS

  act -P macos-latest=-self-hosted \
  --workflows .github/workflows/mobile-apps-build-ios.yaml \
  --eventpath="${HOME}/Documents/IT/GitHub Actions/Mobile Apps/app-input.json" \
  --secret-file="${HOME}/Documents/IT/GitHub Actions/Mobile Apps/.secrets"

Part 3 - Testing Things out from GHA

Once you're happy with your changes locally, send a PR & merge them in! Remember this is only the basic setup to get the action avaible from the GitHub Actions interface so maybe leave a comment in the PR notifying your teammates about how they should expect a follow-up PR
Go to the "Actions" tab at the top of your GitHub repo. You should now see your actions for building iOS and Android on the left. Since it's a workflow_dispatch action you'll trigger the actions manually.

Part 4 - Implement the Full Workflow for iOS and Android

Adjust the files below. This is where you may end up needing to modify things that affect your App Center build. Try to keep them to a mimimum so you can still use App Center for builds should anything not work as expected. Fastlane is a tool that helps with automating build and release processes for mobile apps. You can think of it as a toolbox of easy-to-use wrapper functions around gradle for Android, and xcodebuild for iOS.

Android

file: .github/workflows/mobile-apps-build-android.yaml

  ---
  name: Mobile App Build for Android
  run-name: "Build: android, ${{ inputs.app }}, ${{ inputs.environment }}, white-label-release=${{ inputs.white-label-release }}"
  on:
    workflow_dispatch:
      inputs:
        app:
          description: 'App'
          required: true
          type: choice
          options:
            - my-first-app
            - my-second-app
        environment:
          description: 'Environment'
          required: true
          type: choice
          options:
            - test
            - acceptance
            - production
        white-label-release:
          description: 'White-labeled Release?'
          type: boolean
          default: false
  jobs:
    build:
      name: Build app for android
      runs-on: ubuntu-latest
      environment: ${{ inputs.environment }}
      steps:
        - name: Checkout ${{ github.repository }}
          uses: actions/checkout@vx.x.x
        - name: Prep Env Variables
          id: prep-env-variables
          working-directory: mobile
          env:
            APP: ${{ inputs.app }} # my-first-app | my-second-app
            ENVIRONMENT: ${{ inputs.environment }}  # production | test | acceptance
            BRAND: ${{ inputs.white-label-release && 'other-company' || 'my-company' }}
            APP_1_ANDROID_KEY_PASSWORD_VANDEBRON: ${{ secrets.APP_1_ANDROID_KEY_PASSWORD_VANDEBRON }}
            APP_1_ANDROID_STORE_PASSWORD_VANDEBRON: ${{ secrets.APP_1_ANDROID_STORE_PASSWORD_VANDEBRON }}
            APP_1_ANDROID_KEY_PASSWORD_WHITE_LABEL: ${{ secrets.APP_1_ANDROID_KEY_PASSWORD_WHITE_LABEL }}
            APP_1_ANDROID_STORE_PASSWORD_WHITE_LABEL: ${{ secrets.APP_1_ANDROID_STORE_PASSWORD_WHITE_LABEL }}
            APP_2_ANDROID_KEY_PASSWORD_VANDEBRON: ${{ secrets.APP_2_ANDROID_KEY_PASSWORD_VANDEBRON }}
            APP_2_ANDROID_STORE_PASSWORD_VANDEBRON: ${{ secrets.APP_2_ANDROID_STORE_PASSWORD_VANDEBRON }}
          run: bash ./env-variable-prep-android.sh
        - name: Create .env file
          working-directory: mobile
          env:
            APP: ${{ inputs.app }}
            BRAND: ${{ steps.prep-env-variables.outputs.inferred-brand }}
            ENVIRONMENT: ${{inputs.environment}}
          run: bash ./env-file-prep.sh
        - name: Enable Corepack
          run: corepack enable
        - name: Setup NodeJS
          id: setup-node
          uses: actions/setup-node@vx.x.x
          with:
            node-version: 18.x
            registry-url: https://registry.npmjs.org
            cache: 'yarn'
            cache-dependency-path: mobile/${{ inputs.app }}/yarn.lock
        - name: Setup Java
          uses: actions/setup-java@vx.x.x
          with:
            distribution: 'temurin'
            java-version: '20'
            cache: 'gradle'
        - name: Setup Android SDK  # sadly no caching capabilities here
          uses: android-actions/setup-android@vx.x.x
          with:
            log-accepted-android-sdk-licenses: false
            packages: 'tools'   # Default is 'tools platform-tools but we don't need platform-tools for packaging'
        - name: Set up ruby env # Fastlane is a "Ruby gem"
          uses: ruby/setup-ruby@vx
          with:
            ruby-version: '3.3.0' # Changing this to 3.3 will give you "Your Ruby version is 3.3.5, but your Gemfile specified 3.3.0"
            bundler-cache: true
        - name: Install Gem Bundler
          working-directory: mobile/${{ inputs.app }}
          run: |
            gem install bundler
            bundle install --quiet
        # Install Dependencies
        - name: Yarn Install
          working-directory: mobile/${{ inputs.app }}
          run: yarn install --immutable
        # Build
        - name: Build App
          run: bundle exec fastlane android build
          working-directory: mobile/${{ inputs.app }}
          env:
            ENVIRONMENT: ${{ inputs.environment }}
            FASTLANE_ENV_INFERRED_BRAND: ${{ steps.prep-env-variables.outputs.inferred-brand }}
            ANDROID_KEYSTORE_FILE: ${{ steps.prep-env-variables.outputs.android-keystore-file }}
            ANDROID_KEY_ALIAS: ${{ steps.prep-env-variables.outputs.android-key-alias }}
            ANDROID_STORE_PASSWORD: ${{ steps.prep-env-variables.outputs.android-store-pass }}
            ANDROID_KEY_PASSWORD: ${{ steps.prep-env-variables.outputs.android-keystore-pass }}
            FASTLANE_ANDROID_FLAVOR: ${{ steps.prep-env-variables.outputs.fastlane-android-flavor }}
        # Upload
        - name: Upload application
          uses: actions/upload-artifact@vx
          with:
            name: ${{steps.prep-env-variables.outputs.artifact-name}}
            path: "mobile/${{ inputs.app }}/android/app/build/outputs/bundle/${{ steps.prep-env-variables.outputs.fastlane-android-flavor }}Release/app-${{ steps.prep-env-variables.outputs.fastlane-android-flavor }}-release.aab"
            retention-days: 30

iOS

file: .github/workflows/mobile-apps-build-ios.yaml

  ---
  name: Mobile App Build for iOS
  run-name: "Build: iOS, ${{ inputs.app }}, ${{ inputs.environment }}, white-label-release=${{ inputs.white-label-release }}"
  on:
    workflow_dispatch:
      inputs:
        app:
          description: 'App'
          required: true
          type: choice
          options:
            - my-first-app
            - my-second-app
        environment:
          description: 'Environment'
          required: true
          type: choice
          options:
            - test
            - acceptance
            - production
        white-label-release:
          description: 'White-labeled Release?'
          type: boolean
          default: false
  jobs:
    build:
      name: Build app for iOS
      runs-on: macos-latest
      environment: ${{ inputs.environment }}
      steps:
        - name: Checkout ${{ github.repository }}
          uses: actions/checkout@vx.x.x
        - name: Prep Env Variables
          id: prep-env-variables
          working-directory: mobile
          env:
            APP: ${{ inputs.app }} # my-first-app | my-second-app
            BRAND: ${{ inputs.white-label-release && 'other-company' || 'my-company' }}
            ENVIRONMENT: ${{ inputs.environment }}  # production | test | acceptance
            APP_1_MY_COMPANY_IOS_MOBILE_PROVISIONING_PROFILE: ${{ secrets.APP_1_MY_COMPANY_IOS_MOBILE_PROVISIONING_PROFILE }}
            APP_1_OTHER_COMPANY_IOS_MOBILE_PROVISIONING_PROFILE: ${{ secrets.APP_1_OTHER_COMPANY_IOS_MOBILE_PROVISIONING_PROFILE }}
            IOS_BUILD_CERTIFICATE_P12_OTHER_COMPANY: ${{ secrets.IOS_BUILD_CERTIFICATE_P12_OTHER_COMPANY }}
            IOS_BUILD_CERTIFICATE_P12_PASSWORD_OTHER_COMPANY: ${{ secrets.IOS_BUILD_CERTIFICATE_P12_PASSWORD_OTHER_COMPANY }}
            APP_2_IOS_MOBILE_PROVISIONING_PROFILE: ${{ secrets.APP_2_IOS_MOBILE_PROVISIONING_PROFILE }}
          run: bash ./env-variable-prep-ios.sh
        - name: Create .env file
          working-directory: mobile
          env:
            APP: ${{ inputs.app }} # my-first-app | my-second-app
            BRAND: ${{ steps.prep-env-variables.outputs.inferred-brand }}
            ENVIRONMENT: ${{inputs.environment}}
          run: bash ./env-file-prep.sh
        - name: Import Build Certificate from Secrets
          uses: apple-actions/import-codesign-certs@vx
          with:
            p12-file-base64: ${{ steps.prep-env-variables.outputs.ios-build-certificate-p12 }}
            p12-password: ${{ steps.prep-env-variables.outputs.ios-build-certificate-p12-password }}
        - name: Import Mobile Provisioning Profile
          uses: nickwph/apple-provisioning-profile-action@vx.x.x
          with:
            profile-base64: ${{ steps.prep-env-variables.outputs.ios-mobile-provisioning-profile }}
        - name: Setup NodeJS
          id: setup-node
          uses: actions/setup-node@vx.x.x
          with:
            node-version: 20.x
        # Mapbox v10 ships with bitcode on XCode 16 which is not allowed by the App Store
        # https://github.com/mapbox/mapbox-maps-ios/issues/2233
        # Once this issue is fixed we can upgrade to 16
        - name: Install XCode
          uses: maxim-lobanov/setup-xcode@vx.x.x
          with:
            xcode-version: 15.4
        - name: Install Ruby
          uses: ruby/setup-ruby@vx.x.x
          with:
            ruby-version: 3.3.0
        - name: Install Bundler
          run: gem install bundler
        # TODO: Cache node_modules similarly to how Pods are cached (compare hash of yarn.lock)
        - name: Yarn Install
          working-directory: mobile/${{ inputs.app }}
          run: yarn install --immutable
        # TODO: Cache Gems similarly to how Pods are cached (compare hash of Gemfile.lock)
        - name: Install Gems
          run: bundle install
          working-directory: mobile/${{ inputs.app }}
        - name: Cache CocoaPods dependencies
          uses: actions/cache@vx
          env:
            FILES_GLOB: mobile/${{ inputs.app }}/ios/Podfile.lock
          with:
            path: |
              mobile/${{ inputs.app }}/ios/Pods
            key: ${{ runner.os }}-pods-${{ hashFiles(env.FILES_GLOB) }}
            restore-keys: |
              ${{ runner.os }}-pods-
        - name: Install Pods
          working-directory: mobile/${{ inputs.app }}/ios
          run: bundle exec pod install
        - name: Build iOS App
          env:
            APP: ${{ inputs.app }}
            SCHEME: ${{ steps.prep-env-variables.outputs.ios-scheme }}  # otherCompanyFirstAppProduction, myCompanySecondAppTest
            BRAND: ${{ steps.prep-env-variables.outputs.inferred-brand }}
            TARGET: ${{ steps.prep-env-variables.outputs.ios-target }}
          run: bundle exec fastlane gym
          working-directory: mobile/${{ inputs.app }}
        # Upload
        - name: Upload application
          uses: actions/upload-artifact@vx
          with:
            name: ${{steps.prep-env-variables.outputs.ios-scheme}}
            path: "mobile/${{ inputs.app }}/ios/build/${{ steps.prep-env-variables.outputs.ios-scheme }}.ipa"
            retention-days: 30
            overwrite: true
            if-no-files-found: error

Part 5 - Troubleshooting

More than likely these won't work the first time. Time to go back and adjust. Note that since the workflow is now in the main branch you can test your workflow changes on a feature branch. Just select your feature branch in the "Branch" dropdown shown above.

Other Things to Note

App Center gives you the ability to write `appcenter-pre-build.sh` and `appcenter-post-build.sh` scripts. The `env-file-prep.sh` is basically that same thing, just without the context of appcenter.
One of our apps is using MapBox which needs a `.netrc` in the root directory. If you need something similar, you can add a step to your action by adding the code shown in the the "optional mapbox" part + the `./my-first-app/prep-mapbox.sh` in the Appendix section below.
After our work on this was done (and only as I'm writing this article) we realized we can use the `sparse-checkout` option from `actions/checkout` to only check out the needed files. This should speed up our workflow runtime even more!
It may be helpful for you to add the build and/or version number onto the artifact. We haven't done that but it's something we're interested in adding for the future.
Troubleshooting certificates for iOS was a HUGE pain in the butt. I don't have any good advice here other than to realize (and communicate with your POs) that this part may take a while.

Benefits

It regularly took over 50 minutes for our mobile app to build in App Center. Part of that could have very likely be improved by adjusting App Center configurations & how we store and bundle app assets but after migrating our builds to GitHub Actions our app build times are now down to 22 minutes - More than twice as fast!
All the rest of the software at Vandebron (backend services in Scala and Python and frontend applications in Typescript + React) is built using GitHub Actions. This move brings mobile apps in line with all other software. This move to GHA for mobile builds has led to several of our mobile devs getting our hands dirty in GHA, which is great because we can now play a role in the larger CICD discussions.
We have full control over our CICD pipeline for mobile builds. In the future we can integrate more Fastlane commands to further automate the release process.
We did a full ADR (shown below) which initiated the work here. Links referenced in image are in Appendix below.

Appendix

Mapbox Integration

file: .github/workflows/mobile-apps-build-android.yaml and .github/workflows/mobile-apps-build-ios.yaml (optional mapbox)

  - name: Create Mapbox .netrc file (my-first-app only)
    if: ${{ inputs.app == 'my-first-app' }}
    working-directory: mobile
    env:
      MAPBOX_READ_TOKEN: ${{ secrets.MAPBOX_READ_TOKEN }}
    run: bash ./my-first-app/prep-mapbox.sh

file: ./my-first-app/prep-mapbox.sh

  #!/usr/bin/env bash

  # WARN: DO NOT use tee here (it prints to console)
  {
    echo "machine api.mapbox.com"
    echo "login mapbox"
    echo "password ${MAPBOX_READ_TOKEN}"
  } >> ~/.netrc
  chmod 0600 ~/.netrc

ADR Links

Choosing Remix as a Server-Side Rendering (SSR) Framework

John Fisher — Thu, 31 Oct 2024 11:29:50 +0000

The Background

We at Vandebron have a mission to get the news out about our good work, and we understand that Server Side Rendering (SSR) can really help with that. Among other things, it provides an easy way for search engines to discover our pages, so you, our (future?!) customer, can find them more easily. That means more people choosing green energy, and ultimately, a cleaner environment! 🎉

We rolled our own

The year was 2017, Covid was still a word that sounded more like a bird than anything else... The world was heating up and Vandebron was 4 years into its mission to bring 100% renewable energy throughout all of the Netherlands.

As far as web technologies are concerned, 4 years was ages ago. It was a time when NextJS was less than a year old, and Remix was still several years from coming out. But we needed a way to deliver that high-quality content to all of you. So, the innovators that we were, we decided to build our own SSR framework. In short, we wanted pizza, but there were no pizza shops in town... So we made our own!

It's been great but not without issue...

A Short Note: Why Server Side Rendering

You might not be satisfied with the short explanation of why we picked an SSR framework in the first place. This article isn't really about that - if you're interested in more analysis on when and where to choose an SSR framework, check out these excellent articles from Splunk:

Decisions Made the Right Way - A Comparison

Nowadays, there are better, industry standard technologies available! I.e. pizza shops have opened nearby!! Let's find a good one. Of course, you don't want to just go to any spot. Especially if there's more than one shop in town - you'd be silly not to check which one is closest, and look at the menu. Which one has better reviews, is that one very angry customer just upset that there wasn't any anchovies in the vegan pizza shop? What were they expecting anyway?

At Vandebron we're a React shop, so we limited ourselves to just SSR frameworks supporting React. The choice of one framework over another is of crucial importance, so, as part of our analysis, we built a small part of our vandebron.nl/blog page twice. Two of our engineers then presented these prototypes to our Front End Guild, and this discussion fed heavily into the Architecture Decision Record that we wrote comparing the results.

* At Vandebron, Guilds are groups of engineers from disparate teams that are interested in a single domain: i.e. Backend, Frontend, IAM and Auth, etc.

The Background for the decision record states this:

"Our Frontend currently uses a custom-built, hard to maintain SSR solution, which we'd like to replace with a modern and standard library. Possible candidates are NextJS and Remix. The goal is to investigate which one suits our needs best."

Yes, there are other options we could have considered but we wanted to stay with a tried-and-tested framework and one that was compatible with our existing React setup.

As you can see, the comparison between the two frameworks was very similar. In the end we favoured the simple, opinionated direction of Remix over that of the more full-featured but potentially complex setup of NextJS. Even though Remix has a smaller community, we attributed this mostly to the age of the framework and not the quality of the framework itself. Though the Positivity has gone down a bit (as listed in the 2023 StateOfJS survey,) the decrease has been relatively minor and in line with most-other frameworks (notable exceptions for Astro and SvelteKit which have both seen big upticks in both Usage and Positivity)

Finally, we noted that NextJS is tightly coupled with Vercel. At Vandebron we value platform independence and not getting tied to specific hosting providers or platforms. Remix gives us the independence we're looking for by providing a SSR framework without a potential to be tied into other solutions/platforms in the future.

Outcome

"Most members favoured Remix’s focus on web standards and usage of standard libraries and were put off (a little) by NextJS’s uncertainty in development direction."

So, How's it Going?

The migration effort is still underway but already we can report that it's going quite well - developers are excited to work on the new tech stack because it's seen as a developer-friendly platform and one of the two leading frameworks in the industry. In the words of one engineer: "Dev experience has improved massively, it's fun, it's easy to work with"
Here are some of the things we still need to work on:

Our Docker image is quite large as it includes all the node_modules. We think we can clean this up a bit by using Yarn's Plug'n'Play (PnP) feature which should lead to faster image-build times and faster container startup times.
With our custom SSR solution, we use Redux Toolkit (RTK) and RTKQuery on the server... This is of course an anti pattern on the server, since server-side logic should be stateless. The Remix framework does already tries to be smart with it's loaders, so the benefits we might have gotten from RTK aren't needed there.
We feel the application we're migrating from is doing too much - it includes our marketing pages like the Blog and Mission pages we've been working on for the initial release, as well as the pages for our our signup and renewal process (become a Vandebron customer here!!!) This is a separate conversation, and ultimately one for the FE Guild, but the existing app's size and purpose is making the migration take longer than it should, and forcing us to put some routing rules in place to make sure the right parts of our old site are getting swapped out for the new.
Previously, many of the images and PDFs we used on our website were checked directly into the repo. Part of our migration to Remix made us realize we should be using a CMS for this. We are already integrated with a CMS, we just need to be making better use of it in some cases.
We haven't explored the Remix-specific linting rules yet. While we're confident in the existing React and TS lint rules we already have, it seems like configs like @remix-run/eslint-config could be quite handy.

Creating a Self-Service Data Model

Mats Stijlaart — Tue, 06 Feb 2024 10:42:38 +0000

The title of this article could have also been "Getting Rid of an Unmanageable Legacy Data Model", but after a year-long migration project the current title does more justice to the progress made.

Compiled Legacy as a Data Model

Our former data model was a series of parallel custom python jobs all covering every step of the Extract-Transform-Load (ETL) process from sources into report. Specific transformation got performed a numerous amount of times in multiple different jobs, daily. This made us prone to bugs, slow on development and maxing out on compute.

The situation became so pressing that keeping alive simple reporting to the business became a daily burden on the Data Analytics team, limiting resources for advanced analytics and leveraging data sources for competitive insights.

We concluded the old set-up to be outdated and looked around for current best practices concerning data infrastructure. Trying not to reinvent the wheel and staying away from designing custom solutions that had bitten us in the past, we decided to adopt a combination of Snowflake, dbt and Lightdash to start forming a new data landscape.

This revamping of the set-up gave us the opportunity to start over, using the power of dbt to create a modular data model where you could leverage different stages of data, while creating shared definitions, a single source of truth and documentation.

What We Came Up With?

We went for a pretty classic dbt data model design, introducing 5 layers of data: staging, entity, intermediate, mart and api. Each layer serving a specific purpose.

Staging

With all data coming in from different sources, this is where we ensure the data all adheres to the same conventions and formatting. This introduces a nice developer experience for the next layers, by introducing consistency across different sources. It also serves as the go to place for advanced or deep dive analysis that do not get answered by the downstream layers, which could potentially spark data modelling developments.

Entity

After uniforming the data, we create entities that form the building blocks of the downstream layers and analyses of our business analysts. We built entities along the core aspects of our product, capturing shared definitions in data and bringing together relevant features using the One-Big-Table (OBT) principle. We try to refrain from long queries or extensive use of CTE's, resulting in simplistic models. These models serve our business analysts by reducing the complexity of their queries with all joins and filters taken care of, denormalizing the database structure. This has shifted the place where ad-hoc BI requests are fulfilled from the central data team to the domain business teams, applying principles of a data mesh.

Intermediate

With some models rising in complexity and computation, we use the intermediate layer to split this complexity and computation across multiple models. These intermediate models are rarely queried because they serve no reporting or analytical purpose. Think of incremental date spine explosions or highly complex business logic broken down into multiple models.

Mart

This is the main layer where we provide self-service to less technical employees within the organization, creating ready-made tables. We aggregate along date spines and dimensions to create readable models. It is where we leverage Lightdash metrics to create dynamic tables to provide business teams with a certain amount of freedom in terms of the granularity and dimensions they want to report on in their dashboarding. The use of entities as building blocks has aligned reporting across domain business teams, creating a single and centralized source of truth and relieving the data team from explaining distinctions. So while the dimensions can be tweaked for specific use cases, the definitions of the metrics are set in code.

API

With some dependencies outside of the data model, we use an API layer on top of our mart to record exposures towards different services and provide views which explicitly contain only the necessary datapoints.

The Process

We decided to take advantage of the chaos created by the old situation: no real single source of truth gave us the opportunity to create a truth. Investigating business teams' needs, we created data definitions in entities. We kept a pragmatic approach to these definitions, being flexible towards business teams' specific needs but also limiting the allowed complexity or number of exceptions. The new data model should answer everyone's questions, but should also be understood by everyone.

We forced ourselves to have descriptions for all data from the entity layer onwards, because only defining and describing the entities in code is not enough. We leveraged the embedded business analysts' knowledge to form the descriptions, noting that the best description is the one the user understands (because they wrote it).

With the ready-made marts in place, we decided to give away most of the dashboarding responsibility to the business teams. The embedded analysts are very apt at defining and designing their relevant insights into dashboards. The central data team only took ownership of company wide dashboards and provided support on the dashboarding where necessary.

After the adoption of the new stack, we noticed that the more technical embedded analysts were very interested in learning a new tool and language. So, we started a data model group and onboarded multiple embedded business analysts as data model developers. This has massively increased the speed of development of the data model. Primarily, because of specific business domain knowledge not needed to be transferred to the developers in the central data team first, but the knowledge holders developed models themselves. The central data team took on a different role: providing infrastructural support, improving on efficiency, monitoring costs and creating a vision and strategy for organic but structured growth.

What Did We Learn?

Few key takeaways:

Some business teams have more requirements in terms of definitions than other teams, so if other teams allow, be pragmatic and just go for the stricter requirements.
Enabling self-service analysis means giving away control, take this into account in your data descriptions. They should be clear and concise.
Educate users on the designed structure of the data model, explain what layer serves which purpose and what questions can be answered how and where.
Create clear communication and support channels for the business to kickstart the adoption, you are not the only one learning a new tool.
Data is not only for the data team, so encourage those passionate and enthusiastic analysts to co-create! (Just keep an eye on the project.)

So, back to the monolith it is then?

Sam Theisens — Tue, 23 May 2023 06:37:33 +0000

Amazon embraces the mighty monolith

In March 2023, Amazon published a blog post detailing how they had managed to reduce the cost of their audio-video monitoring service by 90%.
The key to this reduction was migrating from a distributed, microservice architecture to a monolith.
The blog post went viral, prompting some software industry celebrities to
question the entire concept of microservices.

What should we learn from this?

So, does this mean microservices are fundamentally flawed? Should we all migrate back to monoliths?
No and definitely no I would say. Instead, my takeaways from this article are:

Microservices aren't about scaling for performance. At least not primarily. Although horizontally scalability for computationally intensive operations can be very useful or even essential in some cases, it tends to be a rare benefit. Very often, performance bottlenecks are IO bound and caused by external systems beyond your control. Nevertheless, there are other compelling reasons to consider microservices: they force you to communicate via contracts, encourage you to organize your functionality around domains, and allow you to scale your organization. Of course, all this comes at considerable costs. There's no free lunch 👇.
Don't underestimate the power of a single CPU in 2023. To judge whether a process is unreasonably slow or not, I tend to think of the fact that already in the 1990s, screens showed 65K pixels at any given time. Back then, multiple arithmetic calculations (additions, subtractions) could be performed for each pixel, fifty times per second. Nowadays, your screen probably displays more than 5 Million pixels at once. So, if the amount of datapoints you are dealing with in the order of millions, you should generally be able to process them in a matter of seconds on a single machine. If you can't, you may be doing something very inefficient.
Software engineering is hard. Mistakes are made all the time, everywhere. Even at the big 4 tech companies. Kudos to Amazon 👏 for openly sharing the mistake they made so that we may all learn. In the next section I will share one of our own experiences, not entirely different from the Amazon example.

The 90% cost reduction case at Vandebron

Microservices or just distributed computing?

Considering that all the functionality used in the Amazon case belongs to the same domain, it arguably does not even serve as
a case against improper use of microservices, but instead a case against misuse distributed computing.

Let's look into an example of misuse of distributed computing at Vandebron now.

Predicting the production of electricity

For utility companies, accurately predicting both electricity consumption and production is crucial.
Failing to do so can result in blackouts or overproduction, both of which are very costly.
Vandebron is a unique utility company in that the electricity that our customers consume is produced by a very large
amount of relatively small scale producers, who produce electricity using windmills or solar panels.
The large number and the weather dependent nature of these producers make it very hard to predict electricity generation accurately.

To do this, we use a machine learning model that is trained on historical production data
and predictions from the national weather institute. As you can imagine, this is a computationally intensive task, involving large amounts of data.
Fortunately, we have tooling in place that
allows us to distribute computations of a cluster of machines if the task is too large for a single machine to handle.

However, here's the catch: the fact that we can distribute computations does not mean that we should. Initially it seemed that
we couldn't analyze the weather data quick enough for the estimation of our production to still be a prediction
rather than a postdiction. We decided to distribute the computation of the weather data over a cluster of machines.
This worked, but it made our software more complex and Jeff Bezos even richer than he already was.

Upon closer inspection, we found an extreme inefficiency in our code. It turned out that we were repeatedly reading the entire weather dataset
into memory, for every single "pixel". After removing this performance bug, the entire analysis could easily be done
on a single machine.

What more is there to say?

So if microservices aren't about performance, what are they about? If I had to sum it up in one sentence It would be:

Microservices are a way to scale your organization

There is a lot of detail hiding in that sentence, which I can't unpack in the scope of this article. If you're interested
what microservices have meant for us, I would recommend you watch the presentation below.

Microservices at Vandebron

At Vandebron, we jumped onto the "microservice bandwagon" circa 2019. This wasn't a decision
made on a whim. We had seen a few industry trends come and go, so we first read up
and did our own analysis. We found that the concept of microservices held promise, but also knew that they would come at a cost.

These are some of the dangers we identified and what we did to mitigate them.

Danger	Mitigation
A stagnating architecture	Compile and unit-test time detection of breaking changes
Complicated and error prone deployments	Modular CI/CD pipelines
Team siloization	A single repository (AKA monorepo) for all microservices and a discussion platform for cross-domain and cross-team concerns
Duplication of code	Shared in house libraries for common functionality

The following presentation to the students of VU University, Amsterdam explains how we implemented
some of these mitigations and what we learned from them.

The difference between a component library and a design system

Petter Andersson — Mon, 08 May 2023 16:26:14 +0000

A while back, our former technology manager Roy Derks covered the subject of component libraries here on the blog. From a technical perspective, he spoke about when you need one (and when you don’t need one) and what to consider when building one. Since then, obviously a lot has happened at Vandebron. But one of the more interesting things to happen is that design became an integrated part of the digital department, as opposed to previously being attached to marketing. In this new setup, one of the first major projects the design team was involved in was the alignment of our component libraries. And no that’s not a typo, that’s libraries as in the plural form of library. Confusing? I thought so too. In this blog I’ll try to explain further why that was the case, how the work actually helped us bridge the gap between design and development, and dissect the work of unifying those component libraries into one single source of truth and ultimately what’s to become our design system.

A bit of a mess

Before we get into it, some background as to where we started out might be useful. As previously mentioned, the design team had just become a part of the digital department and one of the first tasks at hand was the creation of a design system. In the design team, we had previously worked with a certain set of brand guidelines, a style guide if you will, which had not necessarily been translated or aligned to the requirements of a digital product or development environment. Development had also created a set of stylesheets and libraries with reusable components which they used to reduce development time. Having it all separately might sound a bit counter-intuitive, but not very surprising if you consider designers and developers not being in the same department, working on a different timeline, different priorities and so forth. However, this only highlighted the importance of designers and developers working together and the need for a proper design system to help prevent creating a fence between the teams causing unnecessary and ineffective work on both sides. The result of this previous “unsynciness”, a rebrand in 2017, and a re-aligned techstack, was the existence of 3 different libraries and subsequently 3 different sources of truth within the development environment. To add to this, we also had separate design guidelines geared more towards brand/marketing purposes in the design team. Now came the rather massive task of unifying these and eventually, rather than having just a library, having a system.

Component library ≠ design system

Now, there’s a lot of terminology here that might be difficult to grasp if you’re new to the subject. So I thought I’d clarify what we mean when referring to these, how they fit into the context of our situation, and how many of them we had!

- Brand guidelines / style guide (design)

A set of guidelines and examples outlining all the visual elements of a brand such as logos, color, typography, imagery etc. and subsequently in what - - manner they should be applied. It can also be expanded to include more things brand related such as tone of voice, brand values and so forth. Often with brand guidelines, they are created from a marketing perspective and the digital experience(or product) aspect of how the brand should be applied/represented is usually thought about in the second hand, or not included at all.

Amount: 1

- Design kit/library (design)

A designer resource file with all the available building blocks that make up the digital design language of a brand and/or the user interface of a product. This is usually only visual(no code) and lives in the design software of the designer's choosing. For us this used to be Sketch, but we recently moved to Figma. Can also include documentation and examples of how the different building blocks should be applied and utilized.

Amount: 1

- Style sheet (front-end)

A set of styling properties to be applied when rendering a web page, usually in the format of CSS. This can include things related to the brand guidelines such as font size, colors, etc. but also things related to web layout such as the margins and paddings of different web elements.

Amount: 1

- Component library (front-end)

A set of dynamic web components that can be used in a development environment in order to quickly build user interfaces. This helps to ensure consistency, to avoid rebuilding the same component more than once and to avoid changing said component in more places than one, and subsequently help reduce development time.

Amount: 3

All of the above mentioned things, together with rigorous documentation, amount to what’s called a design system. Having it all combined in a structured way is key to getting the most out of such a system. In our case, most of these things were separate and not necessarily connected to each other. But what stands out most of the things above is probably the fact that we, over time, had amounted to 3 different component libraries. I mentioned earlier how that scenario had transpired so I won’t go into too much detail as to how that happened, but if you’re a developer in a small to medium-sized company and I mention “rebrand” and “new techstack” you can probably figure out how. However complex, this also proved to be an excellent opportunity for our developers and for us in the design team. We finally get to unify our component libraries into one, while simultaneously aligning it with our design kit and expanding the guidelines with new and updated documentation. Thus ensuring that designers and developers speak the same language and share the same single source of truth.

A guild forms

To kickstart this process we formed a project group(or ‘guild’) composed of 2 designers and 2 developers, each designer and developer from the two consumer-facing scrum teams. The idea was to let the developers work on the migration and unification of the component libraries in collaboration with us designers in the same project, making it easier to align and to create co-ownership of the product. Our first step was to decide on the structure of our component library, this way the developers could slot all the existing, reworked and new components into the right place in the new library. Easy enough right? Well, here comes our first challenge. We initially wanted to take an atomic approach and build our components from the well known and widely used atomic design principles. We also needed to consider the 3 different “product groups” which the library should apply to, all still utilizing the same style properties.

Vandebron has a wide range of products serving different platforms, with the visual language remaining the same but where the user interface might differ. This requires the top elements of the system(such as colors and typography) to be shared across all products, whereas the lower you get the more product-specific an element becomes. This is the reason why we wanted to structure the system according to the principles of Atomic Design first, in order to assign the components to a hierarchical structure.

With this approach the atoms would work like design tokens and the molecules would be components general enough that they’d be shared across all product groups, this CORE part of the library would essentially be the stylesheet that impacts all visual aspects of the digital brand experience. Only on organism-level do we start to differentiate what product group the component belongs to. So a change to the CORE parts of the library(atoms or molecules) would impact all components in all product groups.

However, this approach actually made less sense from a development perspective. Not that it wouldn’t work or that the categorization didn’t make sense, but it would require us rewriting all the already existing components. Components that are actively in use. We deemed this approach a bit too high-risk and high-effort for the time being and started looking into alternatives, while still keeping the atomic structure as a more long-term goal. Another thing our initial idea didn’t take into account was the experience of the future main user of the library, the developer! Organizing a design system after the brand properties and product groups makes a lot of sense from a designers or a marketeers perspective, and it should probably still be presented outwards that way, but a component library is something else(remember?). So based on our development environment and the way we build our websites and apps our developers suggested a different structure:

In this structure, similar to the previous one, the components are instead categorized and sorted by how they should be applied to the page or application that’s being built. Styles, layouts and inputs are general enough to be applied to all product groups whereas from the surface level the components start becoming more specific in their use case. That way, the components can be separated into specific or even several product groups. In this format the components themselves are not as intertwined as in the atomic structure, albeit still connected by the style element. So while it’s a bit more resistant to overall changes the main idea of having the same style properties applying to everything still works, and it helps us designers to better relate and contextualize what we’re designing from more of a development perspective, thus helping bridge the gap between development and design even further. The main insight we drew from this experience is to not let industry standards and certain trends dictate what you should do. Sure they’re important to keep an eye on, but do it with carefulness and always apply an asterisk to it. Figure out what works best for your specific situation and what’s realistic in the short-term vs. in the long-term. There’s no one-size-fits-all.

Speaking the same language

With the component library migration now underway, we started looking into ways to improve our system from the designers' side of things. As previously mentioned, we had just gone from using Sketch to using Figma and with that came a good opportunity to rebuild, adjust and expand our design kit also. We did that by removing, adding, simplifying and renaming a lot of what was in there since before and with the design kit now adjusted to match the component library we were now also speaking the same language. We can actually now compare this side-by-side with the tools we’re using. In Storybook we have attached the Figma design of every component, simply by activating the feature and pasting the link to its page or artboard in the Figma file. This will refresh in almost real-time if any changes are made so we can easily spot any differences and inconsistencies between what’s live and how the design looks. In Figma, we try to document all our components and give some context as to how it works and should be applied. This is now also directly visible to the developer in the context of the component library. Expanding on our documentation and exposing our digital design guidelines like that has been a great way to create a shared understanding of our designs. Rather than just an image being tossed over a fence, there is now quite literally a direct connection between design and development and therefore also more of a shared ownership.

Further defining the process

As all the alignment on the design side and the migration neared completion, we started seeing a lot of things that could be improved upon or even added to our component library. When we started logging these things down on our project backlog we quickly realized that the scope of our project had quickly been growing into something beyond what was initially intended, and that rather than giving us focus this guild format was instead at risk of creating an isolated bubble of knowledge surrounding the design system. This prompted us to gauge the opportunity and capacity among our development teams to instead tackle these tasks together, either alongside or within their respective day-to-day tasks. In order to do so we needed the buy-in from key stakeholders such as the product owners from the respective development teams. It’s obviously a big ask to get 1 developer from each team to work on improving a component library, especially when they’ve already given us a quarter on migration and have other important business and/or user needs to tend to. So instead, we looked into how we can embed the improvement and further development of our design system into the developers current processes and primary day-to-day work. We structure this by linking our component library improvement/addition tickets to relevant tickets in their respective sprints. In defining the workflow like this, our 2 designer 2 developer guild in effect rendered unnecessary and instead we opened up the co-ownership and contribution to all developers in all customer-facing development teams and in the process of it preventing isolating knowledge too much. In opening up the process like this, another positive side effect we see is the involvement, engagement and subsequent use of our component library going up. With the product designers now also actively a part of the front-end guild meetings, we have an ever bigger forum and bigger opportunity to build a world class component library and design system while also having more hands on deck to work on maintenance and improvements. We still have a long way to go, but all parts are now even more aligned and the future is looking bright!

What’s next

In the newly formed designer+developer guild, the work of defining requirements and improvements on the design system continues. From the design side we’re also looking to constantly improve on the documentation and the presentation of our system. This is something we imagine we’ll keep on doing continuously and iteratively for as long as it’s needed, if not forever. After all, “design is never done” and a design system can and should be a living thing constantly evolving along with the products and the brand it serves, and in extension even the promise of the brand and it’s products. In our case, that’s to aid in accelerating the energy transition towards 100% renewable energy. More on how we exactly do that, and how we always aim to design for impact, in the next blog post. Thanks for reading and stay tuned!

Petter Andersson, Product Designer at Vandebron

If the type of work mentioned in this blog post sounds interesting to you, take a look at our job openings here.

"Fueling" the Energy Transition With Spark - Part 1

Rosario — Fri, 05 May 2023 09:05:18 +0000

Here at Vandebron, we have several projects which need to compute large amounts of data. To achieve acceptable results, we had to choose a computing tool that should have helped us to build such algorithms.

As you may have read in other articles our main backend language is Scala so the natural choice to build distributed parallel algorithms was indeed Spark.

What is Spark

We will briefly introduce Spark in the next few lines and then we will dive deep into some of its key concepts.

Spark is an ETL distributed tool. ETL are three phases that describe a general procedure for moving data from a source to a destination.

Extract is the act of retrieving data from a data source which could be a database or a file system.
Transform is the core part of an algorithm. As you may know, functional programming is all about transformation. Whenever you write a block of code in Scala you go from an initial data structure to a resulting data structure, the same goes with Spark but the data structures you use are specific Spark structures we will describe later.
Load is the final part. Here you need to save (load) the resulting data structure from the transformation phase to a data source. This can either be the same as the extract phase or a different one.
Distributed: Spark is meant to be run in a cluster of nodes. Each node runs its own JVM and every Spark data structure can/should be distributed among all the nodes of the cluster (using serialization) to parallelize the computation.

Spark data structure: RDD, DataFrame, and Dataset

The core of Spark is its distributed resilient dataset (RDD).

An RDD is a collection of elements partitioned across the nodes of the cluster that can be operated on in parallel. Extracting data from a source creates an RDD. Operating on the RDD allows us to transform the data. Writing the RDD loads the data into the end target like a database for example). They are made to be distributed over the cluster to parallelize the computation.

A DataFrame is an abstraction on top of an RDD. It is the first attempt of Spark (2013) to organize the data inside and RDD with an SQL-like structure. With dataframe, you can actually make a transformation in an SQL fashion. Every element in a dataframe is a Row and you can actually transform a dataframe to another by adding or removing columns.

A DataSet finally is a further abstraction on top of a dataframe to organize data in an OO fashion (2015). Every element in a dataset is a case class and you can operate transformation in a scala fashion from a case class to another.

Spark in action

Let’s see now some code samples from our codebase to illustrate in more detail each of the ETL phases.

Extract

The extraction phase is the first step in which you gather the data from a datasource.

val allConnections = sparkSession
.read
.jdbc(connectionString, tableName, props)
val selectedConnections = allConnections
.select(ColumnNames.head, ColumnNames.tail: _*)
val p4Connections = selectedConnections
.filter(allConnections("HasP4Day activated").equalTo(1))
.filter(allConnections("HasP4INT activated").equalTo(1))
.as[Connection]
p4Connections.show()

For most people the extraction phase is just the first line (the invocation to the read method), they are not wrong because extracting means reading data from a datasource (in this case an SQL server database). I decided to include in this phase also some filtering and projection operations because I think these are not really part of the algorithm, this is still the preparation phase before you actually process the data. We can ultimately say that preparing the data is something in between extraction and transformation therefore it is up to you to decide which phase it belongs to.

Transform

Transformation phase is the core of the algorithm. Here you actually process your data to reach your final result.

usageDF
.groupBy('ConnectionId, window('ReadingDate, "1 day"))
.agg(
    sum('Consumption).as("Consumption"),
    sum('OffPeak_consumption).as("OffPeak_consumption"),
    sum('Peak_consumption).as("Peak_consumption"),
    sum('Production).as("Production"),
    sum('OffPeak_production).as("OffPeak_production"),
    sum('Peak_production).as("Peak_production"),
    first('ReadingDate).as("ReadingDate"),
    first('marketsegment).as("marketsegment"),
    collect_set('Source).as("Sources"),
    collect_set('Tag).as("Tags"),
    max('Last_modified).as("Last_modified")
)
.withColumn(
    "Tag", when(array_contains('Tags, “Interpolated”),
lit(Tag.Interpolated.toString)).otherwise(lit(“Measured”)))
.withColumn("Source",
when(size('Sources) > 1,
lit(Source.Multiple.toString)).otherwise(mkString('Sources)))
.orderBy('ConnectionId, 'ReadingDate)
.drop("window", "sources", "tags")

In this specific example, we are processing connection usage data by aggregating it daily. In the usageDF we have 15 minutes interval usage data, now we want to show to the user the same data but with a different aggregation interval (1 day). So we group the whole data by connection id and window the reading date by 1 day (A window function calculates a return value for every input row of a table based on a group of rows Introducing Window Functions in Spark SQL - The Databricks Blog.

Once the data is grouped we can aggregate it, using the agg method which allows us to call the aggregation functions over the dataframe (for example: sum, first,max or collect_set). Successively we transform the dataframe to suit our visualization needs, the methods used are self-explanatory and the documentation is very clear. Getting Started - Spark 3.0.1 Documentation

Load

The final phase is the one which save, put, show the transformed data into the target data source.

dataFrame
.select(columns.head, columns.tail: _*)
.write
.cassandraFormat(tableName, keySpace)
.mode(saveMode)
.save()

In this specific case, we will save our dataframe into a Cassandra database. In Spark, methods used to achieve the load phase are called actions. It is very important to distinguish Spark actions from the rest because actions are the only ones that trigger Spark to actually perform the whole transformation chain you have defined previously.

If our transformation phase, as we described above, wasn’t followed by an action (for example save) nothing would have happened, the software would have simply terminated without doing anything.

One concept to rule them all

val rdd1 = sc.parallelize(1 to 10)
val rdd2 = sc.parallelize(11 to 20)
val rdd2Count = rdd1.map(
x => rdd2.values.count() * x //This will NEVER work!!!!
)

One does not simply use RDD inside another RDD. (Same goes for Dataframes or Datasets).

This is a very simple concept that leads very often to lots of questions because many people just want to use Spark as a normal scala library. But this is not possible due to the inner distributed nature of Spark and its data structures. We have said that an RDD is a resilient distributed dataset, let’s focus on the word distributed, it means that the data inside it is spread across the nodes of the cluster. Every node has its own JVM and it is called Executor, except for the master node where your program starts which is called Driver:

Your code starts from the Driver and a copy is distributed to all executors, this also means that each executor needs to have the same working environment of the Driver, for Scala it is not a problem since it just needs a JVM to run. (but we will see that if you use pySpark you need to take extra care when you distribute your application.) Every Spark data structure you have defined in your code will also be distributed across the executors and every time you perform a transformation it will be performed to each chunk of data in each executor.

Now let’s go back to our example, a map is a transformation on rdd1 this means that block inside will be executed at the executor level, if we need rdd2 to perform this block Spark should somehow serialize the whole rdd2 and send it to each executor. You can understand now that it is really not possible to serialize the whole RDD since it is by its nature already a distributed data structure. So what can you do to actually perform such computation we showed in the example? The solution is “simple”: prepare your data in such a way that it will be contained in one single RDD. To do so you can take advantage of all the transformation functions Spark has to offer such map join union reduce etc.

Next step…

We have explained all the main concepts of Spark and we have shown some real snippets of our codebase. In the next article, I would like to show you a real-life problem we have solved in our company using pySpark. I will show you how to customize Spark infrastructure to correctly parallelize the ETL algorithm you have built.
Footer

The Why and How of Dagster User Code Deployment Automation

Pieter Custers — Mon, 01 May 2023 14:53:58 +0000

This post originally appeared on vandebron.tech.

TL;DR

If you want to deploy new Dagster user code respositories, you need to modify and redeploy the whole Dagster system (while they are presented as separate in the docs). This is undesirable for many reasons, most notably because it slows down a migration or the regular development process. This post presents a way to avoid this and build a fully automated CI/CD-pipeline for (new) user code.

This article assumes that:

you (plan to) host Dagster on Kubernetes and manage its deployment with Helm and Ansible;
you want to automate the deployment of new Dagster user code repositories with a CI/CD pipeline automation tool of choice;
and you want to be able to (re)deploy the whole Dagster system and user code from scratch.

Why Dagster?

In short Dagster is a tool to build and orchestrate complex data applications in Python. For us, in the end, Dagster improved the development cycle for things like simple cron jobs as well as for complex ML pipelines. Testing the flows locally was never so easy, for instance. And with features like asset materialization and sensors, we can trigger downstream jobs based on the change of an external state that an upstream job caused, without these jobs having to know of each other's existence.

However, deployment of new user code respositories caused us some CI/CD related headaches...

System and user code are separated

Dagster separates the system deployment - the Dagit (UI) web server and the daemons that coordinate the runs - from the user code deployment - the actual data pipeline. In other words: the user code servers run in complete isolation from the system and each other.

This is a great feature of which the advantages are obvious: user code repositories have their own Python environment, teams can manage these separately, and if a user code server breaks down the system is not impacted. In fact, it even doesn't require a restart when user code is updated!

In Helm terms: there are 2 charts, namely the system: dagster/dagster (values.yaml), and the user code: dagster/dagster-user-deployments (values.yaml). Note that you have to set dagster-user-deployments.enabled: true in the dagster/dagster values-yaml to enable this.

Or are they?

That having said, you might find it peculiar that in the values-yaml of the system deployment, you need to specify the user code servers. That looks like this:

workspace:
    enabled: true
    servers:
      - host: "k8s-example-user-code-1"
        port: 3030
        name: "user-code-example"

This means system and user deployments are not actually completely separated!

This implies that, if you want to add a new user code repository, not only do you need to:

add the repo to the user code's values.yaml (via a PR in the Git repo of your company's platform team, probably);
do a helm-upgrade of the corresponding dagster/dagster-user-deployments chart;

but because of the not-so-separation, you still need to:

add the user code server to the system's values.yaml (via that same PR);
and do a helm-upgrade of the corresponding dagster/dagster chart.

Formally this is the process to go through. If you are fine with this, stop reading here. It's the cleanest solution anyway. But it is quite cumbersome, so...

If you are in a situation in which new repositories can get added multiple times a day - for instance because you are in the middle of a migration to Dagster, or you want a staging environment for every single PR - then read on.

Give me more details

How it works is that for every new repo Dagster spins up a (gRPC) server to host the user code. The separation is clear here. But the Dagster system also needs to know about these user code servers, and it does so through a workspace-yaml file. If you run Dagit locally it relies on a workspace.yaml file; on Kubernetes it relies on a ConfigMap - a Kubernetes object used to store non-confidential data in key-value pairs, e.g. the content of a file - which they named dagster-workspace-yaml.

This workspace-yaml is the connection between the system and the user code. The fact that the charts are designed as such that this workspace-yaml is created and modified through the system deployment rather than the user code deployment is the reason we need to redeploy the system.

But what if we could modify this workspace-yaml file ourselves? Can we make the system redeployment obsolete? Short answer: we can.

Our solution

Disclaimer: what we present here is a workaround that we'll keep in place until the moment Dagster releases a version in which the Dagster user code deployment is actually completely separated from the system deployment. And it works like a charm.

Remember: the desired situation is that we do not have to edit the values-yaml files (through a PR) and redeploy all of Dagster for every new repo.

First of all, we added an extra ConfigMap in Kubernetes that contains the values.yaml for the dagster/dagster-user-deployments chart. We named it dagster-user-deployments-values-yaml. The fact that this is a ConfigMap is crucial to prevent conflicts (see next section).

With the extra ConfigMap in place, these are the steps when a repo gets added:

Add the new repo to the dagster-user-deployments-values-yaml Configmap.
Helm-upgrade the dagster/dagster-user-deployments chart with the content of that ConfigMap.
Add the server to the dagster-workspace-yaml ConfigMap.
Do a rolling restart of the dagster-dagit and dagster-daemon deployment to pull the latest workspace to these services.

Refresh the workspace in the UI and there it is, your new repo!

Notes:

The steps above are completely automatable through your favorite CI/CD pipeline automation tool.
There is no interaction with a (platform team) Git repo.
The process, unfortunately, still requires a restart of the system in order to pull the latest workspace-yaml to the system services. The daemon terminates, then restarts, and it might cause a short interruption. Note that this is unavoidable if you add a new repo, no matter how you add it. This could be avoided if a reload of the ConfigMap would be triggered upon a change, which is possible but not enabled.
If you want to make changes to an existing repo (not code changes but server setting changes), you only have to do the first step (and modify instead of add).

How to prevent conflicts

With many of your team members adding new Dagster repositories through an automated CI/CD pipeline, you might face the situation that 2 people are adding a new repo at around the same time.

When this happens, the dagster-user-deployments-values-yaml ConfigMap cannot be uploaded in the first step because Kubernetes demands that you provide the last-applied-configuration when doing an update. If it doesn't match, the upload fails.

This is perfect as we do not want to overwrite the changes of the conflicting flow. You can optionally build in a retry-mechanism that starts over with pulling the ConfigMap again.

How to deploy from scratch

The above does not yet cover how we are able to deploy the Dagster system and user code completely from scratch. Why do we want this? Well, for instance when somebody accidently deletes the dagster namespace for instance. Or hell breaks loose in any other physical or non-physical form. Or when we simply want to bump the Dagster version, actually.

The key to this is that we version both the dagster-user-deployments-values-yaml and dagster-workspace-yaml as a final step to the flow described above (we do it on S3, in a versioned bucket). Whenever we redeploy Dagster (with Ansible) we pull the latest versions and use them to compile both the values-yaml files from it.

How to clean up old repositories

The above described automation adds new repos but doesn't take care of old obsolete repos. The steps for removing a repo are the same for adding one. The exact implementation depends on your situation. You might want to automatically remove PR staging environments after closing a PR, for instance.

Conclusion

Dagster is an incredibly powerful tool that enabled us to build complex data pipelines with ease. This posts explains how we streamlined the CI/CD pipeline for user code respositories, which enabled us to migrate to Dagster very quickly and saves us lots of time on a daily basis.

How to Spin Up A Kubernetes Cluster On Your Macbook

Vandebron Technology — Tue, 25 Apr 2023 19:12:35 +0000

In Vandebron we have been using container clusters to host our services since the foundation of our Big Data team. Recently our cluster of choice has declared End-Of-Life development stage, so we decided to take a step forward and get a ticket for the Kubernetes boat.

A change in the OS that is used to run your services and applications can look quite challenging and not everyone is on the same experience level. To make everyone comfortable it is a good choice to give everyone the possibility to play with the new tools and learn what can be done and how: you need a sandbox.

Our developers are provided with a Macbook and at the moment of writing there some options you can go for when deciding how to set up your playground:

Docker CE Kubernetes: This is the easiest solution since there is a handy button to run your containers into a Kubernetes environment.
Vagrant and Virtualbox: This solution is the one that can give you more control and you can easily create a cluster the size you want, but you need to be handy with VMs, the hypervisor of choice, and Vagrant. It's the old school way to do it but, while it's a chunk your platform engineers can bite, it can be a steep and frustrating process for people that are not used to handle VMs.
Multipass + some bash magic glue: Since Canonical created this tool for macOS, creating an Ubuntu VM became a breeze and you can have a single, easily manageable VM with its networking up and running in less than a minute, without having to handle disks, distros, and stuff. On top of it, the command line interface is straight forward and it has just the basic commands we will need, so wrapping the entire process into a bash script is a piece of cake.

I have found this super cool in-depth article from Jason Yee (kudos to you bruh) that guided me through the installation of my first single node Kubernetes cluster.

The process is not that long but it involves a lot of copy/pasting and, once learned the basics, I didn't want to go under the same process more times, plus it could be interesting for me as a Platform Engineer, but it may be boring and pointless for developers who just want to have a sandbox replica of what they are working on in the remote environment. My automator (aka do-it-once-never-do-it-again) spirit kicked in and I decided to wrap every step in a small command-line tool with only 3 options:

install
cleanup
help

What is happening under the hood

What the script does is substantially automating all the steps needed to:

Create a new VM using Multipass (tool released by Canonical)
Fetch the VM IP address and adding it to your local /etc/hosts file
Install k3s (a lightweight distribution of Kubernetes) on top of the VM
Install the Kubernetes command-line tools on your laptop
Install Helm (the Kubernetes package manager) on your laptop
Install cert-manager (certificate manager) package on top of your k3s cluster
Install Rancher (a Kubernetes control plane) package on top of your k3s cluster

If you are looking for a more in-depth breakdown of the single steps you can download and inspect the script (one of the many advantages of OpenSource projects) or checkout and read the original article: it explains line by line what the specific commands are doing.

1. Multipass VM

Multipass is a tool from Canonical (the company developing and maintaining the Ubuntu Linux distribution) that leverages Hyperkit (macOS feature to handle virtualization) to create and handle a Virtual Machine directly on your Mac.

2. Edit /etc/hosts

Once we have our VM up and running we need to make it available with an easy url that is also gonna be used to generate the SSL certificate, in our case we picked up rancher.localdev. It is important to have a name setup in the beginning since this one will need to match with the certificate so we can use it programmatically.

3. Install K3S

This step is pretty straightforward: just fetch a script that is publicly available on the k3s official website and feed it to your bash. K3s is a lightweight version of Kubernetes with all the needed dependencies and executable packaged in a convenient installation script. Because of its light nature, it is often used in embedded devices that have a limited amount of resources to offer.

4 & 5. Kubernetes and Helm cli

Kubernetes cli (kubectl) is used to talk and interact with your Kubernetes cluster. It can be used to manage multiple clusters according to the content of your KUBECONFIG environment variable. The variable itself contains just a path to where your cluster configuration is stored, so you can switch from a cluster to another by simply pointing to another file that contains the configuration of another cluster.

Helm instead is the "package manager" of Kubernetes: you can use it to add repositories to specific charts which are the blueprint that contains a way to install a specific tool on your cluster. Both of these tools have to be installed and run from your local laptop, either in the case you are managing a local VM or in the case you are interacting with a remote cluster.

6 & 7. cert-manager and Rancher

Rancher is the control plane for our cluster: it provides a GUI and an overview of our single node cluster. It offers other goodies like management of multiple clusters, deployed on different locations like AWS Azure and GCP or even on your own hardware, plus certificate deployment and some other handy functionalities.

cert-manager is installed via Helm chart and it is the tool used by Rancher to generate and deploy a certificate across the entire cluster.

How to use it

All the steps will involve the use of a Terminal window

Installation

The first thing you need to do is download this script and save it in a folder on your Mac (let's assume ~/bronernetes) by executing

    ```
{% endraw %}
bash
mkdir ~/bronernetes
    cd ~/bronernetes
    curl https://gist.githubusercontent.com/nikotrone/50b1a5f8d137411879eb2467e689bfbe/raw/090b4b4323d96ac28d96bbb346e2e657073722e6/bronernetes > bronernetes
    export PATH=$PATH:$(pwd)
{% raw %}




Now we have the toolset and you can confirm it works by simply running `bronernetes help`.

#### Spin up Kubernetes

The next step is to run the installation process with the command `bronernetes install`

#### Clean up

When you are done or you just want to hard reset your environment you can just type `bronernetes cleanup` and it will take care of cleaning up the VM you just used, leaving you with a pristine machine, as nothing ever happened :)

### Conclusion

Having a sandbox is very useful to play around with the concepts of a new setup or service and it packs up a huge amount of positive sides. No matter what is the language or the nature of the system you are trying to replicate, it can be challenging and involve a long list of instructions or manual operations and, sometimes, even dedicated hardware. Although with some bash glue, it is possible to automate most of those processes and the investment cost can be enormously beneficial for yourself (less work the next time you do it) and for the other people working with you (they can use the tool, comment and suggest improvements). Most of all, in the case of infrastructure, it helps raise the knowledge of "what's going on here" and documents for the ones interested in taking a trip down the rabbit hole.

***

This post was originally published on [vandebron.tech](https://vandebron.tech/blog/spin-up-kubernetes-on-macbook). Reposted automatically with [Reposted.io](https://reposted.io?utm_source=postFooter).

Cypress.io Component Design Technique for React Applications

Vandebron Technology — Tue, 25 Apr 2023 19:11:32 +0000

Cypress is a game-changer in the automation testing world, the way that Cypress was built and its architecture allows us as testers to cover more scenarios.

Cypress is not Selenium; in fact, it is different. And the way to build and design a framework should be different as well.

The most famous design technique in Selenium is the Page Object Model, and many testers use the same design technique with Cypress. Even that Cypress on their official website recommended us not to go with that approach.

Page Object Model

The main benefit of using the page object model Is to make the automation framework maintenance-friendly. We can define a specific page's selectors in a separate file and then use these selectors in our test cases.

class SignInPage {
  visit() {
    cy.visit("/signin");
  }
  getEmailError() {
    return cy.get(`[data-testid=SignInEmailError]`);
  }
  getPasswordError() {
    return cy.get(`[data-testid=SignInPasswordError]`);
  }
  fillEmail(value) {
    const field = cy.get(`[data-testid=SignInEmailField]`);
    field.clear();
    field.type(value);
    return this;
  }
  fillPassword(value) {
    const field = cy.get(`[data-testid=SignInPasswordField]`);
    field.clear();
    field.type(value);
    return this;
  }
  submit() {
    const button = cy.get(`[data-testid=SignInSubmitButton]`);
    button.click();
  }
}
export default SignInPage;

The main two downsides using the typical page object model with cypress are:

Page objects introduce an additional state into the tests, separate from the application’s internal state. This makes understanding the tests and failures harder.
Page objects make tests slow because they force the tests to always go through the application user interface.

Component-Based Architecture

On the other hand, a React application is component-based, where a specific page will be built from a collection of components. And components in React can be used on different pages too. So if we want to use the Page Object Model, we may define the same locator twice on different pages.

So having these two facts, At Vandebron, we came up with a new way to design our Cypress Automation framework by creating a separate JavaScript file for every component in our application, inside a folder called components within our Cypress project as below:

// Locators
export const getEmailError = () => cy.get(`[data-testid=SignInEmailError]`);
export const getPasswordError = () =>
  cy.get(`[data-testid=SignInPasswordError]`);
export const emailField = () => cy.get(`[data-testid=SignInEmailField]`);
export const passwordField = () => cy.get(`[data-testid=SignInPasswordField]`);
export const submitButton = () => cy.get(`[data-testid=SignInSubmitButton]`);

// Actions
export const visit = () => cy.visit("/signin");
export const performLogin = (email, password) => {
  emailField().clear().type(email);
  passwordField().clear().type(password);
  submitButton().click();
};

Having it built this way, we eliminated all the previous problems mentioned earlier; we are not adding any classes, and we are defining objects within our test cases. And the most important part is that we are following the way that Cypress recommends it.

And after defining the component locators and actions, we can import them inside our test case and use them as below:

import LoginComponent from "../components/loginComponent";
import Menu from "../components/Menu";

describe("Test Login Page", () => {
  it("should show an error message if the password in wrong", () => {
    LoginComponent.visit();
    LoginComponent.performLogin("email@gmail.com", "wrongPassword");
    LoginComponent.getPasswordError().should("be.visible");
  });
  it("should show the logout button if the user logged in succesfully", () => {
    LoginComponent.visit();
    LoginComponent.performLogin("email@gmail.com", "correctPassword");
    Menu.LogoutButton().should("be.visible");
  });
});

And as you can see, our test cases are readable for anyone! And if any locator changes in any of the components, we can easily fix it in one location and from the same file. And lastly, if a component will be used in different places, we can use the same code.

In the next article, I will talk about how we use Cypress in our manual testing during the sprint and how it saves us tons of time and effort.

This post was originally published on vandebron.tech. Reposted automatically with Reposted.io.

Migrating from DCOS to Kubernetes, dealing with the l4lb loadbalancer

Vandebron Technology — Tue, 25 Apr 2023 18:55:39 +0000

In October 2020 D2IQ announced that they are moving onwards with their Kubernetes offering. Vandebron has been a D2IQ customer for their DCOS offering, we were just in the middle of a migration of our first workloads to DCOS Enterprise. We have evaluated the D2IQ K8s offering and decided to go for another Kubernetes product. We had a few migrations over the years, we migrated from Azure to AWS, we migrated workloads from normal instances to spot instances and all these migrations were done with nearly any downtime. We plan to reduce the downtime to a couple of minutes this migration and this is a real challenge. The first challenge that we will discuss today: We want to pair our Kubernetes clusters to the DCOS/Mesos clusters, while we move a workload it should be able to connect to its dependencies in the DCOS cluster. We use DCOS for our NoSQL databases like Cassandra, internal data that we want to keep internal. Pairing DCOS and Kubernetes clusters enable us to reduce downtime, enabling us to switch back if we run into issues and move faster because it reduces complexity.

L4LB

The internal layer 4 load balancer DCOS provides is used in the majority of our workloads. When our data scientists schedule a spark driver, they connect to the spark dispatcher through the Layer 4 load balancer. Most of the DCOS frameworks use this Layer 4 load balancer as an internal service discovery tool, with Vandebron we use this layer 4 load balancer to communicate between services. In a default DCOS set up this load balancer responds on domain names like: spark-dispatcher.marathon.l4lb.thisdcos.directory:7077

When we ping the spark dispatcher we get the following:

PING spark-dispatcher.marathon.l4lb.thisdcos.directory (11.155.161.35) 56(84) bytes of data.
64 bytes from 11.155.161.35 (11.155.161.35): icmp_seq=1 ttl=64 time=0.024 ms

After some investigation we found out that this IP range is not actually on a network interface, it is a Linux kernel functionality called IPVS. With IPVS you can do layer 4 load balancing, you provide the target location and the location you want to respond on.

When we search for the IP from the spark dispatcher with ipvsadm, we get 3 results:

sudo ipvsadm -L -n |grep --color '11.155.161.35\|$'
TCP  11.155.161.35:80 wlc
  -> 10.2.7.146:16827             Masq    1      0          0
TCP  11.155.161.35:4040 wlc
  -> 10.2.7.146:16826             Masq    1      0          0
TCP  11.155.161.35:7077 wlc
  -> 10.2.7.146:16825             Masq    1      0          0

As you can see the IP 11.155.161.35 points towards 10.2.7.146, even the ports are configured and forwarded. We can add our route with ipvsadm, to understand IPVS a bit better. For example:

sudo ipvsadm -A -t 1.2.3.4:80 -s wlc # we add the target server and assign the scheduler
sudo ipvsadm -a -r 10.2.7.146:16825 -t 1.2.3.4:80 -m # we configure the real server and target server and configure Masquerading
curl 1.2.3.4:80
{
  "action" : "ErrorResponse",
  "message" : "Missing protocol version. Please submit requests through http://[host]:[port]/v1/submissions/...",
  "serverSparkVersion" : "2.3.4"
}

This results in that the spark dispatcher now also is available on 1.2.3.4:80. As mentioned before we wanted to connect our DCOS and Kubernetes clusters, getting hundreds of entries from ipvsadm and manually adding them one by one didn’t sound appealing to us. Especially if you consider that sometimes services fail and run on a different port or different host after recovery, maintaining this by hand would be a nightmare. We therefore decided to build a tool to sync IPVS entries from DCOS to Kubernetes.

Stack

Within Vandebron we have our tech stack, we strongly believe it is good to eat your own dog food. When possible and when our use cases are similar we use the same tools as our Developers use. The parts of the stack we will be using are:

AWS ELB in front of Traefik 1.7
DCOS
Kubernetes

Within our platform team, we use Golang as our scripting language. Golang gives us the ability to build binary files with all the required libraries in the binary, we don’t have to install any packages, we do not even need to install Golang on the machine the application will be running on.

In our DCOS cluster we use Traefik 1.7, this version of Traefik only forwards HTTP requests. We decided to use Traefik to expose a JSON endpoint so we can gather the IPVS information from this location.

ipvs-server

Within our DCOS cluster we will expose the IPVS information through a JSON endpoint. We have built a tool for this to expose this information in multiple ways. In the next section, we are going to discuss some of the concepts and choices we made, we won’t deep dive into Go specifics. We have provided the entire code for this project in the examples directory of our GitHub repo: https://github.com/Vandebron/tech-blog

First, let’s discuss the library we use: https://github.com/nanobox-io/golang-lvs. This library in its essence translates to ipvsadm commands, it helped save us time to implement this ourselves. There are some gotcha’s, such as newlines are not filtered out from the output. We solved this by cleaning up some of the data.

In the childChan function we create a go channel that is responsible for polling ipvsadm every 10 seconds and stores the result in a couple of variables we use in our HTTP endpoints. IPVS is a Linux kernel functionality and should be highly performant, we do not want to trigger kernel panics when the server gets overloaded with requests. We expect that every 10 seconds gives us accurate enough results, we can always lower this interval to ensure faster results. We also added in this function the string manipulation to ensure all the newlines were gone in the JSON output. The newline gave issues when we tried to add the IPVS scheduler entries.

func childChan(c chan bool) {
   fmt.Println("Starting time based IPVS Admin poll")

   pollInterval := 10
   timerCh := time.Tick(time.Duration(pollInterval) * time.Second)
   // Time based loop to generate Global variable
   for range timerCh {
       select {
       // when shutdown is received we break
       case <-c:
           fmt.Println("Received shutdown, stopping timer")
           break
       default:
           var err error
           listIpvs.Save()
           ipvsString = fmt.Sprintln(listIpvs.Services)

           res := &responseObject{
               Services: listIpvs.Services,
           }

           ipvsJSONbyte, err := json.Marshal(res)
           if err != nil {
               logToErr.Printf("ERROR: -- Marshal JSON -- %v\n", err)
           }

           ipvsString = string(ipvsJSONbyte)
           ipvsJSON = strings.Replace(ipvsString, `\n`, ``, -1)
           if debug != false {
               logToOut.Println("DEBUG: -- ipvsJSON --", ipvsJSON)
           }
       }
   }
}

Next is the index handler, we set our headers correctly and print the result as we would receive through ipvsadm. The index is mainly for our platform engineers to debug and verify the output. Thanks to this overview we found much faster that there was a newline hidden in the scheduler output.

func index() http.Handler {
   // Generating the Index
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

       // Only available when debug is on
       if debug != false {
           logToOut.Println("DEBUG: -- index --", ipvsString)
       }

       if r.URL.Path != "/" {
           http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
           return
       }
       w.Header().Set("Content-Type", "text/plain; charset=utf-8")
       // Site security testers expect this header to be set
       w.Header().Set("X-Content-Type-Options", "nosniff")
       w.WriteHeader(http.StatusOK)
       fmt.Fprintln(w, ipvsString)
   })
}

The JSON endpoint is what we use in the client communicate with the server.

func jsonz() http.Handler {
   // Generating the Index
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {

       // Only available when debug is on
       if debug != false {
           logToOut.Println("DEBUG: -- jsonz --", ipvsJSON)
       }

       if r.URL.Path != "/json" {
           http.Error(w, http.StatusText(http.StatusNotFound), http.StatusNotFound)
           return
       }
       w.Header().Set("Content-Type", "application/json; charset=utf-8")
       // Site security testers expect this header to be set
       w.Header().Set("X-Content-Type-Options", "nosniff")
       w.WriteHeader(http.StatusOK)
       fmt.Fprintln(w, ipvsJSON)
   })
}

We ask our Developers often to implement a basic health endpoint, in DCOS we use this to see if a service needs to be restarted. In our application we enable set the statusOK in the index or in the JSON endpoint.

func healthz() http.Handler {
   // Generating the healthz endpoint
   return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
       if atomic.LoadInt32(&healthy) == 1 {
           w.WriteHeader(http.StatusNoContent)
           return
       }
       w.WriteHeader(http.StatusServiceUnavailable)
   })
}

In our logging and tracing functions we want to register the clients that are connecting, this gives us information where calls are coming from. It helps us debugging if we see weird behaviour.

func tracing(nextRequestID func() string) func(http.Handler) http.Handler {
   // Tracing the http requests so its easier to check if server is reached
   return func(next http.Handler) http.Handler {
       return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
           requestID := r.Header.Get("X-Request-Id")
           if requestID == "" {
               requestID = nextRequestID()
           }
           ctx := context.WithValue(r.Context(), requestIDKey, requestID)
           w.Header().Set("X-Request-Id", requestID)
           next.ServeHTTP(w, r.WithContext(ctx))
       })
   }
}

func logging(logToOut *log.Logger) func(http.Handler) http.Handler {
   // Creating logging entry tracing the http requests
   return func(next http.Handler) http.Handler {
       return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
           defer func() {
               requestID, ok := r.Context().Value(requestIDKey).(string)
               if !ok {
                   requestID = "unknown"
               }
               logToOut.Println(requestID, r.Method, r.URL.Path, r.RemoteAddr, r.UserAgent())
           }()
           next.ServeHTTP(w, r)
       })
   }
}

IPVS needs to be executed with root privileges, to ensure this is correct we get the userid and print it when starting the server.

// getProcessOwner function is to see who is running the process. It needs to be a sudo / root user
func getProcessOwner() string {
   stdout, err := exec.Command("ps", "-o", "user=", "-p", strconv.Itoa(os.Getpid())).Output()
   if err != nil {
       logToErr.Printf("ERROR: -- getProcessOwner -- %v\n", err)
       os.Exit(1)
   }
   return string(stdout)
}

We added the init function to ensure we print results the moment the server starts up, if we would not do this it would take 10 seconds for the go channel to activate

func init() {
   // Placing the Save and val in the init, else we will need to wait for channel to perform its first run
   listIpvs.Save()
   ipvsString = fmt.Sprintln(listIpvs.Services)
}

In the main function, we set the configurable flags, such as debugging to show error messages. It proved useful during the creation of this tool to keep track and print output. If we would print the output at every call to our logs, our Elastic cluster would get thousands of logs that add little to no value.

We configure the listen port in the flags, we can use the portIndex from DCOS to assign a random port on the host to listen on. We also provided to print the version we are running. In our versioning, we use a constant to list the application semver version, we also provide the git-commit hash. When we begin the server we print the version information, the port we listen on and the user running the process. We then start the server process with the go channel, in setting up the go channel we ensure that when the server stops we try to gracefully stop the server within a 30-second timeframe. Since our ipvsadm timer is 10 seconds it should be able to cleanly shutdown within that period.

Docker build

In the repository, we have included a Dockerfile and a script to build the Dockerfile. In this Dockerfile, we pass the git commit hash to the go install. This way we always get the Git Hash from our GitHub repo and we can use this information in our version output.

DCOS service.json

In the repository, we have provided the service.json file, since it is opinionated on using Traefik you might need to change it. But in this service.json you see how we set up Traefik, the health check, and port index. Since the Mesos UCR container has fewer abstractions and has fewer limited capabilities. We can run the IPVS server inside a UCR container and get all the output as if we were running this directly as root on the host machine.

ipvs-client

The IPVS client is the component we use in the Kubernetes environment. The client connects to the server and gets the IPVS entries from the IPVS server inside our DCOS cluster. It then adds these IPVS entries to each node in the Kubernetes cluster. You, therefore, need to run each client per Kubernetes node.

You can find the code from the IPVS client in our repository.

func httpGet(remoteURL string) []byte {
   if debug != false {
       _, err := url.ParseRequestURI(remoteURL)
       if err != nil {
           panic(err)
       }
   }

   req, err := http.NewRequest(http.MethodGet, remoteURL, nil)
   if err != nil {
       logToErr.Fatalf("ERROR: -- new HTTP request -- %v", err)
   }

   ipvsClient := http.Client{
       Timeout: time.Second * 2, // Timeout after 2 seconds
   }
   req.Header.Set("User-Agent", "go-ipvs-get \tversion: "+version+"\t Git Commit: "+gitCommit)
   res, err := ipvsClient.Do(req)
   if err != nil {
       logToErr.Fatalf("ERROR: -- ipvsClient -- %v\n", err)
   }

   if res.Body != nil {
       defer res.Body.Close()
   }

   body, readErr := ioutil.ReadAll(res.Body)
   if readErr != nil {
       logToErr.Fatalf("ERROR: -- body -- %v\n", readErr)
   }

   return body
}

In the httpGet function we can debug the URL and check if it is valid. Again we set the correct headers and retrieve the JSON body.

func unmarshal(body []byte) []lvs.Service {

   res := &responseObject{
       Services: listIpvs.Services,
   }

   jsonErr := json.Unmarshal(body, &res)
   if jsonErr != nil {
       logToErr.Fatalf("ERROR: -- Unmarshal -- %v \n", jsonErr)
   }

   if debug != false {
       logToOut.Fatalf("DEBUG: -- res -- %v \n", res.Services)
   }

   r := res.Services

   return r
}

In the unmarshal function we unmarshal the JSON and turn it in a slice of lvs.Service.

func addServers(remoteAddr string) {
   body := httpGet(remoteAddr)
   jsonData := unmarshal(body)

   for i, v := range jsonData {
       if debug != false {
           logToOut.Printf("DEBUG: -- range jsonDATA --\n")
           logToOut.Printf("ipvsCount=%v, value=%v", i, v)
       }

       err := lvs.DefaultIpvs.AddService(v)
       if err != nil {
           logToErr.Printf("ERROR: -- AddService -- %v", err)
       }

       i++
       ipvsServerCount = float64(i)
   }
}

In the addServers function we add the servers to IPVS.

func clientChan(c chan bool) {
   logToOut.Println("Starting time based IPVS Admin add")

   pollInterval := 10
   timerCh := time.Tick(time.Duration(pollInterval) * time.Second)
   // Time based loop to generate Global variable
   for range timerCh {
       select {
       // when shutdown is received we break
       case <-c:
           logToOut.Println("Received shutdown, stopping timer")
           break
       default:

           logToOut.Println("Clearing & Adding servers...")
           // Before we add Servers we need to clear the existing list
           lvs.Clear()
           addServers(remoteAddr)
           if debug != false {
               logToOut.Printf("IPVS servers added:\t%v", ipvsServerCount)
           }
       }
   }
}

Like we did in the IPVS server we create a go channel to poll every 10 seconds the server endpoint. We perform this to get at a set interval the IPVS entries.

Since we run the IPVS client as a binary directly on the Kubernetes hosts we build the binary with a few parameters we pass to the go build command. The binary we build with this command we host on an internal s3 bucket, we can download this binary with systemd unit files.

GOOS=linux
GOARCH=amd64
GIT_COMMIT=$(git rev-list -1 HEAD)

export GOOS
export GOARCH
export GIT_COMMIT

env GOOS=${GOOS} GOARCH=${GOARCH} go build -v -ldflags "-X main.gitCommit=${GIT_COMMIT}" .

When we run the IPVS client we can verify if the IPVS routes are added by running the ipvsadm -L -n command.

Unit files

Since IPVS is part of the Linux kernel it is hard to deploy this in a docker container, the capabilities are more restricted in Kubernetes. We decided to deploy the IPVS client on each host machine through a systemd unit file, the main reason was that we ran into restrictions that slowed us down and this is not a permanent solution. By adding the IPVS client on the machines alone does not make it possible for containers to use the IPVS routes. We needed to add NET_ADMIN capabilities to all containers using the l4lb loadbalancer locations and configure hostNetworking: true in the Kubernetes pods.

We provided a deployment.yml file that runs a Ubuntu docker container with ipvsadm only installed extra. When the pods are deployed in this deployment you can use kubectl exec to get into the pod and run the ipvsadm -L -n command.

Vacancy at Vandebron

We are looking for a platform engineer in Vandebron. As you can understand this is not a typical scenario we daily run across, but it is part of the workloads that we will support when working on our platform. Within Vandebron we try to use the best technology available, when it is not available we build it. Due to this as platform engineers, we have many interesting challenges and offer engineers to support further than only a strict domain. We support all components of our entire platform, regardless if it is a Linux kernel issue like this, involves setting up and maintaining a NoSQL cluster, or helping the business with something like requesting a certificate.

If you are interested in learning more about this position, take a look at our Vacancy and get in contact with us. https://werkenbij.vandebron.nl/

This post was originally published on vandebron.tech. Reposted automatically with Reposted.io.

Looking back at the Vandebron GreenTech Hackathon 2021

Vandebron Technology — Tue, 25 Apr 2023 15:19:12 +0000

At the end of March, we organized a public hackathon to create solutions to tackle climate challenges. After having done internal hackathons, we thought it was time to share our technologies with other innovative people and companies. Together with a group of partners, and enthusiastic participants, spend three full days of (remote) hacking with great results.

Why organize a public hackathon?

Climate change is one of the many pressing challenges our society is currently facing. At Vandebron, we want to continue finding ways to tackle this immense challenge. That’s why we decided to organize a 3-day GreenTech hackathon that ran from March 31st to April 2nd, 2021. We've been organizing internal hackathons for the past four years, to foster innovation within our company and allow our developers to work on something exciting without any constraints. If you want to read more about why we organize internal hackathons, you can find an article by our CTO here.

By organizing a public hackathon, we hoped to attract a bigger audience, possibly even outside our country, The Netherlands, and attract partners to work together with. We succeeded in both, and together with Hack the Planet and Top Dutch Solar Racing, we wanted to find technological solutions to problems in wildlife conservation and renewable energy. For these three days, all participants got the opportunity to work on challenges from our partners, access their technology and knowledge, and got the chance to win unique prizes. Also, we organized a free event with speakers Florian Dirkse (The Ocean Cleanup), Thijs Suijten (Hack the Planet) and Heleen Klinkert (Nieuw Groen).

Looking back

The event started on March 31st, when all hackathon challenges were presented and the participants could select which challenge they wanted to work on. People from all over The Netherlands (and even beyond) signed up for the hackathon, ranging from students from the University of Amsterdam to young professionals looking for a job. The first challenge the participants could subscribe to was from Vandebron itself, where teams got the opportunity to use a selection of our Electronic Vehicle (EV) data. With this data, they could for example make a forecast on the amount of charging sessions we could expect on a typical day. Second, our partner Hack the Planet presented their challenge that was aimed at thinking of innovative solutions for their project Hack the Poacher. With Hack the Poacher, they install smart camera traps in African wildlife reservations to detect poachers. The teams could use their camera traps and data to create more solutions to map the poachers or use the camera traps for other needs. Finally, the students from Top Dutch Solar Racing presented a challenge to simulate the race they were supposed to join at the end of the year in Australia. Using their weather and traffic data, the teams could simulate the race and predict how much time they would need to complete the race. After selecting a challenge, all teams started the hackathon and participated in sessions to learn more about the challenges to get started.

All teams continued working on the hackathon challenge on the second day, after a nice warming-up quiz about climate change in the morning. For most teams this second day was when their project started to take shape, and they got a better idea about what they would be presenting on the final day. This second day was also an opportunity for non-technical people to get to know Vandebron and their partners better as we organized inspirational sessions with talks from different speakers in the afternoon. One of the co-founders from The Ocean Cleanup, Florian Dirkse, inspired us with his story behind making a difference in the world. After which, one of our hackathon partners Thijs Suijten, from Hack the Planet, demonstrated how technology can be used for the good. Our third, and final, speaker Heleen Klinkert (Nieuw Groen), showed how we can compensate for our CO2 emissions by storing them in the soil.

On the final day of the hackathon, all teams had to finalize their projects and create a presentation for the closing ceremony. During this ceremony, all participants and partners looked back at the past three days and shared what they had been working on during the hackathon. For every challenge, one team could win and take home several prizes, sponsored by Marie-Stella-Maris, EV Experience, and Klimaatroute. The first presentations were for the Vandebron challenge about EV forecasts. This challenge was won by not one but two teams as the jury and audience were so impressed by their solutions. Both teams created not only the forecast based on the sample data provided, but also created interactive dashboards. On the challenge for Hack the Planet, the team that won came up with a unique solution to use the camera traps to detect wild animals on the streets. For countries like India, this is a huge problem, as wild animals get stuck in traffic or walk through rural areas. The final winner of the hackathon was a group of students that simulated the Top Dutch Solar Racing trip through Australia and forecasted they could complete the race within 7 days.

Thanks everyone

I'd like to thank all the participants, prize/challenge partners, and speakers for their efforts during these days. The GreenTech Hackathon 2021 was a huge success thanks to everyone that has been involved. Keep following the vandebron.tech to be updated on future hackathons and events.

This post was originally published on vandebron.tech. Reposted automatically with Reposted.io.

How Vandebron helps balancing the Dutch energy grid together with OnLogic & Talos Linux

Tim van Druenen — Tue, 25 Apr 2023 09:19:25 +0000

Vandebron is a Dutch green-tech energy company on a mission to accelerate the transition to 100% renewable energy, 100% of the time. As part of our mission and strategy, we are constantly innovating and looking for ways to optimize energy operations and reduce negative impacts when it comes to energy production.

Our new mission: 100% renewable energy, 100% of the time

The importance of curtailment and flexibility services

One area where we are currently focusing our efforts is the area of curtailment and flexibility of wind turbines, solar parks, industrial batteries and electric vehicles. Curtailment refers to the practice of reducing the electricity inflow to balance the electricity grid. In other words, it involves adjusting the operation of, for example, a wind turbine in order to match the demand for electricity at any given time.

This is often necessary because the output of renewable energy sources can vary significantly due to changes in weather conditions. If the output of these sources exceeds the demand for electricity, it can lead to an excess of electricity on the grid, which can cause stability issues. On the other hand, if the output of wind turbines is too low, it can lead to a deficit of electricity on the grid, which can cause blackouts or other disruptions. To tackle this, we look at our customer’s batteries and electric vehicles offering flexibility capabilities.

Our journey to finding reliable, secure and energy-efficient hardware and software

To optimize these curtailment and flexibility efforts, we were in need of a gateway device that we could place at the installations of the producers on our platform. To keep it close to our mission, we preferred an ARM-based CPU for its energy efficiency compared to an x86-based CPU. After all, we don’t want to consume all of the produced energy to power an actively cooled NUC… 😉

While gathering our hardware requirements, we concluded there was really only one competitor. Therefore, we partnered up with OnLogic! We chose their Factor 201 device, which boasts the ARM-based Raspberry Pi CM4 module packed in a small and beautiful orange industrial chassis. The model also enables a lot of custom configurations. For example, we are able to configure multiple (wireless) networks, add extra SSD storage or optionally mount on DIN rails.

To ensure our gateway devices are secure and agile (like us, developers, 😛) we needed them to integrate well into our existing technology landscape based on Kubernetes. After struggling for some time to harden several (lightweight) operating systems and bootstrapping lightweight Kubernetes clusters our eyes fell on a new kid in town: ‘Talos Linux, the Kubernetes Operating system’ built by Sidero Labs. Again our predetermined wishlist was covered (even more), and what we got is a minimal OS tailored for Kubernetes, hardened, immutable and ephemeral out-of-the-box. Can you survive even more buzzwords than that?

Until the present day though, they have fulfilled every promise made on their website. It initially didn’t work on our ARM CM4-based device from OnLogic. But after testing a lot together with their team (thank you!) the latest release (v1.3.0) officially supports our ARM devices. Ready for action! Right after the stable release the first batches were shipped and connected to the installations of our producers on the platform.

Overall, Vandebron's use of OnLogic's fabricated gateway devices running Talos Linux demonstrates the potential of IoT computing to drive innovation and sustainability in the renewable energy industry. By leveraging the power of these technologies combined, we are one step closer to achieving our goal of 100% renewable energy, 100% of the time. Care to join our mission? Look for open positions.