DEV Community: Zachary Hamm

Optimizing a load balancing algorithm to minimize the runtime of a static process

Zachary Hamm — Wed, 29 Oct 2025 19:53:23 +0000

This is a followup to the following article: Load balancing Cypress tests without Cypress Cloud.

Months ago, I built a simple load balancer for the Cypress testing framework. While others exist, like the really well-designed cypress-split and an existing one in paid Cypress Cloud, I set out to see how to improve and combine all the necessary commands into one package. Originally, tests were balancing using a "round-robin" algorithm, but after time, I found that the extremes between the highs and lows from the process execution times to be inefficient. Thus, I set out to improve upon this with a new approach.

Everything below defines specifically version 0.2.9 of the “cypress-load-balancer” NPM package; please note it may change over time from how it is described here.

I feel confident that the new algorithm has reduced the timings significantly while ensuring all runners are evenly distributed, and the results at the end will demonstrate as such. Here's how I used some system design techniques to accomplish and implement this enhanced change.

The issue with using a "round-robin" approach in this case

A "round-robin" algorithm is a common, simple approach mainly used for handling dynamic traffic. VMWare does a good explanation of it here:

In a nutshell, round robin network load balancing rotates connection requests among web servers in the order that requests are received. For a simplified example, assume that an enterprise has a cluster of three servers: Server A, Server B, and Server C.

The first request is sent to Server A.

The second request is sent to Server B.

The third request is sent to Server C.

The load balancer continues passing requests to servers based on this order. This ensures that the server load is distributed evenly to handle high traffic.

There is a stark difference between an open, dynamic process and the cypress run command, however: The cypress run command is a static process that has a defined end point, whereas balancing traffic usually means that the line of communication is left open intentionally. For network requests, this means the most optimization is needed in the present load of traffic, but it does not necessarily care about the total time the whole process took if it is finally shut down. For running tests, however, we want to get tests to run as fast as possible: while improving each individual test file is an option, a second option exists to distribute them across parallel processes to minimize the amount of time it takes to fully complete testing all files.

With my initial round-robin algorithm, I sorted the test files by their expected run time, and then assigned them out to each runner, or worker process. Like the description above, if I had three runners and nine files, then the first runner gets the first file, the second gets the second file, third gets the third, and repeat. However, what this means is that since they are sorted by runtime, the first runner should always be the longest to run, and the last runner should be the shortest. The differences between these runtimes is very unbalanced: if we divide the nine files into three sets of three files each, then the first runner is always going to get the highest running time from each of those sets:

File timings (in milliseconds) [900, 800, 700, 600, 500, 400, 300, 200, 100]

Runners after load balancing:
[ [900, 600, 300], [800, 500, 200], [700, 400, 100] ]

Total run times:
[ 1800, 1500, 1200 ]

Take note of the difference between the longest runner and shortest runner is 600ms. This can definitely be improved.

Defining a new solution

Now, after noticing the extremes above, I wanted to restart and determine what I actually needed out of load balancing. Originally, it was just to evenly distribute X files between Y runners. This does not really optimize for time, however. Instead, I came up with a new goal statement:

The load balancing algorithm should keep the total runtime of the Cypress test execution process as low as possible, while evenly balancing test files amongst all available runner processes allowed.

With that in mind, I now could design a real working solution. Before even defining the boundaries of the algorithm, let's take a look at what we know about the affected process.

Initial observations of the Cypress `run` command

I collected some thoughts around Cypress and the cypress run process. Here's my list I uses to define the requirements later on:

Cypress has both an open and a run process and they are distinct from each other. The open process launches an interactive testrunner UI and can remain available after a test completes. The run process executes a set of files provided to it and ends when it is completes test execution and any post-test events.
The cypress run command is static: it has a defined start point and end point, with required inputs and various outputs.
The inputs to cypress run include a set of file names or file patterns to determine the testfiles to use; without a list of files (that it can find within the suite), it will exit after starting up without executing any tests.
Excluding setup and teardown of the process, the total test set execution time is the accumulation of the runtimes from all test files being executed.
- Furthermore, if every test could exist in a separate process run in parallel, then the minimum test set execution time of the entire process is equal to the longest-running test file. To get the lowest possible test set execution time, we must wait for the longest-running file to complete.
When parallelized runner processes are used that can have multiple tests run, then the total execution time of that process should be kept lower than the process containing the longest-running test file.
- For example, if a runner has only one file that is the longest-running file, then the accumulation of test timings across all other runners should be kept lower than the longest-running process, if possible.
To know how long a file runs, it must be executed at least once and have its timing recorded in a persistent location, like a statistics file that keeps a history of test file timings.

That's a lot of observations, but they helped immensely to shape a working solution.

Defining requirements of the new algorithm

Here's where it got fun: how do I communicate to myself what the algorithm must do? Well, based on the earlier observations, I can pick out a lot of things that are already well-defined.

Requirements for the load balancing function

Load balancing only affects the execution of the test set and does not consider any setup or teardown events, since it only records timings for test files.
The algorithm must know the following inputs:
- the list of files to balance
- a count of runners (worker processes)
- a way to read a statistics file containing the timings of the files.
- In this case, it records the durations, average, and the median execution time of a file: my algorithm uses the median execution time when balancing.
- If the timing of a file is not known, for instance, it is a new file, assume an initial timing as 0.
A plugin must be registered in the Cypress nodeSetupEvents hook to record the duration of each test after it has been executed, update the file statistics, and save them to a persistent statistical file. In this case, it goes to a directory of /.cypress_load_balancer with a file named spec-map.json. This file will be reused by the balancing function to know the timing of each file.
Each runner process will only contain the name of each file it is executing.

Requirements of the algorithm

It accepts a list of filenames, the runner count, and can access the statistical file, spec-map.json.
To start, get the statistics of each file, and then sort the file by longest to shortest runtime as sortedFiles.
Initialize X runners as empty arrays []. For each runner, remove the longest running file from the sortedFiles array and add it to the runner array. For example, for 3 runners, runner 1 gets the longest running file, runner 2 gets the second longest, and so on.
Calculate the total timing of each runner, then save the total time of the largest runner to a variable; this variable may be updated later. This variable will be named highestTotalRunnerTime.
Finally, sort the runners from shortest to longest runtime runtime; this is the inverse of how the files are sorted on purpose.
Next, it will iterate over every runner except for the largest runner. For each:
- If the runner has a larger run time than the highestTotalRunnerTime, skip adding any files to it in this iteration.
- If not, then remove the next highest file from sortedFiles and add it to the runner.
- Repeat until:
- no more files remain
- all runners have a total time equal to or greater than the original highestTotalRunnerTime.
If there are more files to add and all runners now have a time equal to or higher than highestTotalRunnerTime, then this process will repeat as such:
- If all runners have the exact same execution time, then:
- iterate over each runner and add the next highest file time; this becomes the new basis of the minimum execution time.
- Sort the runners from shortest to longest runtime again.
- Get the largest runner and set its total execution time as the new value of highestTotalRunnerTime.
- (This is to prevent a deadlock state when more files remain but no runners are equal to the highestTotalRunnerTime.)
Continue until all files have been placed in a runner.
Return the array of runners.

The "weighted-largest" algorithm in depth

Here is the algorithm function in TypeScript with some additional comments to explain how it works:

interface LoadBalancingMap {
  e2e: {
    [relativeFileName: string]: {
      stats: {
        durations: number[];
        average: number;
        median: number;
      };
    };
  };
  component: {
    [relativeFileName: string]: {
      stats: {
        durations: number[];
        average: number;
        median: number;
      };
    };
  };
}


function balanceByWeightedLargestRunner(
  loadBalancingMap: LoadBalancingMap,
  testingType: TestingType,
  runnerCount: number,
  filePaths: FilePath[]
): Runners {
  if (runnerCount === 1) return [filePaths];

  //Helper methods
  const getFile = (fp: FilePath) => loadBalancingMap[testingType][fp];
  const getTotalTime = (fps: FilePath[]) => sum(fps.map((f) => getFile(f).stats.median));
  const sortByLargestMedianTime = (fps: FilePath[]) =>
    fps.sort((a, b) => getTotalTime([a]) - getTotalTime([b])).reverse();

  //Sort **files** by highest to lowest "expected run time" (median runtime)
  const sortedFilePaths = [...sortByLargestMedianTime(filePaths)];
  const addHighestFileToRunner = (runner: FilePath[]) => {
    const file = sortedFilePaths.shift();
    if (file == null) {
      return;
    }
    runner.push(file as string);
  };

  //Initialize each runner empty
  let runners: Runners = Array.from({ length: runnerCount }, () => []) as Runners;

  //This could be done more efficiently by using array indices alongside an array of every runners' total time,
  // instead of resorting each iteration.
  sortRunners: do {
    if (sortedFilePaths.length === 0) break;

    const areAllRunnersEqualInRunTime = runners.every((r) => getTotalTime(r) === getTotalTime(runners[0]));
    if (areAllRunnersEqualInRunTime) {
      //When all runners are equal in time, pop out the file with the next highest runtime for each runner
      //This will prevent a deadlock state while also keeping files evenly spread amongst runners while still balanced
      runners.map(addHighestFileToRunner);
    }

    //Sort **runners** by lowest to highest runtime
    runners = runners.sort((a, b) => getTotalTime(a) - getTotalTime(b));

    //Get the highest runner runtime of this iteration to compare against the other smaller runners
    const highestRunTime = getTotalTime(runners[runners.length - 1]);

    //"runners.length - 2" means it will not add files to the largest runner at "runners.length - 1", since we are using it as a basis for all other runners.
    for (let i = 0; i <= runners.length - 2; i++) {
      if (sortedFilePaths.length === 0) break sortRunners;
      const currentRunner = runners[i];
      const currentRunnerRunTime = getTotalTime(currentRunner);

      if (currentRunnerRunTime >= highestRunTime) continue;
      addHighestFileToRunner(currentRunner);
    }
  } while (sortedFilePaths.length > 0);

  //Remove empty values just in case
  return runners.map((r) => filterOutEmpties(r)) as Runners;
}

Keep in mind that this algorithm is not very optimized and can be made much better, but in this case, memory usage and O(n) is not going to be a concern, even with large file sets.

If we take the "weighted-largest" algorithm and compare it against the "round-robin" with the same 9 files and 3 runners, we get this output:

File timings: [900, 800, 700, 600, 500, 400, 300, 200, 100]

Runners after load balancing:
[ [800, 500, 100], [700, 600, 200], [900, 400, 300] ]

Total run times:
[ 1400, 1500, 1600 ]

The highest time is only 200 ms less than the results of the “round-robin” algorithm, but the difference between the largest and smallest runner is now only 200ms as well. While this may seem like only a marginal improvement, the examples below demonstrate a more stunning improvement.

Examples: Comparisons of different file timing distributions

To get an accurate idea of how these will handle different file timings, I created 8 different variations of file time distributions and compared results of the "round-robin" and "weighted-largest" algorithms.

Example: Every file has the exact same timing

Here there were 9 files, each having a timing of 100. Both algorithms ended up with the same result, which should be expected.

Both had an equal distribution amongst all runners. This seems reasonable.

Example: Bell curve distribution

Values: 
[
1, 
10, 10, 
20, 20, 20, 
30, 30, 30, 30, 
40, 40, 40, 40, 40, 
50, 50, 50, 50, 50, 50, 
60, 60, 60, 60, 60, 
70, 70, 70, 70, 
80, 80, 80, 
90, 90, 
100
]

Here, we can already start to see improvement. The "weighted-largest" algorithm appropriately balanced each runner, with a spread of 1, whereas the "round-robin" approach has a spread of 80. That's pretty good so far!

Example: Extreme high-end distribution

Values:
[
100, 100, 
500, 500, 500, 500, 500, 500, 500, 500
]

I used only two runners for this trial. The results of both algorithms were equal in balancing abilities. This looks reasonable.

Example: Extreme low-end distribution, version 1: total sum of low values equals single high value

Values:
[
100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 
1000
]

Wow! We can totally see improvements with the "weighted-largest" algorithm. Both of its runners have equal time, where as the "round-robin" approach is incredibly unbalanced, with a difference of 1000 total time between its runners.

Example: Extreme low-end distribution, version 2: total sum of low values is greater than single high value

Values:
[
100, 100, 100, 100, 100, 100, 100, 100, 100, 100, 100,
1000
]

There is not much difference from the first low-end distribution trial. The "weighted-largest" algorithm produces better results.

Example: Extreme center distribution

Values:
[
10, 
20, 
30, 
40, 
50, 50, 50, 50, 50, 50, 
60, 
70, 
80, 
90, 
100
]

What is most interesting here is that it is much easier to visualize how the "round-robin" algorithm handles each value, where each runner's runtime is shown in a descending order. Still, the "weighted-largest" algorithm produces very even results.

Example: Extreme "ends" distribution

Values:
[
10, 10, 10, 10,
20,
30, 
40, 
50, 
60,
70, 
80, 
90,
100, 100, 100, 100
]

"/>

Again, the "round-robin" approach is easy to visualize in a descending order. The "weighted-largest" algorithm has much more even results.

Example: Uniform value distribution

Values:
[
100, 100, 100, 100, 100, 100, 
200, 200, 200, 200, 200, 200
]

Like the very first trial with all timings equal to one another, both algorithms will balance exactly the same in this case.

Conclusion from the initial trials

These results definitely show the "weighted-largest" algorithm to be much more optimized in terms of balancing files appropriately. Not only are each runner more evenly distributed in timing, but the longest runtime is kept as low as possible as well. However, one thing to consider is that a lot of these values do not have much variance from one another; most are in steps of 10 or 100 away from each other. So, I decided to try this with some real-world results.

Examples: Real world trials

130 component tests

Skipping showing the values as there are too many, but the longest running file takes 3 minutes, and most take under 10 seconds. There is a greater variance of timings across all files in this example.

For this trial, there were 130 tests across 6 runners. There is a stark difference between both algorithms, with the "weighted-largest" being the more effective solution here. All of its jobs take under 4 minutes, as compared to the longest job of the other algorithm being over 9 minutes!

45 end-to-end tests

Values (in minutes):
[
  0.01, 0.01, 0.01, 0.01, 0.01, 0.06,
  0.14, 1.22, 1.31, 1.76, 1.81, 2.01,
  2.22, 2.31, 2.32, 2.32, 2.42, 3.15,
  3.19, 3.28, 3.32, 3.41, 3.68, 4.33,
  4.86, 5.09, 5.1, 7.05, 8.37, 9.01,
  9.99, 10.06, 11.95, 12.75, 13.18, 13.39,
  15.69, 17.21, 19.19, 20.03, 20.62, 20.75,
  22.45, 45.08, 46.21
]

The end-to-end tests have a lot more variance between file timings. The longest running test has a median runtime of over 45 minutes!!

In this trial, I used 16 runners as this is more appropriate in an actual organization. The great thing is that the "weighted-largest" was able to keep that single file in its own runner, and all other runners were kept lower than it. I believe this to be a huge success.

Additional considerations and final conclusions

This exercise was really refreshing to me -- it was great to actually see improvement over a common solution by readdressing the actual goal. For static processes, it makes much more sense to balance all runners to stay below the limit of the largest runner, since ultimately the total execution time is dependent on the total runtime of any one of its parallel processes. The "weighted-largest" algorithm feels much more fitting as a solution here.

As a final note, I wanted to compare the results versus the "cypress-split" package as well, so I took the component tests and split them amongst 6 runners there as well. The results are nearly identical across both packages!

If you'd like to add this package to your Cypress suite, see it here!

https://github.com/hammzj/cypress-load-balancer

How I perform code reviews

Zachary Hamm — Tue, 01 Jul 2025 18:47:09 +0000

In the distant past, I had been a bit too quick with how I checked pull requests: I’d read the description and then look over each file in order as presented in the changelist. An approach like that is only partially effective. Over time, I’ve learned how to collect my thoughts and separate items by importance. A certain conclusion must be drawn when conducting a code review: “The system can accept these changes as currently presented, and we can handle any complexity that may arise from them. Therefore, I approve.” To get to this state, I put together a little guide to organize reviews and become more efficient with them.

When requested for a code review, I manage it under three principles in this order: context, affordability, and standardization. If I find issues that require new changes, I let the developer know and only continue to the next stage if any additional suggestions would remain applicable. Let’s look at each below.

Context

Context deals with the meaning behind the changes proposed: "Can I understand the purpose from how code translates into written requirements?" Better yet, does the code correctly translate into a description of them? This is the first thing I look for when beginning a review. It is most useful for anyone who has knowledge of the system but needs to understand the reasoning for new changes.

Discovering the context will communicate the “chain of command” within the code: I can see how each changed file connects to one another, from high-levels down into deeper system logic. This helps detect glaring problems or “miscommunications” very early on. If any exist, then the review can be pushed back early to the developer as it likely means the code isn’t doing what it says it should be doing.

How do I try to discover the context? Start reading the files closest to the user or end state of the application, and move downwards to files closer to the internal system logic. A change is more likely to have a more noticeable impact the closer it is to the user or end process. As I trace the logic from those higher order functions down to lower order ones, I can map out the changes more effectively. Context is drawn from seeing how the technical implementation translates between its written communication of changes.

Affordability

It's easy to think that affordability means the monetary cost...but not quite. Affordability deals with total cost of maintaining the changes in the long term, including everything from data collisions, performance, security, scalability, and quality. It limits overall complexity by identifying risks. “How much time and energy do we need to spend on corrections if this code behaves incorrectly?”

Understanding the cost distinguishes those who can anticipate future issues. They can stay one step ahead of the system by having greater knowledge of the full architecture and language paradigms. While I may not have deep knowledge of every possible impact, I can manage what I do know. This principle is necessary for everyone to learn, as it lessens knowledge gaps and opens discussion around possibilities for decay. It takes a village.

Balancing affordability also means finding compromise with the code. It is not about finding things to prevent approval, but instead, is finding solutions that limit unexpected behaviors. Sometimes follow-up work is required when the feature is more important than the assumptions. Sometimes, the work requires additional shaping to be safe. Ask yourself, “Can we afford to manage the identified risks?”

How is affordability uncovered? This part of the process is difficult since it requires delicacy with sufficiently understanding the changes to the system, but can be summed up with few small statements: Understand the context first. Study the system. Read up on the languages used and their own paradigms. Map out the changes the architecture, low to high level. Learn about consequences of any new implementations and dependencies.

Lastly, check for tests as they protect against undesired costs and report on unwanted behaviors. Tests document your system and ensure that previous behaviors remain stable. Advocate for quality, so even when you don't know the entire system, ensure an agreeable level of testing occurred against the new changes, and call out untested behaviors.

Standardization

These checks are the most noticeable within reviews, since standardization deals with agreements on following established code patterns and paradigms. Most standardization issues can be automatically detected through tools like linters and type checkers. However, suggestions here can get a bit opinionated when tools or prop standards for patterns are not already established for the system.

For instance, I may find that a functional programming approach may work better than using long for/each loops, or that case statements are easier to read over complex if/else blocks, or that certain method names are awkward. Others may have differing opinions.

This is why standardization is the least important part of my pull request review: it can be automated or is initially agreed upon as a "contract" when making a change, but anything else depends on managing opinions. Having written prop standards removes bias and reduces variance in the code: in essence, they're similar to requirements themselves.

After I understand the context and affordability, I then scan the files more thoroughly for standardization quirks. Cleaning up code is a chore, or task, that can be done once the higher priorities are discussed.

I still find this important to do, though, as having code that can look the same throughout the suite makes it easier to manage, read, and document. It creates a visual language that describes the system. Variance and customization should be necessary when standards do not exist for describing a given code style, or other approaches have been found to be less effective.

Conclusion

One last note is that I don’t need to end a review as soon as I find an issue with the context or affordability, unless it is considerably large and may require reorganization. Instead, I will still read over and try to clean up or make suggestions along the way. Some suggestions don’t need to hold up an approval, either, so it is wise to make calls based on what is necessary and what is nice-to-have. I’ve been trying to adapt conventional comments a bit more, but without them sounding too sterile. It’s ok to leave a bit of personality in your comment style.

Pull request reviews come down to looking at these three principles in order: defining the context behind the changes proposed and how they translate into a written description, the affordability of the total cost of maintaining the changes in the long term, and ensuring standardization on following established code patterns and paradigms. Doing this has helped me become more efficient in understanding the system through the changes made to it.

I'm always curious how other people handle code reviews. Please share your approaches as I am happy to answer!

Compare changes to encrypted files without revealing secrets in a GitHub Actions pull request workflow!

Zachary Hamm — Wed, 09 Apr 2025 18:03:02 +0000

Recently I've worked on something I can describe as a very fun solution that has relieved much stress.

On a project with frequent updates to encrypted environment files, it became tedious to review pull requests with changes to these files. We needed to manually verify changes locally, and some of them could be missed since GitHub can't show diffs on encrypted files. As a remedy, I thought, "why not just diff which keys were changed within the file?" And luckily, we can compare files between branches with GitHub Actions!

The workflow below will demonstrate an example of how this is possible using the exciting power of the pull_request trigger: it be set to run only when certain files are changed in a PR!

Problem statement:

I need to compare changes made to an encrypted file within a pull request. The file is stored as key-value pairs of information, similar to a flat JSON structure, or an .env file.

The solution:

During a pull request, use a GitHub Actions workflow to post a comment that displays which keys were changed without outputting any sensitive data.

Building a GitHub Workflow

The process

A pull request is opened that contains edits to an encrypted file, which triggers the workflow. It then will:

Check out the repository from the current (source) branch
In a subdirectory, check out the repository from the target branch (where you want to merge)
With a custom-built JavaScript command, decrypt the files using a stored repository secret, and then perform a diff on them to detect Additions, Deletions, and Modifications. More on this below!
Report back only non-sensitive key names of anything changed as a new JSON object
Share the JSON object to a bot that can post a comment on the pull request of those changes

Let's get to work!

Set up the workflow file

The trigger

For this workflow, we are going to assume changes are being made to .env files, which are text files of key-value pairs of information. They look like this:

EXAMPLE_KEY=value
MY_SECRET=abcd1234!

Whenever this file is updated and committed back to the repository, we want the workflow to run and verify what keys were changed.

Something really really cool with workflow triggers is that can be set to run against individual file or directory changes within pull requests:

on:
  pull_request:
    paths:
      - '.env.enc'

For the above, the workflow will only run when 1. the action is a pull request, and 2. a change happens to the .env.enc file at the root of the project directory.

Next, two jobs will be set up. The first will handle producing the diff based on changes to that file. The second will post a formatted comment on the PR based on the diff.

The first job: get-file-differences

This job will checkout both the current branch's full repository, and then only the encrypted environment file from the base branch. Then, it will use a custom JavaScript command to decrypt, diff the files, and then provide outputs back to the workflow.

  get-file-differences:
    runs-on: ubuntu-latest
    outputs:
      message: ${{ steps.produce-diff.outputs.message }}
    steps:
      # Use the current branch's repository to run all commands
      - name: Checkout head branch
        uses: actions/checkout@v4

      # Checkout only the file from the base branch into a new directory named `base`
      - name: Checkout file from base branch
        uses: actions/checkout@v4
        with:
          ref: ${{ github.base_ref }}
          path: base
          sparse-checkout-cone-mode: false
          # submodules & sparse-checkout allow checking out only a portion of the repository into another directory!
          submodules: true
          sparse-checkout: |
            .env.enc

      # Run a clean install of the repository
      - run: npm ci

      # This is a custom JS script that can diff the files and output the results
      # It will be described below in more detail!
      - run: npm run diff-env-files
        id: produce-diff
        env:
          BASE_DOTENVENC_FILE_PATH: ./base/.env.enc
          CURRENT_DOTENVENC_FILE_PATH: ./.env.enc
          DOTENVENC_PASS: ${{ secrets.DOTENVENC_PASS }} #Needed for decryption by dotenvenc

The second job: post-or-edit-comment

The second job will use an action that can post comments as a bot to the pull request using the action from peter-evans/create-or-update-comment. For this job, we will be adding or amending a single comment only, so if more changes are pushed to the pull request, the original comment will be updated instead of a new one published. It will wait on the first job to finish, and then use its message output to use in the comment.

  post-or-edit-comment:
    runs-on: ubuntu-latest
    # These permissions are necessary for the `create-or-update-comment` action to work!
    permissions:
      issues: write
      pull-requests: write
    needs: [ get-file-differences ]
    steps:
      - name: Find Comment
        uses: peter-evans/find-comment@v3
        id: find-comment
        with:
          issue-number: ${{ github.event.pull_request.number }}
          comment-author: 'github-actions[bot]'
          body-includes: 'Updates to ".env" file in this pull request'

      - name: Add or replace comment with newest file differences
        uses: peter-evans/create-or-update-comment@v4
        with:
          comment-id: ${{ steps.find-comment.outputs.comment-id }}
          issue-number: ${{ github.event.pull_request.number }}
          body: |
            # Updates to ".env" file in this pull request

            ${{ needs.get-file-differences.outputs.message }}
          edit-mode: replace

Here is the complete YAML file:

# ./github/workflows/diff-env-files.yml

name: Display differences between ".env.enc" files
description: |
  This annotates the current pull request with differences between the ".env.enc" files in the base and head branches.
  It output the key names that have been added, removed, and modified on the head branch as a comment on the branch.

on:
  pull_request:
    paths:
      - '.env.enc'

jobs:
  get-file-differences:
    runs-on: ubuntu-latest
    outputs:
      message: ${{ steps.produce-diff.outputs.message }}
    steps:
      - name: Checkout head branch
        uses: actions/checkout@v4

      - name: Checkout file from base branch
        uses: actions/checkout@v4
        with:
          ref: ${{ github.base_ref }}
          path: base
          sparse-checkout-cone-mode: false
          submodules: true
          sparse-checkout: |
            .env.enc

      - run: npm ci

      - run: npm run diff-env-files
        id: produce-diff
        env:
          BASE_ENV_ENC_FILE_PATH: ./base/.env.enc
          CURRENT_ENV_ENC_FILE_PATH: ./.env.enc
          DOTENVENC_PASS: ${{ secrets.DOTENVENC_PASS }}

  post-or-edit-comment:
    runs-on: ubuntu-latest
    permissions:
      issues: write
      pull-requests: write
    needs: [ get-file-differences ]
    steps:
      - name: Find Comment
        uses: peter-evans/find-comment@v3
        id: find-comment
        with:
          issue-number: ${{ github.event.pull_request.number }}
          comment-author: 'github-actions[bot]'
          body-includes: 'Updates to ".env" file in this pull request'

      - name: Add or replace comment with newest file differences
        uses: peter-evans/create-or-update-comment@v4
        with:
          comment-id: ${{ steps.find-comment.outputs.comment-id }}
          issue-number: ${{ github.event.pull_request.number }}
          body: |
            # Updates to ".env" file in this pull request

            ${{ needs.get-file-differences.outputs.message }}
          edit-mode: replace

Set up the custom JavaScript command for `diff-env-files`

Warning: because this command handles these files in plaintext, take extra caution to not log any information you do not want displayed in the pull request!

With workflow written, let's get the JavaScript command in place. It will handle decrypting the files, diffing them for Additions, Deletions, and Modifications, and setting outputs to the GitHub Actions step. The outputs are a markdown message used for posting the comment, and the JSON file created of the diffs.

Notes about the packages:

dotenvenc package will handle encryption and decryption of .env files in this example.
@actions/core contains GHA workflow helpers for JavaScript! Here is an example of the script:

const path = require('path');
const dotenvenc = require('@tka85/dotenvenc');
const core = require('@actions/core');

function performDiff(baseBranchFile, currentBranchFile) {
    const getKeyDifferences = (base, current) => {
        const baseKeys = Object.keys(base);
        const currentKeys = Object.keys(current);
        const diffs = [];
        for (const baseKey of baseKeys) {
            if (!currentKeys.includes(baseKey)) diffs.push(baseKey);
        }
        return diffs;
    };

    const removed = (base, current) => {
        return getKeyDifferences(base, current);
    };

    const added = (base, current) => {
        return getKeyDifferences(current, base);
    };

    const modified = (base, current) => {
        const baseKeys = Object.keys(base);
        const currentKeys = Object.keys(current);
        const diffs = [];
        for (const baseKey of baseKeys) {
            if (currentKeys.includes(baseKey)) {
                if (JSON.stringify(base[baseKey]) !== JSON.stringify(current[baseKey])) {
                    diffs.push(baseKey);
                }
            }
        }
        return diffs;
    };

    return {
        Added: added(baseBranchFile, currentBranchFile),
        Removed: removed(baseBranchFile, currentBranchFile),
        Modified: modified(baseBranchFile, currentBranchFile),
    };
}

/**
 * Writes the diffs of each key in a readable GitHub Markdown Format
 * @example Changes for each diff type
 * ## Added
 * - MY_API_KEY
 * - CUSTOMER_KEY
 * ## Removed
 * - MY_PASSWORD
 * ## Modified
 * - MY_API_URL
 *
 * @example Changes for additions only
 * ## Added
 * - MY_API_KEY
 * - CUSTOMER_KEY
 */
function getAsMarkdown(diffs) {
    const createList = (items) => {
        return items.map((i) => `- ${i}`).join('\n');
    };

    return Object.entries(diffs)
        .map(([key, vals]) => {
            //Only build if there are values for the key.
            if (vals.length > 0) {
                return [`## ${key}`, createList([vals].flat()), '\n'].join('\n');
            }
        })
        .join('')
        .trim();
};

async function main() {
    //base-ref (target branch) file
    const baseBranchFile = await dotenvenc.decrypt({encryptedFile: path.resolve(process.env.BASE_DOTENVENC_FILE_PATH)})
    //head-ref (source branch) file
    const currentBranchFile = await dotenvenc.decrypt({encryptedFile: path.resolve(process.env.CURRENT_DOTENVENC_FILE_PATH)})

    //Get diffs
    const diffs = performDiff(baseBranchFile, currentBranchFile);
    const hasDiffs = Object.values(diffs).some((d) => d.length > 0);

    //Add outputs to GitHub Actions workflow
    const message = hasDiffs ?
        getAsMarkdown(diffs) :
        'No differences exist between the files.'
    core.setOutput('diffs', diffs);
    core.setOutput('message', message);
}

main();

Three environment variables are needed, which can be passed in directly from the workflow:

BASE_ENV_ENC_FILE_PATH: the path to the base branch's encrypted file. This is set in the workflow as ./base/.env.enc
CURRENT_ENV_ENC_FILE_PATH: the path to the current branch's encrypted file. This is set in the workflow as ./.env.enc
DOTENVENC_PASS: this is used to decrypt the files, as required by the package dotenvenc. In my demo repository, I have it set as a repo secret.

When this script is run, no output will be displayed to the user; instead, outputs are set directly for the GitHub Actions workflow to consume.

Finally, we should add it to our package.json for ease-of-use:

  "scripts": {
    "encrypt": "./node_modules/.bin/dotenvenc -e",
    "decrypt": "./node_modules/.bin/dotenvenc -d",
    "diff-env-files": "node diff-env-files.js"
  }

Note: This is just an example -- you could separate the decryption and diff processes if needed. The main portion is just comparing a flat JSON file structure!

Bonus: Hiding sensitive keys

Let's assume you have a list of keys that, no matter if they are edited, you cannot display the key name in the output. No worries! We can edit the diffs before setting it to the output using a replacer function, and use a comma-delimited list of keys to hide provided to the process.env.

In the JavaScript command file, let's add this function:

const hideKeys = (diffs) => {
    let sensitiveKeysChanged = 0
    //@example: DB_PASSWORD,SECRET_TOKEN
    const SENSITIVE_KEYS = `${process.env.DOTENV_SENSITIVE_KEYS}`.split(',')

    Object.entries(diffs).map(([section, keys]) => {
        keys.map(k => {
            if (SENSITIVE_KEYS.includes(k)) {
                sensitiveKeysChanged++;
                const keyIndex = diffs[section].indexOf(k)
                diffs[section].splice(keyIndex, 1)
            }
        })
        if (diffs[section].length === 0) delete diffs[section]
    })

    if (sensitiveKeysChanged > 0) {
        diffs['Other sensitive keys changed'] = [`${sensitiveKeysChanged} key(s)`]
    }
    return diffs;
}

Then, in the main function, we can pipe the JSON from performDiffs to have additional keys hidden:

const diffs = hideKeys(performDiff(baseBranchFile, currentBranchFile));

Now, the diffs output will look something like this:

::set-output name=diffs::{"Modified":["MY_API_URL"],"Other sensitive keys changed":["2 key(s)"]}

These keys can be supplied as a repository secret to the command as well, so the keys won't ever need to be displayed!

The workflow in action

Here is an example of a pull request that edits the file and executes the workflow:

And here is another pull request that does not edit the file, so the workflow is not run!

Conclusion

GitHub Actions can run workflows against changed filesets in a pull request. See here.
We can compare encrypted files by decrypting them in a workflow, using a JS script to diff them, and then output a list of Additions, Deletions, and Modifications as a comment on the pull request. Additional sensitive data to hide can be specified using repository secrets!
This can work for other files, like flat JSON file structures, with some modifications.

Here is a demo repository!

Load balancing Cypress tests without Cypress Cloud

Zachary Hamm — Fri, 28 Feb 2025 20:41:59 +0000

Recently I've been asked to work on a solution of efficiently running Cypress component tests on pull requests without taking a lot of time. At first, my standing solution was to just evenly spread out the files against a number of parallel jobs on GitHub Actions workflows, but there is a big discrepancy between the slowest job and the average job times. Thus, we've been wondering if there is a smarter way of evening out the runtimes.

With that, I created a new plugin of cypress-load-balancer, which allows us to solve that problem. This plugin saves the durations of the tests it runs and calculates an average, which then then can be passed into a script; that script uses an algorithm to perform load balancing for a number of job runners.

What is a load balancer?

Wikipedia's summary is as such:

In computing, load balancing is the process of distributing a set of tasks over a set of resources (computing units), with the aim of making their overall processing more efficient. Load balancing can optimize response time and avoid unevenly overloading some compute nodes while other compute nodes are left idle.

The general approach of using a load balancer for tests

This is the basic idea of steps that need to occur to utilize results from load balancing properly. A persistent load balancing map file known as spec-map.json is saved on the host machine. The load balancer will reference that file and perform calculations to assign tests across a given number of runners. After all parallel test jobs complete, they will create a key-value list of test file names to their execution time; these results can then be merged back to the main spec map file, recalculate a new average duration per each test file, and then overwrite the original file on the host machine. Then the spec map can be consumed on the next test runs, and run through this process all over and over again.

For this tool, here are the general steps:

Install and configure the plugin in the Cypress config. When Cypress runs, it will be able to locally save the results of the spec executions per each runner, depending on e2e or component tests.
Initialize the load balancer main map file in a persisted location that can be easily restored from cache. This means the main file needs to be in a place outside of the parallelized jobs to can be referenced by the parallelized jobs in order to save new results.
Execute the load balancer against a number of runners. The output is able to be used for all parallelized jobs to instruct them which specs to execute.
Execute each parallelized job that starts the Cypress testrunner with the list of spec files to run across each runner.
When the parallelized jobs complete, collect and save the output of the load balancing files from each job in a temporary location.
After all parallelized test jobs complete, merge their load balancing map results back to the persisted map file and cached for later usage. This is where the persisted file on the host machine gets overwritten with new results to better perform on the next runs. (In a GitHub Actions run, this means on pull request merge, the load balancing files from the base branch and the head branch need to be merged, then cached down to the base branch.)

So, for Docker Compose, a persistent volume needs to exist for the host spec-map.json to be saved. It can then run the load balancing script, and execute a number of parallelized containers to run those separated Cypress tests. When each test job completes, the duration of each test can be merged back to the original file and re-calculate a new average.

For GitHub Actions, it's a bit more complex. More on that later.

How does it work for Cypress automated tests?

Installation

The current installation guide as of February 2025 is as such:

Install the package to your project:
npm install --save-dev cypress-load-balancer
yarn add -D cypress-load-balancer
Add the following to your .gitignore and other ignore files:
.cypress_load_balancer
In your Cypress configuration file, add the plugin separately to your e2e configuration and also component
configuration, if you have one.
This will register load balancing for separate testing types
import { addCypressLoadBalancerPlugin } from "cypress-load-balancer";
defineConfig({
  e2e: {
    setupNodeEvents(on, config) {
      addCypressLoadBalancerPlugin(on);
    }
  },
  component: {
    setupNodeEvents(on, config) {
      addCypressLoadBalancerPlugin(on);
    }
  }
});

Usage

Cypress tests are run for e2e or component testing types.
When the run completes, the durations and averages of all executed tests are added to spec-map.json.
The spec-map.json can now be used by the included executable, cypress-load-balancer, to perform load balancing against the current Cypress configuration and tests that were executed. The tests are sorted from slowest to fastest and then assigned out per runner to get them as precise as possible to each other in terms of execution time. For example, with 3 runners and e2e tests:
- npx cypress-load-balancer --runners 3 --testing-type e2e
The script will output an array of arrays of spec files balanced across 3 runners.

Scripts

There are included scripts with npx cypress-load-balancer:


shell
$: npx cypress-load-balancer --help
cypress-load-balancer

Performs load balancing against a set of runners and Cypress specs

Commands:
  cypress-load-balancer             Performs load balancing against a set of
                                    runners and Cypress specs          [default]
  cypress-load-balancer initialize  Initializes the load balancing map file and
                                    directory.
  cypress-load-balancer merge       Merges load balancing map files together
                                    back to an original map.

Options:
      --version                Show version number                     [boolean]
  -r, --runners                The count of executable runners to use
                                                             [number] [required]
  -t, --testing-type           The testing type to use for load balancing
                               [string] [required] [choices: "e2e", "component"]
  -F, --files                  An array of file paths relative to the current
                               working directory to use for load balancing.
                               Overrides finding Cypress specs by configuration
                               file.
                               If left empty, it will utilize a Cypress
                               configuration file to find test files to use for
                               load balancing.
                               The Cypress configuration file is implied to
                               exist at the base of the directory unless set by
                               "process.env.CYPRESS_CONFIG_FILE"
                                                           [array] [default: []]
      --format, --fm           Transforms the output of the runner jobs into
                               various formats.
                               "--transform spec": Converts the output of the
                               load balancer to be as an array of "--spec
                               {file}" formats
                               "--transform string": Spec files per runner are
                               joined with a comma; example:
                               "tests/spec.a.ts,tests/spec.b.ts"
                               "--transform newline": Spec files per runner are
                               joined with a newline; example:
                                "tests/spec.a.ts
                               tests/spec.b.ts"
                                          [choices: "spec", "string", "newline"]
      --set-gha-output, --gha  Sets the output to the GitHub Actions step output
                               as "cypressLoadBalancerSpecs"           [boolean]
  -h, --help                   Show help                               [boolean]

Examples:
  Load balancing for 6 runners against      cypressLoadBalancer -r 6 -t
  "component" testing with implied Cypress  component
  configuration of `./cypress.config.js`
  Load balancing for 3 runners against      cypressLoadBalancer -r 3 -t e2e -F
  "e2e" testing with specified file paths   cypress/e2e/foo.cy.js
                                            cypress/e2e/bar.cy.js
                                            cypress/e2e/wee.cy.js

Example on GitHub Actions

I included two workflows in the package that show how this can work for tests executed on pull requests.

Generally, here is what occurs:

Running tests on pull requests

get-specs:
- A cached load balancing map is attempted to be restored. It tries for the current target branch, then for the source branch, and if none can be found, it initializes a basic map of the files to be run.
- Load balancing is performed based on the user's input of the number of jobs to use. It outputs an array of specs for each runner.
cypress_run_e2e:
- These are the parallelized jobs that run a subset of the files obtained from the load balancer output.
- When this job completes, it produces a temporary spec-map.json of just those files, and uploads the artifact.
merge_cypress_load_balancing_maps:
- After all parallel jobs complete, download their artifacts of their temporary spec-map.json files, merge them to the branch's map file, and then cache and upload it. This is how it can be saved per branch.


yml
name: Testing load balancing Cypress E2E tests

on:
  pull_request:
  workflow_dispatch:
    inputs:
      runners:
        type: number
        description: Number of runners to use for parallelization
        required: false
        default: 3
      debug:
        type: boolean
        description: Enables debugging on the job and on the cypress-load-balancer script.

env:
  runners: ${{ inputs.runners || 3}}
  CYPRESS_LOAD_BALANCER_DEBUG: ${{ inputs.debug || false }}

jobs:
  get_specs:
    runs-on: ubuntu-22.04
    outputs:
      e2e_specs: ${{ steps.e2e-cypress-load-balancer.outputs.cypressLoadBalancerSpecs }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}

      - run: |
          yarn install
          yarn build

      - name: Get cached load-balancing map
        id: cache-restore-load-balancing-map
        uses: actions/cache/restore@v4
        with:
          fail-on-cache-miss: false
          path: .cypress_load_balancer/spec-map.json
          key: cypress-load-balancer-map-${{ github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }}
          # Restore keys:
          ## 1. Same key from previous workflow run
          ## 2. Key from pull request base branch most recent workflow. Used for the "base" map, if one exists
          restore-keys: |
            cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }}
            cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-${{ github.run_id }}-
            cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-
            cypress-load-balancer-map-${{ github.base_ref }}-

      - name: Perform load balancing for E2E tests
        id: e2e-cypress-load-balancer
        #TODO: this can eventually be replaced with a GitHub action. The executable should be used for Docker and other CI/CD tools
        run: npx cypress-load-balancer -r ${{ env.runners }} -t e2e --fm string --gha
        #run: echo "specs=$(echo $(npx cypress-load-balancer -r ${{ env.runners }} -t e2e --fm string | tail -1))" >> $GITHUB_OUTPUT

      - name: "DEBUG: read restored cached spec-map.json file"
        if: ${{ env.CYPRESS_LOAD_BALANCER_DEBUG == 'true' }}
        run: cat .cypress_load_balancer/spec-map.json

  cypress_run_e2e:
    runs-on: ubuntu-22.04
    needs: get_specs
    strategy:
      fail-fast: false
      matrix:
        spec: ${{ fromJson(needs.get_specs.outputs.e2e_specs) }}
    steps:
      - name: Generate uuid to use uploading a unique load balancer map artifact
        id: generate-uuid
        run: echo uuid="$(uuidgen)" >> $GITHUB_OUTPUT

      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}

      - name: Cypress run e2e tests
        uses: cypress-io/github-action@v6
        with:
          browser: electron
          build: yarn build
          spec: ${{ matrix.spec }}
          # Fix for https://github.com/cypress-io/github-action/issues/480
          config: videosFolder=/tmp/cypress-videos

      - name: Upload temp load balancer map
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: ${{steps.generate-uuid.outputs.uuid }}-cypress-load-balancer-map-temp-from-parallel-job
          path: .cypress_load_balancer/spec-map.json

  merge_cypress_load_balancing_maps:
    runs-on: ubuntu-22.04
    needs: [get_specs, cypress_run_e2e]
    if: ${{ needs.get_specs.result == 'success' }}
    steps:
      - name: Checkout
        uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}

      - run: |
          yarn install
          yarn build

      - name: Get cached load-balancing map
        id: cache-restore-load-balancing-map
        uses: actions/cache/restore@v4
        with:
          fail-on-cache-miss: false
          path: .cypress_load_balancer/spec-map.json
          key: cypress-load-balancer-map-${{ github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }}
          # Restore keys:
          ## 1. Same key from previous workflow run
          ## 2. Key from pull request base branch most recent workflow
          restore-keys: |
            cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }}
            cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-${{ github.run_id }}-
            cypress-load-balancer-map-${{github.head_ref || github.ref_name }}-
            cypress-load-balancer-map-${{ github.base_ref }}-

      - name: If no map exists for either the base branch or the current branch, then initialize one
        id: initialize-map
        run: npx cypress-load-balancer initialize
        if: ${{ hashFiles('.cypress_load_balancer/spec-map.json') == '' }}

      - name: Download temp maps
        uses: actions/download-artifact@v4
        with:
          pattern: "*-cypress-load-balancer-map-temp-from-parallel-job"
          path: ./cypress_load_balancer/temp
          merge-multiple: false

      - name: Merge files
        run: npx cypress-load-balancer merge -G "./cypress_load_balancer/temp/**/spec-map.json"

      - name: Save overwritten cached load-balancing map
        id: cache-save-load-balancing-map
        uses: actions/cache/save@v4
        with:
          #This saves to the workflow run. To save to the base branch during pull requests, this needs to be uploaded on merge using a separate action
          # @see `./save-map-on-to-base-branch-on-pr-merge.yml`
          key: cypress-load-balancer-map-${{ github.head_ref || github.ref_name }}-${{ github.run_id }}-${{ github.run_attempt }}

          path: .cypress_load_balancer/spec-map.json
      # This is to get around the issue of not being able to access cache on the base_ref for a PR.
      # We can use this to download it in another workflow run: https://github.com/dawidd6/action-download-artifact
      # That way, we can merge the source (head) branch's load balancer map to the target (base) branch.
      - name: Upload main load balancer map
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: cypress-load-balancer-map
          path: .cypress_load_balancer/spec-map.json

      - name: "DEBUG: read merged spec-map.json file"
        if: ${{ env.CYPRESS_LOAD_BALANCER_DEBUG == 'true' }}
        run: cat .cypress_load_balancer/spec-map.json

Merging back on pull requests

When the pull request is merged, the newest map uploaded from the source branch's testing workflow is downloaded, merged with the base branch's map, and then cached to the base branch. This allows it to be reused on new pull requests to that branch.


yml
# See https://github.com/brennerm/github-actions-pr-close-showcase/
name: Save load balancing map from head branch to base branch on pull request merge
on:
  pull_request:
    types: [closed]

jobs:
  save:
    # this job will only run if the PR has been merged
    if: github.event.pull_request.merged == true
    runs-on: ubuntu-latest
    steps:
      - run: |
          echo PR #${{ github.event.number }} has been merged

      - uses: actions/checkout@v4

      - uses: actions/setup-node@v4
        with:
          node-version: ${{ env.NODE_VERSION }}

      - run: |
          yarn install
          yarn build

      - name: Download load-balancing map from head branch using "cross-workflow" tooling
        id: download-load-balancing-map-head-branch
        uses: dawidd6/action-download-artifact@v8
        with:
          workflow: cypress-parallel.yml
          # Optional, will get head commit SHA
          pr: ${{ github.event.pull_request.number }}
          name: cypress-load-balancer-map
          path: .cypress_load_balancer

      - name: Restore cached load-balancing map on base branch
        id: cache-restore-load-balancing-map-base-branch
        uses: actions/cache/restore@v4
        with:
          fail-on-cache-miss: false
          path: /temp/.cypress_load_balancer/spec-map.json
          key: cypress-load-balancer-map-${{ github.base_ref }}-${{ github.run_id }}-${{ github.run_attempt }}
          restore-keys: |
            cypress-load-balancer-map-${{ github.base_ref }}-

      - name: Merge files
        run: npx cypress-load-balancer merge -G "./temp/.cypress_load_balancer/spec-map.json"

      - name: Save merged load-balancing map
        uses: actions/cache/save@v4
        with:
          path: .cypress_load_balancer/spec-map.json
          key: cypress-load-balancer-map-${{ github.base_ref }}-${{ github.run_id }}-${{ github.run_attempt }}

And that's it! This is probably a very niche example, but the general approach should be the same:

Save a spec map on the host machine
Perform load balancing against the spec map
Run parallel test jobs organized by the list of files separated by the load balancer
Collect their results
Merge those results back to the host map and recalculate the average
Repeat!

Misconceptions with testing paradigms and how to address them

Zachary Hamm — Thu, 05 Sep 2024 15:48:19 +0000

The realm of quality is a much misunderstood one and conclusions are drawn without understanding its ability to protect a product. It should not be sacrificed in order to meet deadlines or bring that product to market -- what if it work as intended? Many of us have seen bugs appear after a release, and probably admit that there wasn't enough time to test for it. Testing is a major part of a quality plan and the workloads of many, so being able to explain why it is necessary could very well alter the opinion in favor of it.

That is why I want to present some possibly controversial statements on ideas around testing. Some of these points are not the first thing that comes to mind thinking of how testing is performed but are incredibly important to understand. I hope this article can bring you insight when the hard questions are being asked, and put the power back in the quality process.

It is separated into three parts.

The first part deals with how to run tests and what they are.

Testing is mainly used as simply checking for a pass/fail state, but it is just one part of a larger method.

Summary: Application testing is an alert mechanism for the rate of change between the system and a set of defined actions performed upon it. It is but one part of the scientific method: most kinds of software testing, however, begin with expectations rather than predictions.

Testing in itself, is just an experiment, where it executes a trial to determine if a prediction will occur based on earlier observations. It is but just one part of the scientific method (additional Wikipedia article). In software engineering, a check, or assertion, is code that ensures if certain effects from an action meet given criteria. It is usually completed with a pass or fail state. Questions are answered in the format of,

Under X circumstances, does the set of Y actions cause Z outcome to occur?

The benefit of these answers define a time when our predictions are met successfully. If, at a later point, a change causes those checks to fail expectations, we are alerted that a change caused an unhandled event In other words, a test demonstrates the rate of change between when a system's observed behavior did or did not expectations under certain conditions, as compared to measuring any possible outcomes that can occur.

Making the conclusion that a failing test found an issue in the application is inaccurate. There could very well be a bug, but there could also be an issue with the test not accounting for new change. A failing test means that the path it expected was not followed. That is why viewing them as an alert mechanism to an unhandled change is more accurate.

Take for example comparing a trial involving a chemical reaction with software functional testing. The early phases of a chemical trial may involve recording outcomes without requiring explicit behaviors. What is desired would be defined later on, based on those observations. Building an app, however, may have requirements defined at the earliest phases of development. Many types of software testing need those checks to be in place early since they begin with a relation to an known behavior, like a product requirement.

This is all for asserting upon knowable items. There are other categories that are named as "testing," but their lifecycle begins much earlier. They initially pose questions that are open-ended and outcome-oriented. Performance testing measures the ability of a system to perform under duress so it can ask questions like, “can it process data in X time when it is stressed?” Chaos testing helps examine the stability of a system to ask, “Will the system remain available if X happens to it?” Frontend accessibility testing observes behaviors to predict if the application can be fully utilized in a non-standard format.

Each of these categories demonstrate that testing is not made for just the present. When a test is added, it is made to be repeatable. If the system is behaving today, how can we know that it will behave tomorrow?

Testing a feature is an approximation of a situation.

Summary: There are many ways to reach an outcome with some being closer to how an actual situation could be observed. Therefore, one must estimate efficiently how to cause and detect it.

For example, imagine checking a theme switcher button in a web app. Our test asks, "Did clicking the button once change the theme?" There are many ways to answer this depending on which actions we take and upon what we assert. Even at a functional level, we could try

clicking the button
performing a touch action for a mobile device
invoking the click listener directly
forcing the emit of the click event

Since the underlying click actions were originally tested within their source code, we will assume that a click is the easiest path. The other ways are also correct, however. What changes is how much we "wrap" around the thing we want to test -- the more “wrapping code” we remove, the less we rely on other functionality that also needs to be correct in order to observe the effects from the true "source" of our test.

It can be even complicated when writing assertions, because they map our predictions as an answer to our question. We could check for

a data ID on an element that denotes "light" or "dark" theme
the color of an element that is affected by theme
stubbing the click listener to test if the method is called

In feature testing, these assertions are much less direct because we attempt to map them as answers to a business requirement. A requirement might state that an element's color shall be light gray or dark gray based on the mode, and another might state that a Theme element is updated when a button is clicked. We would need to answer the question with a test that best represents the user interaction.

As scenarios become more complex and integrated in highly-connected environments, the paths towards asserting an outcome become much more variable. Therefore, an engineer should the most direct way that evaluates the scenario in the context of its environment. In this case, clicking the button and checking if the color changed is probably the easiest path -- but is it the most accurate and stable? Or does another path provide better insight?

A test is arbitrary unless sufficiently described.

Summary: The title of a test needs to succinctly describe its actions or else it loses value.

Test code, like application code, is essentially grouped functions that execute a static flow. It runs our entire trial from start to finish. Yet, unlike application code, these flows are situational. Remember from earlier that a test function can be used as an alert mechanism. It is imperative, then, that test code is described succinctly in terms of what is being checked.

Unlike application code, test code needs to define its situation, or scenario. Application code has established paradigms for which it can be separated into understandable and maintainable parts, meaning smaller functions, which helps to explain each individual action with high specificity. However, because test functions are a culmination of a chain of actions, it can be difficult to ascertain what it does. Its name explains its value.

Naming a test function as clickingTheSubmitButtonLogsInAnExistingUser rather than clickSubmitTest adds succinct definition. Extracting value from clickSubmitTest is unknowable: its actions, its outcome, or anything past the notion that it is a test — it communicates little as compared to the first option.

Luckily, there are some suggestions that help define test code. The Arrange-Act-Assert pattern explains how to arrange a test, and frameworks like JavaScript’s Jest or Mocha, Ruby’s RSpec, and Python’s Pytest allow for organizing tests into contexts. These contexts build a hierarchy of explaining each function at a human-readable level (assuming the tests are written to be, well, human-readable). BDD frameworks, like Gherkin, can describe both a scenario and its actions in human-readable format, and require a higher degree of using effective language.

The second part here deals with where tests are run. Everything deals with an environment is — for a test, it is the place that provides the conditions of its initial state — our “control” situation.

Higher degrees of manageability is directly related to greater assumptions of reality.

Summary: Testing closer to the source code involves assumptions of a situation that may not match what actually happens.

Let’s start by stating that with more externalities comes more possibilities to assume. That means that testing closer to the source leads to less variability and more reliable tests. The highest degree of control is over what you can change directly.

Moving up in environments, a test’s initial state becomes more variable. Each external item introduced is incorporated into the initial state for our test. These include other systems and applications, but also the amount of users allowed to access them. Each user also has their own unique circumstances that form the conditions surrounding their own environment. When testing, we assume parts of the initial state to be accurate without actually knowing that they are. Even unit testing on one system may match a second system, since they are not the same environment themselves.

Now, as external items are integrated, that degree of control is reduced and variability is heightened. Because we normally cannot directly control any external items outside of the application, we must make assumptions about their behaviors. User permissions, connectivity issues, bugs, and system availability limit our control over them. We may be able to indirectly influence them through injecting data and other flows that set them in a wanted state, but cannot force an outcome. If we can influence it into a wanted state, then we assume it works as expected, and thus, we regain some control over our test. This is not always be possible, though.

Finally, production is just another environment where our application exists. Because what matters is where a user interacts with it, a user’s environment is an extension from our understanding of it. That leads to a very crucial sub-point...

Sub-point: Production is not reality.

Summary: Reality is the situation in which an end user observed the behavior of the application. The only way to measure that situation is by approximating it. Production is just a name for a place that allows the application to be available to the general public, but is just a layer within the user's own environment.

“Production” is just the name of an environment where we’ve agreed to release the application to the greatest set of users: the public. They could be either human users or other systems whose actions alter reality. However, production is only an abstraction of reality, because only the environment in which a user interacts with the application will produce outcomes that affect others. Production makes the assumptions that still allow for expected system behavior like any other environment below it, yet, does not define the user’s environment. Our production application is influenced by everything within the user's environment.

A medical drug could be effective for 99 percent of people, but cause side effects in that last percent. A drug produced does not cause effects until it is taken by a person. Compared to a software production environment, having a full test set may catch those "side effects" that happen for our one-percent of users. We make assumptions that the most reliable parts of the system (meaning, the parts with the most trusted tests) should work for any user.

When a software bug comes in raised by a customer, then the only way to determine how it was caused is to approximate how that person observed its occurrence. We can only ever approximate a scenario within an environment because we did not observe it originally. Thus, we estimate as close as possible to what actually happened.

For a web app, this means we attempt to account for the conditions that created the initial state of a user's observation: their browser type, network connectivity, timings, physical location, device hardware, etc. But again, production is not necessarily the user’s environment. Production is an abstraction of one because it exists as only a single layer there. Thus, if a user encounters an issue, then that user experienced it in their own version of reality constructed with their own conditions, actions, timings, and assumptions.

We may not be able to encounter the bug even when executing those same actions, unless their initial state can be mimicked. If not, then we did not make accurate assumptions about their environment. Being unable to recreate an issue does not make it less real, but instead means we cannot define its situation.

Production is not the end result. It is where “public” users can experience it within the conditions of their environment.

Integrating external systems assumes a level of trust has been met.

Summary: Integrating a system involves trust that it acts as described by its maintainers. Testing is a commonly agreed method for which trust can be derived. Thus, utilizing a system has an implicit agreement where the engineer believes that a certain level of trust has been fulfilled.

If we involve an external system for which we do not directly test, then we agree that it works for our use cases. Each integration of our application to a higher environment means we need to account for a greater number of systems for which we cannot directly control. That also means we trust in the functionality of those systems because their expectations meet their descriptions.

There are many ways trust can be derived, with some ways being more transparent than others: the developers of the application could say it works without making their code public, a third party can use it and give it a stamp of a approval, or I could use the application directly and check for how I want it to work. However, each of these ways includes executing trials, recorded or not. If no tests or trials are performed, then how can one predict what will occur in reality? Since test code must use application code to make a prediction, wouldn’t it be an independent entity from the application?

If we agree that test code forms trust surrounding the functionality of the system, we also agree that the test code can form a basis of trust in itself. Its inherent purpose is to conduct trials and alert the results, so passing tests means that we expect its trials to be conducted properly.

Sub-point: Test code may be considered an independent entity from the application.

Summary: Test execution code is considered as trustworthy and accurate when it performs its duties to ensure stability of a system — thus, when effective, it could be considered an an external system for which trust is derived from it in itself.

Being able to see the results of other engineers' tests builds trust since we agree there is no need for additional testing of that system. But we also did not write the test code, and running the test may not explain how it was conducted. What if they are inaccurate?

Test code is implied to be correct and carries a degree of honesty with it. Application code itself does not validate its own functionality alone. To write tests that make correct predictions about that functionality means that the writers are trusted to be honest, since we may not see the actions taken in the test. The test code is a third party that has no bond with the application, and also exists as an external system itself because it both requires and affects the application.

Unreliable tests, unaccounted situations, and misleading test titles lessen that bond. Undefined functionality and scenarios are understood to be hard to conduct, but core functionality as described by its maintainers must be thoroughly tested for well-defined situations. Anything less can harm the integrity of the owners of the software. Writing tests to always pass, regardless of its trial, is deceptive behavior. But at that point, the external system is untrustworthy because it fails to explain the behavior of the application it supports.

I always like to ask the question, “who tests the tests?” It's hard to do -- the chain of tests would grow forever. The test code has to be checked itself to be correct. Unreliable functionality may not mean the test writer is dishonest, but it does mean that the application is difficult to trust. There may be real reasons why certain scenarios are missed — unable to be recreated, external systems misbehaving, or the application’s behavior not being well-defined enough to design a test trial. A test’s value is determined as a cost-relationship — how much value do its checks provide versus its complexity to maintain? It is subjective and hard to calculate.

This third and final part deals with the who and when of running tests. Testing is just one part of quality, and knowing where it fits is of upmost importance.

Everyone owns quality, but those who provide the product bear the ultimate responsibility.

Summary: While testers perform the fullest duties of quality, it is something everyone should advocate and own. The ones who provide the product to others take the fullest responsibility of communicating that the product works as intended.

It is the responsibility of everyone involved in the making of a product to ensure it works as described. Quality is owned by everyone. If quality must be restricted in order to release a product, it is not the fault of that department when faults are discovered. Those who provide the product to others must answer to any problems that arise.

This goes for letting improper tests exist as well. If there is a tester who doesn’t produce accurate or trustworthy tests, then that person will be responsible for their own actions. That person harms the product and the organization. However, letting those tests remain in place to sign off on behaviors is the responsibility of those who allow them to exist there. Tests that are missed or unable to be completed for the sake of meeting deadlines also are placed in the realm of those that allowed them to be removed.

Measuring the usefulness of quality in the present is unrealized, but constant discovery can bring future stability.

Summary: Tests need to have a specific focus, large test sets need to cover expected situations, and discoverability allows for constant attention to observing new behaviors. Alone, they are not enough, but together, they forge the path to ensuring high levels of application stability and sustainability. Testing does not complete -- is an ongoing effort.

Writing a test to pass for the present is great! We need to know it works. However, writing tests to make sure those checks are maintained upon future changes to an application is necessary. Each change to the base application brings in uncertainty, much like adding external sources. If test code is meant to be trustworthy, a test producing a consistent and reliable check creates that trust. A failing test means an unexpected change was detected, and that it.

Adding more tests means making extra predictions (or, business scenarios), but does not inherently creates reliable coverage. A combination of multiple test disciplines carries more trust than a permutation of a single type of test. There needs to be a focus placed — an intention defined — or we cannot understand what we evaluate.

However, a test’s cost is determined by how accurate it answers a prediction against its complexity to maintain.

Designing scenarios for the present that are executed in the future are great, but become static — they only check for what we know already. They can never account for new behaviors unless new tests are created. Thus, testing is not static — it is ongoing as a source of discovering new scenarios, new behaviors, new outcomes. Just because it works as intended for us does not mean it works as intended for others.

Choosing to skip testing for the present may mean deadlines are hit, but the effects of it will cost more over time. A bug found tomorrow is an issue not fixed today.

Quality is a combination of many facets, and testing is a major portion. Quality is always important as it does not end at the application’s release to “production.” It does not end with the user. It continues for as long as the application exists. Therefore, always continue to test as if its users require greater trust.

Conclusion

Testing is mainly used as simply checking for a pass/fail state, but it is just one part of a larger method.
- Summary: Application testing is an alert mechanism for the rate of change between the system and a set of defined actions performed upon it. It is just one part of the scientific method: most kinds of software testing, however, begin with expectations rather than predictions.
A test is arbitrary unless sufficiently described.
- Summary: The title of a test needs to succinctly describe its actions or else it loses value.
Higher degrees of manageability is directly related to greater assumptions of reality.
- Summary: Testing closer to the source code involves assumptions of a situation that may not match what actually happens.
- Sub-point: Production is not reality.
- Summary: Reality is the situation in which an end user observed the behavior of the application. The only way to measure that situation is by approximating it. Production is just a name for a place that allows the application to be available to the general public, but is just a layer within the user's own environment.
Integrating external systems assumes a level of trust has been met.
- Summary: Integrating a system involves trust that it acts as described by its maintainers. Testing is a commonly agreed method for which trust can be derived. Thus, utilizing a system has an implicit agreement where the engineer believes that a certain level of trust has been fulfilled.
- Sub-point: Test code may be considered an independent entity from the application.
- Summary: Test execution code is considered as trustworthy and accurate when it performs its duties to ensure stability of a system — thus, it could be considered an an external system for which trust is derived from it in itself.
Everyone owns quality, but those who provide the product bear the ultimate responsibility.
- Summary: While testers perform the fullest duties of quality, it is something everyone should advocate and own. The ones who provide the product to others take the fullest responsibility of communicating that the product works as intended.
Measuring the usefulness of quality in the present is unrealized, but constant discovery can bring future stability.
- Summary: Tests need to have a specific focus, large test sets need to cover expected situations, and discoverability allows for constant attention to observing new behaviors. Alone, they are not enough, but together, they forge the path to ensuring high levels of application stability and sustainability. Testing does not complete -- is an ongoing effort.

DEV Community: Zachary Hamm

Optimizing a load balancing algorithm to minimize the runtime of a static process

The issue with using a "round-robin" approach in this case

Defining a new solution

Initial observations of the Cypress run command

Defining requirements of the new algorithm

Requirements for the load balancing function

Requirements of the algorithm

The "weighted-largest" algorithm in depth

Examples: Comparisons of different file timing distributions

Example: Every file has the exact same timing

Example: Bell curve distribution

Example: Extreme high-end distribution

Example: Extreme low-end distribution, version 1: total sum of low values equals single high value

Example: Extreme low-end distribution, version 2: total sum of low values is greater than single high value

Example: Extreme center distribution

Example: Extreme "ends" distribution

Example: Uniform value distribution

Conclusion from the initial trials

Examples: Real world trials

130 component tests

45 end-to-end tests

Additional considerations and final conclusions

How I perform code reviews

Context

Affordability

Standardization

Conclusion

Compare changes to encrypted files without revealing secrets in a GitHub Actions pull request workflow!

Building a GitHub Workflow

The process

Set up the workflow file

The trigger

The first job: get-file-differences

The second job: post-or-edit-comment

Set up the custom JavaScript command for diff-env-files

Bonus: Hiding sensitive keys

The workflow in action

Conclusion

Load balancing Cypress tests without Cypress Cloud

What is a load balancer?

The general approach of using a load balancer for tests

How does it work for Cypress automated tests?

Installation

Usage

Scripts

Example on GitHub Actions

Running tests on pull requests

Merging back on pull requests

Misconceptions with testing paradigms and how to address them

Testing is mainly used as simply checking for a pass/fail state, but it is just one part of a larger method.

Testing a feature is an approximation of a situation.

A test is arbitrary unless sufficiently described.

Higher degrees of manageability is directly related to greater assumptions of reality.

Sub-point: Production is not reality.

Integrating external systems assumes a level of trust has been met.

Sub-point: Test code may be considered an independent entity from the application.

Everyone owns quality, but those who provide the product bear the ultimate responsibility.

Measuring the usefulness of quality in the present is unrealized, but constant discovery can bring future stability.

Conclusion

Initial observations of the Cypress `run` command

Set up the custom JavaScript command for `diff-env-files`