Consistent tests and builds by freezing npm registry states

Getting inconsistent and non-deterministic results across different environments is a problem that needs to be addressed by any dev team using JavaScript.
If left un-attended you run the risk of getting different results in different environments or worse - spend the whole QA process testing a different state than the one being built by your build server and deployed into production.

So resolving this issue is within everyone's interest.
Less confusion, better results!

Let's have a closer look at how to achieve consistent builds and tests for the JavaScript ecosystem, and along the way look at

• The solution used with Bytesafe (bytesafe.dev) for deterministic results - using a private registry and freezing registry states
• How this compares to npm's solutions for consistent results, package-lock.json & npm ci

image source https://donthitsave.com/

Disclosure:
I am a member of the team behind the service Bytesafe that offers package management for teams and lets you take of your software supply chain with private npm registries

The challenge - achieving deterministic results across environments

Getting conflicting results is frustrating and sometimes you just can't understand how something can differ between environments.
In reality though, such inconsistencies usually originate from some difference in the package versions available in the different environments.

The problem when using JavaScript is that this scenario can easily occur as part of the normal workflow. Even when the same actions have been performed in two different environments!
This is due to the nature of how dependencies and versions are handled with npm and how the timing of when you add your dependencies affect the exact versions you receive.

This problem is further accentuated by the extensive use of open source libraries in the JavaScript ecosystem.
It's not uncommon for a top-line project to have hundreds of dependencies, either direct (included in package.json) or transitive (dependency of a direct dependency).

Essence of the problem - Timing and how it comes into play

Most dependencies receive regular updates and over time it gets increasingly difficult to guarantee that each environment is using the exact same package versions without using some tooling for this.

Consider the scenario below where your team is finalizing development of a project and one of the project's dependencies receives multiple updates during its duration:

Waterfall-ish example of package versions changing over time

• Development - When you initialized development one of the dependencies was available from the public registry with version 3.1.1.
• QA / Test - When the project is ready for final testing, a new compatible patch version is available, 3.1.2
• CI/CD - When the project is pushed to build servers, a compatible minor version, 3.2.0 has been released.

Normally, a project's dependencies are listed in its package.json file with the compatible ( caret (^) ) or the approximate ( tilde (~) ) version of a dependency instead of the exact version. Implying that any compatible version of the module can be used.

So for the scenario above, unless preventative actions are taken to avoid differences in package versions for the different environments and project phases, it is very likely that there will be differences in dependency versions. Especially as the versions 3.1.1--3.2.0 in the example above were compatible.

Now this could go two different ways:

1. The difference in dependency versions made no difference, package works and all is well, or...
2. The changed package dependencies alter your application in some way that you have not seen yourself.

If 2 is the outcome, worst case you run the risk of breaking your application as you build with an untested dependency.

So how do you introduce consistency and deterministic results into the JavaScript world?

Npm's solutions revolve around using package-lock.json and npm ci.

The downside of this solution is that it depends heavily on developers' knowledge on how to use these features to be effective.

Bytesafe's solution takes a different approach.

By using a private registry and freezing registry states, we let the registry control the flow of packages. Leaving the regular workflow unaffected and removing the knowledge barrier for specific npm commands.

Freezing registry states with Bytesafe policies

Our idea for using Bytesafe and freezing registry states in addition to relying on npm's toolbox of features is to solve some additional issues:

• Make consistent package installations independent of users' knowledge level
• Moving overall responsibility for packages versions to the ones responsible for maintaining the registry - be it DevSecOps, knowledgeable developers etc.

Additionally our goal with Bytesafe has always been to simplify the workflow when using JavaScript.

So for this we've developed the Freeze Policy:

Simply put, by enabling the freeze policy for one of your Bytesafe registries, it freeze's the registry state preventing push or pull of new package versions into a registry.

So how does Freeze and private registries introduce consistency?

As Bytesafe supports multiple private registries, teams now have the option of creating registries for each scenario.
This allows a registry to be tailored to the exact needs of a project or a specific sprint.

Combining this with the read-only state of the freeze policy allows you to have complete control over the packages, what versions are used and the state of the registry is preserved with no changes allowed.

This fulfils a need of consistency and freezing registries can be applied for different phases of the project cycle:

• Before / during development phase - control the package versions used for the entire project life cycle
• Before QA / testing phase - make sure tests are performed using the same package versions as was used during development
• Before build phase - make sure builds are consistent and use the same packages that were used for testing.
• After project completion - preserve final state to test and analyse or clone for future projects

So effectively you are removing a lot of obstacles from the individuals that neither want or have the know-how of how to use the tools npm offers.
You do this by shifting responsibility from shared responsibility to a person curating a registry for a unique scenario so that you can achieve deterministic results across environments.

Freezing registry state by example: Team preparing dedicated registry for a new sprint

With a well put together and frozen registry, all users will be guaranteed to only be using the same versions when installing packages from the new registry.

1. Create new registry or clone from existing registry (re-using packages and configuration)

2. Add the modules needed for the sprint

3. Enable freeze policy when registry contains the required packages and at the desired time

If packages need to be added during the sprint, the registry can easily be un-frozen for as long as required, by the persons whom are responsible and have access to do so.

After the sprint completion: registry can be archived for future use, still with freeze policy enabled to ensure that the registry keeps the state is consistent.

Setting up registries with purpose

A comparison with npm's solution

Npm offers two distinct solutions for this problem and both aim to add consistency:

• package-lock.json - exact state of a generated dependency tree. Primary lockfile created and used by npm client
• npm ci - clean install npm cli command meant for use in build and test environments (instead of npm install for more consistent results)

Characteristics of package-lock.json:

• Representation of the node_modules tree as it looked when the file was generated, with exact versions of dependencies
• Lockfile is created and updated by the npm client each time you add (install) a module to a project
• package-lock.json supersede's package.json as source for project dependencies (when available) for npm install

Characteristics of npm ci:

• Requires an existing package-lock.json file
• Does not alter the state of either the package.json or package-lock.json files (unlike npm install)
• Compares dependencies between package.json & package-lock files, if any discrepancies are found it exits with error.
• If node_modules exists, it will delete this folder and contents
• only works for complete installations

$ cat package-lock.json
...
   "dependencies": {
      "some-pkg": {
                   "version": "2.1.0"
                   "resolved": "link to registry source"
                   "integrity": "sha512-hash"    
            },
...

Potential issues with the options npm offers?

On paper npm's solutions should also solve the problem, right? But then why do so few developers understand and use package-lock and npm ci? Why is package-lock's perception among many developers that it causes more issues than it solves?

Let's look at some reasons why your team may not want to use these solutions exclusively:

• Lack of knowledge amongst developers - to some extent npm's biggest issue is that they have too many similar commands and features. Most developers are unaware of npm ci and the ins and outs of each lockfile is not widely known.
• Transitive dependencies - Older versions of npm have limited functionality to manage indirect dependencies
• Lockfile needs to be commited to VCS - even when you've done no others changes to a projects code base
• Merge conflicts - package-lock is notoriously difficult merge commits of due to its structure. Often results in discarding current versions and generating new file instead

Lets review: Does freezing the registry state solve the problem?

So, does adding a Bytesafe private registry and Freeze to your toolbox solve the problem (and handle some shortcomings of npm's solution)?

Consistent and deterministic results - Check!
The Bytesafe registry contains only the required packages - and it is frozen and essentially read-only - all interactions with the registry will add the exact same modules, indifferent of environment used and time of interaction with the registry.

Handle transitive dependencies - Check!
When resolving package dependencies and requesting packages from the registry, it will be restricted to the packages available in the frozen registry. As the content of the registry has been curated with reproducibility in mind it should contain ALL packages needed (including transitive dependencies).
As such, all future installs using the frozen registry will receive exactly the same versions, no matter how many indirect dependencies your project has.

Remove dependency on knowledge of specific npm commands - Check!
No need to change developer behaviour. No need to make sure that everyone make use of the lockfiles in the same way. Control is maintained by the Bytesafe registry and not by file states.

No chance for un-intended changes to dependencies, due to incorrect use of npm commands or un-committed files.

Simplify workflow - Check!
Since all users of the frozen registry use the guaranteed same versions, there should be no more merge conflicts for lockfiles (yay!).
Either you skip committing the lockfile altogether or if you commit them, the file should be identical anyways.

Closing thoughts

When used correctly package-lock.json and npm ci are powerful tools to maintain consistency over dependencies, but they do not solve the whole problem (and is subject to user knowledge of them to be efficient).

I hope you are willing to give Bytesafe a try! Either to address consistency issues, enjoy the workflow improvements or address supply chain security.

Thanks for reading!