Lockfile merge conflicts, how to handle it correctly?

During a frontend development project, a new npm package was added, and conflicts were encountered when merging the lockfile.

Lockfile refers to the lock file of package management tools, such as package-lock.json, yarn.lock, pnpm-lock.yaml.

Manually resolving conflicts can be highly inefficient and error-prone. Here are several commonly used solutions:

1. Delete the lockfile and reinstall the dependencies later.
2. Reset the lockfile to that of one branch and reinstall the dependencies later.
3. Run the dependency installation command to leverage the built-in mechanism of the package management tool to resolve lockfile conflicts.

Option 1 will result in the loss of lock records and is generally not preferred.

Are options 2 and 3 feasible? What are the considerations? This article will discuss these questions and provide the best practices at the end.

If you prefer to skip the details, you can scroll to the end directly to see the "Best Practices."

Before we dive into these questions, let's first understand the situations in which lockfile merge conflicts can occur.

Reasons for Lockfile Merge Conflicts

The reason for conflicts during a Git merge is when two branches make modifications to the same region of a single file. If the modifications are in different regions, Git will attempt an automatic merge (auto-merge) to resolve the conflict.

If you're not familiar with resolving Git merge conflicts, you can refer to the article How to Resolve Merge Conflicts in Git – A Practical Guide with Examples.

The key to merge conflicts is changes in the same region. Let's take the dependency configuration in package.json as an example. The first two examples below illustrate a conflicting scenario and a non-conflicting scenario, respectively.

For lockfiles, changes in the same region typically occur in the following two situations:

• Incompatible merging due to differences in the structure of the lockfile caused by inconsistent versions of the package management tool. To address this issue, it is recommended to maintain consistency in the package management tool's version. You can refer to my other article: Why is it so hard to maintain consistent frontend dependencies? - 掘金.
• Both branches have made changes to the dependency configuration in package.json and modified the same region in the lockfile.

When does the same region of the lockfile get modified?

This is a complex issue as different package management tools and their versions have different generation strategies. It is difficult for developers to avoid conflicts by adjusting the writing style of package.json. Therefore, it is not necessary to focus too much on this aspect. The important thing is to know how to resolve the issue when it occurs.

Next, we will discuss two commonly used resolution strategies.

Analysis of the Solutions: Resetting Branch Lockfile

TL;DR: Resetting the branch lockfile means restoring the lockfile to the version from the target branch or the current branch, which also means losing some lock records from a branch and may lead to errors. This issue is challenging to resolve completely and can only be mitigated by improving the development process and performing necessary manual reviews.

Resetting the branch lockfile refers to "merging by using the lockfile from the target branch or the current branch" and then re-executing the dependency installation command to update the lockfile.

Three Resetting Solutions

The following three solutions can be used to easily reset the branch lockfile:

1. Ignore the conflict prompt and stash the lockfile changes, then discard those changes, indicating the continuation of the lockfile from the current branch.
2. Execute the git checkout --ours "*lock*" or git checkout --theirs "*lock*" command to automatically resolve lockfile conflicts based on the current branch or target branch.

3. Implement lockfile conflict resolution automation based on Git configuration. This requires two essential steps, both of which are necessary:

   1. Add a .gitattributes file to configure the merge strategy. Example:

     # .gitattributes
     # When there's a conflict in pnpm-lock.yaml, use the lockfile from the current branch
     pnpm-lock.yaml merge=ours
   

   1. Execute the git config merge.ours.driver true command to enable the merge driver configuration (if using the theirs merge strategy, the command should be changed to merge.theirs.driver).
   2. Reference documentation:
      1. Have Git Select Local Version On Merge Conflict on a Specific File?
      2. Merge Strategies

Existing Issues

Regardless of whether the lockfile is reset based on the current branch or the target branch and the subsequent update of dependencies, it means losing some lock records and may lead to errors.

Here's an example encountered in a real business scenario:

1. From the main branch, create a development branch called feat1 with a new dependency A^1.0.0, and the installed version is 1.0.0.
2. During the development of feat1, another development branch, feat2, is merged into the main branch and adds a dependency B^2.0.0, with the installed version being 2.0.0.
3. After completing the development in feat1 and preparing to merge it into the main branch, a lockfile conflict is discovered.
4. Using the aforementioned "Resetting Branch Lockfile" solution, the dependencies are reinstalled.
5. At this time, the community releases versions A@1.0.1 and B@2.0.1 with BREAKING CHANGE. Since the "Resetting Branch Lockfile" solution ignores lock records added by feat1 or the main branch (feat2), it installs the newer versions A@1.0.1 or B@2.0.1.
6. feat1 is directly merged into the main branch, causing errors in the production code.

Solution

Some people may suggest using fixed versions when installing dependencies instead of using version ranges. For example, using "A": "1.0.0" and "B": "2.0.0" as dependencies instead of using version range notation (^).

{
    "dependencies": {
      "A": "1.0.0",
      "B": "2.0.0",
  }
}

Firstly, for application projects, using fixed versions is feasible. However, for library projects, it is not recommended to use fixed versions for the following reasons:

• The dependencies of the library project cannot be efficiently reused by dependent application projects (e.g., ^1.0.0 and ^1.1.0 can be merged into ^1.1.0).
• In the event of security vulnerabilities in the indirect dependencies of the library project, they cannot be directly fixed by reinstalling the dependencies.

Furthermore, locking the versions of direct dependencies is not a complete solution as losing the lockfile can still lead to the automatic upgrade of indirectly dependent dependencies, which can result in BREAKING CHANGE.

Therefore, once this solution is chosen, either trust that other dependencies will not have issues (leave it to chance) or perform necessary manual lockfile reviews and ensure it through a well-defined development process.

• Well-defined development process: During the development stage, promptly merge or rebase code from the main branch, resolve conflicts ahead of time rather than waiting until just before testing and deployment. Before merging after testing, if code conflicts are found, resolve them and retest the relevant code to ensure it works as expected (if project importance and manpower allow).
• Necessary manual lockfile reviews: Focus only on changes to direct dependencies (e.g., the specifier and version sections in the pnpm-lock.yaml file). It is less likely to encounter automatic upgrade errors for indirect dependencies introduced by direct dependencies (since multiple projects would be affected if errors occurred) and the review cost would be too high. Trusting the community or performing "change testing" is a viable option.

Summary of Solutions

1. Resetting the branch lockfile means restoring the lockfile to the version from the target branch or the current branch.
2. There are three solutions available to facilitate the resetting of the branch lockfile.
3. Resetting the branch lockfile leads to the loss of lock records from one branch, which can result in errors.
4. Locking the versions of direct dependencies is not the ultimate solution and can lead to other issues.
5. Rely on a well-defined development process, perform conflict testing promptly, and conduct sufficient manual reviews.

Analysis of the Solution: Package Manager's Built-in Mechanism

TL;DR: The conflicting lockfile can be resolved by parsing the conflicting file into different versions of lock objects and then merging those lock objects. Each package manager has its own merge strategy, with pnpm being the most effective overall. However, regardless of the approach, some data will be lost, and it cannot guarantee 100% error-free merging.

When a lockfile merge conflict occurs, the mainstream package managers support running dependency installation commands (npm install/yarn/pnpm install) to automatically resolve the conflict.

• npm: Resolving lockfile conflicts, supported from v5.7 onwards
• yarn: Auto-merging of lockfiles, supported from v1.0 onwards
• pnpm: Merge conflicts, supported from v5.11 onwards

It can be assumed that most users are using versions of the package managers that support this feature.

So, how do these package managers resolve conflicts?

npm's Conflict Resolution Strategy

I have analyzed this issue in another article, "A Discussion on the Algorithm for Resolving Merge Conflicts in package-lock.json".

In general, the strategy is to base the resolution on the target branch (theirs) and apply the changes from the current branch (ours).

For example, when merging a development branch into the main branch (git merge feat-branch), theirs refers to the development branch, and ours refers to the main branch. In this case, the lock records will be based on the development branch and the changes from the main branch will be applied.

In other words, if both branches update the version of the same module, the version from the main branch (ours) will be used, which may lead to errors in rare cases.

The suggested solution in the article is to reduce the impact through processes and retesting, as explained later.

yarn's Conflict Resolution Strategy

The PR for Auto detect and merge lockfile conflicts provides insight into the latest implementation in /src/lockfile/parse.js.

In fact, understanding just one line of code is sufficient:

Object.assign({}, parse(variants[0], fileLoc), parse(variants[1], fileLoc));

This shallow merge combines the YAML objects from both branches, and when the same property exists in both objects, the value from the target branch (theirs) will be used.

In other words, when conflicting versions of the same dependency are encountered, the version from the target branch is used. This is the opposite of npm's strategy, but the issues and solutions are similar.

pnpm's Conflict Resolution Strategy

The conflict resolution algorithm for pnpm is maintained by the @pnpm/merge-lockfile-changes project.

The implementation involves breaking down the conflicting parts into the contents from the target branch (theirs) and the current branch (ours), and then merging them. However, unlike yarn's straightforward shallow merge, pnpm performs a deep merge (the lockfile structure only has two levels) and selects the version with the higher version number when there is a version conflict.

Here's a demo test:

const { mergeLockfileChanges } = require("@pnpm/merge-lockfile-changes");
const simpleLockfile = {
  importers: {},
  lockfileVersion: 5.2,
};
const mergedLockfile = mergeLockfileChanges(
  {
    ...simpleLockfile,
    packages: {
      ".": {
        version: "1.1.0",
        dependencies: {
          foo: "1.2.0",
          bar: "3.0.0_qar@1.0.0",
          zoo: "4.0.0_qar@1.0.0",
          ktv: "5.0.0"
        },
      },
    },
  },
  {
    ...simpleLockfile,
    packages: {
      ".": {
        version: "1.2.0",
        dependencies: {
          foo: "1.1.0",
          bar: "4.0.0_qar@1.0.0",
          zoo: "3.0.0_qar@1.0.0",
          pua: "5.0.0"
        },
      },
    },
  }
);
console.log(JSON.stringify(mergedLockfile, null, 2));

The output will be:

{
  "importers": {},
  "lockfileVersion": 5.2,
  "packages": {
    ".": {
      "version": "1.2.0",
      "dependencies": {
        "foo": "1.2.0",
        "bar": "4.0.0_qar@1.0.0",
        "zoo": "4.0.0_qar@1.0.0",
        "ktv": "5.0.0",
        "pua": "5.0.0"
      }
    }
  }
}

In pnpm's strategy, the updated version is chosen. If there is a problem, it implies that the new version has introduced a BREAKING CHANGE. However, the probability of encountering this issue is lower compared to selecting an older version and facing a BREAKING CHANGE.

Currently, the pnpm team is continuously improving the lockfile solution to reduce conflicts, including:

1. Using different lock files for different branches: https://github.com/pnpm/pnpm/pull/4475
2. Changing the lockfile format to reduce the number of conflicts: https://github.com/pnpm/rfcs/pull/1

Additionally, pnpm provides a resolution-mode configuration where users can decide the dependency version selection strategy during installation: lowest (default), highest, or time-based (related to the last direct dependency).

Summary of the Solutions

All three solutions involve merging the lockfile, but they have different merge strategies:

• npm: Deep merge based on the current branch (ours).
• yarn: Shallow merge based on the target branch (theirs).
• pnpm: Deep merge based on the version number, selecting the higher version.

The pnpm solution has the smallest probability of encountering issues, but it is not without its potential problems, as stated in the official documentation:

It is recommended to inspect the changes before submitting, as we can't guarantee that pnpm will choose the correct head—it builds most of the updated lockfile, which is ideal in most cases.

In conclusion, while the pnpm solution has the lowest probability of encountering issues, it is not guaranteed to be error-free. As shown in the diagram above, some issues may still arise.

Best Practices

The built-in mechanisms provided by package managers result in fewer lost lockfile records and fewer issues compared to the approach of resetting the branch's lockfile. Based on the analysis above, we can summarize the best practices as follows:

1. Avoid manually resolving conflicts as it can lead to syntax errors.
2. Utilize the conflict resolution mechanisms provided by the package manager whenever possible.
3. If you have the flexibility to choose a package manager, consider using pnpm.

Additionally, it's important to note that while the package manager solutions are generally effective, extreme scenarios may still result in issues. If the project is of significant value, it is recommended to perform manual reviews and conduct regression testing. You can refer to the "Analysis of the Solution: Resetting Branch Lockfile" section for specific guidelines on how to approach these situations.

Conclusion

This article provided a comprehensive analysis of common approaches to resolving lockfile conflicts and presented the best practices. It is essential to broaden our perspectives and elevate our understanding of best practices at the software level, which can then be shared with the community.

---

Finally, if you found this article helpful, please consider giving it a thumbs up, bookmarking it, and sharing it with others. 🍻

Further Reading

• Avoid lockfile conflicts in Rush
• Thoughts on a yarn.lock conflict

Comments

[GaHing]

welcome to discuss / 欢迎讨论

[Junxiao Shi]

I put pnpm-lock.yaml into .gitignore, so there's never a conflict.
I often build libraries rather than applications, so that I should ensure my package.json contains the proper version constraints, because the dependents of my libraries would not consider what's in my lockfiles.