DEV Community

Okuto Oyama
Okuto Oyama

Posted on • Updated on

Thank you Webpacker, Goodbye Webpacker

Original Post (Japanese) : ありがとう Webpacker さようなら Webpacker - クラウドワークス エンジニアブログ

Hello. I'm Okuto Oyama, an engineer at CrowdWorks. I'm on a team that leads in eliminating technical debt.

This article is replace Webpacker with Simpacker and webpack.

As an assumption, our products have been running on monolithic Ruby on Rails applications for over a decade. Currently, we are trying to break the tight coupling between front-end and back-end as part of our technical debt resolution.

I hope this is helpful for anyone who wants to separate front-end responsibilities from Webpacker!

What is Webpacker?

Webpacker is the official library for webpack front-end development in Ruby on Rails applications. The advantage of this library is that it provides a generic configuration for those who are not familiar with the webpack build system, making it easy to quickly develop modern front-ends.

For more information on CrowdWorks' implementation of Webpacker, please see our previous blog posts.

CrowdWorks流!Webpacker活用術 - クラウドワークス エンジニアブログ

Why quitting Webpacker?

I have decided to quit Webpacker, and I will give the reasons why I have decided to quit.

Because Webpacker was declared retired in January of this year.

  • There is no need to continue using Webpacker as the next RC version will not be officially released in the future.
    • There is an option to use Shakapacker
    • However, since we plan to move away from Rails-dependent front-end development at CrowdWorks, this is not the right choice for us.

Because the modules that depend on Webpacker are hidden.

  • The internal Webpacker dependencies are hidden, making it hard to see what is being used.
    • Babel settings were not included in CrowdWorks package.json, causing some build failures.
    • Debugging itself is not easy.
      • I tried the v14 series of stylelint postcss must be included, but even if the lint passed, the webpack build failed.
        • I wanted to find the cause, but at first glance I couldn't figure out what was wrong with the configuration.
    • By migrating to bare webpack, we can think about the configuration ourselves.

Because it is hard to do performance tuning

  • Not only webpack, but also Rails assets:precompile builds are done by CI.
    • We are extending resource classes because CI and deployments are failing due to the long time it takes.
    • Not enough fine-tuning of webpack.
    • It may be possible to reduce this problem by updating Webpacker itself.
      • The webpack included in the stable version of Webpacker used to be v4.

In light of the above, we decided to eliminate the use of Webpacker itself and allow the front-end responsibilities to be handled by the front-end.

Incidentally, CrowdWorks would like to migrate to a front-end framework in the future, and if everything can run on that framework, the removal of Webpacker itself will be unnecessary. However, since the transition to a framework is not yet in sight, we have decided that using webpack is the right way to maintain the status quo.

Organizing what to do

First of all, when I quit Webpacker, I organized what I was going to do. There have been several articles about quitting Webpacker in the past, so I used them as a reference.

I mainly referred to pixiv's article and iCARE's article. I would like to take this opportunity to thank them.

  • webpack configuration
    • Migrate Webpacker DSL (config/webpack) to webpack.config.js
    • Install necessary plugins and modules.
    • Configure webpack-dev-server
  • Remove Webpacker
    • Remove from gem
    • Make config/webpacker.yml obsolete
  • Implementing webpack view helpers in Rails
    • Create helper code
    • Create test code
  • Docker, CI configuration changes
    • Incorporate webpack build mechanism into CI
    • Adding webpack processing to docker-compose configuration
    • Staging CI checks
    • Checking docker-compose behavior
  • Checking the behavior of docker-compose in local staging
    • Local screen checks (webpack build-related screens only)
    • Screen check at staging (only screens related to webpack builds)

Examine the Webpacker implementation

The first thing I did was investigate the Webpacker implementation.

First, I tried to output the Webpacker configuration to get a rough idea of how the whole thing works.

The way to do this is quite simple. I put the logs in console.dir(webpackConfig, { depth: null }); at the end of module.exports in each environment's DSL file placed in config/webpack. I have tried this. I can't go into the details of the function-based code, but this gave me a rough idea of the general shape of the code.

The following is the webpack configuration included in Webpacker.

summary
  • entry
    • Configuration of files to be loaded by Webpacker.
  • output
    • Configuration of the file to be built and exported.
  • resolve
    • Configuration of how modules can be resolved.
  • resolveLoader
    • Configure only name resolution for webpack loader package.
  • node
    • Configures whether certain Node.js globals and modules can be polyfilled or mocked
      • Allows code written for the Node.js environment to run in other environments, such as browsers.
  • loaders
    • webpack only understands JS and JSON. Therefore, the part that converts other files so that they can be deciphered.
      • Things like xxxx-loader are the equivalent.
  • plugins
    • webpack plugin settings
  • mode
    • Development environment settings
    • You can specify production, development, etc.
  • devtool
    • Source map generation and how it is generated.
  • stats
    • Settings to control bundle information displayed.
  • bail
    • Settings to fail when an error occurs.
  • optimization
    • Settings related to optimization execution

The next step is to investigate the detailed Webpacker implementation based on the results.

https://github.com/rails/webpacker/tree/5-x-stable

The version of Webpacker used at CrowdWorks was version 5, so we checked the 5-x-stable branch. The part I was interested in was webpack.config.js, so I just looked at its source.

  • webpacker/package/environments/ ... Environment settings. It contains basic webpack settings.
  • webpacker/package/rules/ ... List of rules for loaders. Only refer to the ones you use.

You should also check the required package.json, babel.config.js, postcss.config.js.

Compare build results between Webpacker and webpack alone

I experimented to see if Webpacker can be built by itself while referring to the internal implementation.

In CrowdWorks, the front-end tools (such as Storybook) are in a separate hierarchy of directories, so I installed a new one there and set up webpack to reference the front-end code in the application.

After confirming that the build went through, I compared the differences between the results built in the staging environment and the results built with webpack alone.

Since CrowdWorks uses CircleCI for our application build pipeline, we saved the built files in artifacts.

- run:
  name: Compress Artifacts
  command: tar -cvzf assets-precompile.tar public/packs
- store_artifacts:
  name: Uploading artifacts - store results of assets:precompile
  path: assets-precompile.tar
Enter fullscreen mode Exit fullscreen mode

We reviewed the missing configuration, the number of files, and the differences in the contents of the files, and finally, we were able to get the files to almost match 1.

Incorporate webpack build results into your Rails application

After confirming that the build results are OK, the next step is to incorporate them into the Rails application.

Webpacker uses webpack-assets-manifest to create manifest.json, which is then delivered to View helper in Rails.

I needed to create an alternative View helper to replace Webpacker, but decided to use the Simpacker gem rather than implement the helper on my own.

Unlike Webpacker, Simpacker does not manage any configuration on the webpack side and is only involved with the path to the output manifest.json. The responsibilities are nicely divided, and the configuration itself is very simple.

I have been using Split Chunks since I was using Webpacker, so I decided to replace it by introducing the following helper.

def javascript_packs_with_chunks_tag(*names, **options)
  paths = names.flat_map{ |name| simpacker_context.manifest.lookup!("entrypoints", name, "js") }.uniq
  javascript_include_tag(*paths, **options)
end

def stylesheet_packs_with_chunks_tag(*names, **options)
  paths = names.flat_map{ |name| simpacker_context.manifest.lookup!("entrypoints", name, "css") }.uniq
  stylesheet_link_tag(*paths, **options)
rescue Simpacker::Manifest::MissingEntryError
  # If you don't extract css, you get an error, but don't return anything
end
Enter fullscreen mode Exit fullscreen mode

Make webpack-dev-server work

Webpacker also includes webpack-dev-server, so we made it work as well. Since I had Simpacker installed, I only had to change the devServer part in webpack.config.js to make it work with npm scripts.

The problem with getting webpack-dev-server to work was that the included version was a 3 series that didn't work well with the same settings as Webpacker.

Rather than investigating the cause in detail, I thought that if it was for development only and didn't affect production, I should just update it to the latest version, so I updated to the latest 4 series and migrated the part about devServer settings.

It works with this as a result. If you're looking to move on from the Webpacker 5, here's what you need to know.

Check the behavior on CI

As mentioned above, deployment of CrowdWorks is managed by CircleCI. Because it's a monolithic application, the front-end build mechanism is built into it, and we'll review the settings when we switch to webpack to make sure it doesn't fail.

Webpacker used to build webpack at assets:precompile time, but since it no longer does so, we need to add our webpack build commands.

While reviewing the configuration, I also included a workflow uploading source map files to Rollbar for the front-end error monitoring system. However, for some reason, I caught an event that prevented the uploading of the files.

After investigating the cause, I found that it was due to the --ignore-optional option that I had added when doing the yarn install.

This was because the source map files generated by webpack depended on optional dependencies, and thus were not generated properly without the required modules. Therefore, we decided to remove this option and let it install.

Ask engineers to check if it works in their local environment

Since CrowdWorks uses Docker in its development, we asked our engineers to check if it works with Simpacker + webpack.

Screenshot: communicating on a Slack channel for engineers outside of the mobile apps team about the Docker runtime process, and what to check based on that.

Docker startup and build results were fine, but we found that Hot Module Reloads in webpack-dev-server were not working properly. After that, we fixed it while checking the operation with my team.

Conduct a mob review session

Since the changes made to the PR were very significant this time, we conducted a mob review session to reduce the prerequisite knowledge and cognitive load among my team.

Screenshot: Notion article titled

Since I had been working on this project for a long time, I was able to look back on what I had done, and the causes of some parts that were unclear as to their operation were clarified. Some of the omissions found in this review have also been corrected to reflect.

Formulate Architecture Decision Records

We have been keeping Architecture Decision Records (ADR) since September of this year to keep the reasons for architecture selection.

This initiative was not in place when we discontinued Webpacker, but since we were going to adopt Simpacker in this transition, we decided to develop an ADR for it and have it reviewed by our engineers.

Screenshot: Migrating from Webpacker to Simpacker (+ webpack) ADR Documentation

Incidentally, this is the first document after the ADR initiative was introduced.

Release

It's been a long process, but now that we have all the necessary responses, we're finally releasing it. Even in the staging environment, I had verified it many times and thought there would be no problem with the release, but I had shared it with all the engineers beforehand just in case.

Screenshot: communicating on the Slack channel about the release of the work to quit Webpacker

I was very nervous about the release because it was a very big change, but after the release was completed, we determined that there were no problems with operation and no alerts and that users would not be affected, so we reported the completion and shared the operating procedures again.

Screenshot: After the release was completed, we are communicating on our Slack channel on how to reflect the new Simpacker + webpack environment.

I was in a daze for a while, maybe glad that I released it successfully...

Reflection

We look back on our efforts to quit Webpacker this time.

Good

  • I was able to complete the migration to Simpacker + webpack itself.
  • We were able to split webpack management and build responsibilities from Rails.
  • We were able to resolve the EOL situation.

Problem

  • adding their own npm packages that were included with Webpacker
    • Try ⇒ We will update and prune it
  • As the npm packages were being updated at the same time, there were many conflicts in package.json, making it difficult to rebase
    • We should have verified the module with the module itself added and released it first
    • Try ⇒ Make a mechanism to verify build diffs (to build psychological safety for future updates)
  • Migration work was being handled by @okuto_oyama alone
    • The work was being done by one person, @okutoyama, who had been doing the research singlehandedly, and was also fixing the problem
      • @bugfire also helped me with some of the stuck areas
    • The work could have been divided among the team since it was related to the development of the entire engineering team
    • Try ⇒ After investigating, you don't start as is, but set tasks one by one after you get a general perspective

Conclusion

We started our research in April of this year, and although there was a period during which we had to stop due to other work within the team, we were finally able to migrate and release it on November 24. We are very happy that we were able to finish it before the end of the year.

I've had a few struggles with Webpacker, but it's also the fact that it's been a mainstay of CrowdWorks' modern front-end development for a long time. Thank you for fighting with us so far.

Webpacker脱却記事も多いので、フロントエンド人材が育ったら引き剥がすことがそこまで難しくなさそうなのもポイントです。

Webpackerをフロントエンド環境の補助輪と考えて、将来存在が足かせになったタイミングでWebpackerのレイヤは剥がしてもらえばよいと考えています。

メンテ不能になったフロントエンド環境を立て直す話 - クラウドワークス エンジニアブログ

After rereading articles about rebuilding the front-end environment that had existed in the past, I personally felt a lingering sense of joy that we had reached a point where it was OK to eliminate this layer.

Although I was able to quit Webpacker, I hope to continue working on improving the build flow without compromising the developer experience, including upgrading webpack.

References


  1. Where there were differences in some of the build results, we decided to allow them because we were able to confirm and determine that there were no problems with the display and operation. 

Top comments (0)