I spent the last 3.5 weeks or so switching our primary app over to using embroider, and getting it working with all the optimized settings plus code splitting across routes. Hopefully reading about my experience will help others through the same process, and help accelerate polishing and adoption of embroider within the Ember ecosystem.
Background
Our application
I work at Rowan Patents, and our main Ember app is a desktop (ember-electron) application for patent attorneys to use to draft patent applications -- somewhat analogous to an IDE like VSCode. Most patents consist of a bunch of pages of prose, and some pages of drawings, so most patent attorneys use Microsoft Word and Visio for their patent drafting.
Rowan Patents brings together the prose and drawings, along with a number of other efficiency tools, into a single multi-window desktop application. When an attorney opens a patent application file, they will have their primary prose window, and then also might simultaneously open a drawings window and a tools window (and there are several other special-purpose windows like a splash screen and a landing page). This means that there's a pretty strong association between routes and windows -- the prose window will never visit any of the drawings routes, and vice versa -- and also that we typically have several copies of the application loaded into memory at a time (one per window).
We also integrate quite a few third-party libraries for our core editor technology, for our core diagramming technology, and various other data analysis and transformations such as natural language processing. We've found that many of these bundle a lot of extra code that we don't actually execute or need.
Statistics on our app (line counts done using cloc):
- 11 addons
- ~55,000 lines of Ember js/ts code
- ~9,500 lines of hbs
- ~3,500 lines of scss
Goals
Our two primary goals in moving to embroider were to be able to take advantage of tree-shaking and code splitting across routes, to keep our code-size-related memory footprint under control without incurring high up-front engineering costs or ongoing maintenance costs.
Pre-embroider solutions
Before moving to embroider, we were able to do quite a bit of "manual tree-shaking" of third-party non-Ember libraries. Using module aliasing, null-loader, and string-replace-loader in the webpack config that we supplied to ember-auto-import, we were able to strip out a lot of unneeded code, but at the cost of significant up-front effort and maintenance cost as we upgrade those third-party libraries.
We did not have a solution for splitting our code across routes.
We considered Ember Engines, but opted to wait for embroider to mature...which it has!
Our experience moving to embroider
Note: all of this was using embroider v0.40.0.
Overall impressions
Based on my overall experience, I would qualify embroider as late-stage alpha or early-stage beta quality software. The core functionality is there and is solid, but I still ran across several fairly severe bugs, and the documentation is limited. I spent a good deal of time debugging through embroider's code to determine why certain aspects of my build weren't working, and I don't think I would have successfully executed the whole transition to embroider if I had been unwilling to roll up my sleeves and dive in like that. I don't think the same is necessarily true of smaller apps with fewer dependencies though.
However, after getting our application building, and fixing some runtime errors, it's been feeling stable, and I'm not feeling nervous about inconsistencies in production build output, frequent but intermittent oddities in the development environment, or any of those other things that would otherwise be keeping me up at night.
Process
My process for getting our app/addons building on embroider with optimized settings was:
- Get all addons building on embroider
- Get the app building on embroider
- Manually test & stabilize the app
- Merge back to
main - Get all addons building with optimized flags
- Get the app building with optimized flags
- Enable route code splitting
- Manually test & stabilize the app
- Merge back to
main - Celebrate, spend a chill weekend backpacking on the Washington Coast, and then write this article
What was hard
I've tried to break down all the work I put into making the app build on embroider into categories, and I'll describe them generally without diving too deep. Hopefully they make some sense and it's useful -- I promise nothing!
ES6 module compliance
One thing we had to update in a number of places was code that imported a module as if it had a default export when really it only had named exports. For example, given a module named lib with two named exports, foo and bar (and no default export), we would sometimes do this:
import lib from 'lib';
lib.foo();
lib.bar();
This would work with Ember's module loader, but is not ES6-compliant, and will not work with embroider/webpack. We had to either update to:
import * as lib from 'lib';
lib.foo();
lib.bar();
or ideally (because it's better for tree-shaking):
import { foo, bar } from 'lib';
foo();
bar();
Another related issue had to do with export stubbing. We had a relatively common pattern in our tests of using sinon to stub or spy on module exports. Using the above lib example, we might have a test that would do:
let fooStub = sinon.stub(window.require('lib'), 'foo');
fooStub.returns({ result: 'value' });
await click('.button-that-calls-lib-foo');
assert.equal(fooStub.callCount, 1);
The window.require() would use Ember's module loader to load the foo module, and then stub out one of its exports. This will not work with webpack-generated modules for a couple of reasons, so we had to find other patterns, usually involving either doing the stubbing at a different point in the call stack, or instrumenting the logic in the code itself to provide a stubbing mechanism that wasn't re-writing the ES6 module objects themselves.
package.json dependencies
Unfortunately I don't have a lot of information to share on this one because I never fully bottomed out my understanding, just treated symptoms. But I'll describe what I do know.
We had a number of build failures related to transitive dependencies. For example, we use ember-power-select, which depends on ember-basic-dropdown, and also use ember-basic-dropdown directly. We did not have ember-basic-dropdown declared in our package.json, so when @embroider/compat assembled our app into the v2 package format, it would not make ember-basic-dropdown accessible to our app (basically it would put it in node_modules/ember-power-select/node_modules/ember-basic-dropdown instead of node_modules/ember-basic-dropdown).
I believe in some cases we ran into errors where the dependency was only listed as a peerDependency and needed to be a dependency or devDependency, although perhaps that was non-addon libraries, I don't recall and unfortunately didn't leave enough of a paper trail to verify.
But overall, given how embroider compiles apps/addons into a more statically analyzable and broadly standards compliant package format, we had to do a fair amount of hardening how we declare our dependencies in our various package.jsons.
Statically analyzable components
This already has a great writeup in the embroider documentation. We weren't doing anything truly dynamic, only using dynamic-looking components as a way of currying components' arguments multiple times, so it was not very difficult for us to refactor to avoid doing that, although in retrospect I think we might have been able to more easily do it using the not-yet-documented (outside code comments, as far as I can tell) packageRules.
CSS/SASS
I spent a good deal of effort getting our SCSS (via ember-cli-sass) and CSS working under embroider. Unfortunately I don't have great detail to offer here, because due to Reasons™ (and probably not great ones), I ended up switching from node-sass to dart-sass as part of the move to embroider, so I'm not actually certain how much fiddling was needed for embroider, and how much for dart-sass. But I am certain that it was a mix of the two, so be warned that there may be some non-trivial effort involved in getting your styles working when moving to embroider.
Third-party addons
Hooboy, this was definitely a biiiiig hunk of the work. Our app has been under development for 6ish years, so it has a whole history of legacy code relying on older addons that aren't really Octane-y, plus several modern addons that haven't yet been brought up to embroider compatibility, let alone embroider optimized compatibility.
Getting all the third-party addons working took a lot of effort, but fortunately webpack provides a lot of configuration and tools to hack on things and make them work. Here is a list of addons that were problematic (that should shrink over time) and what we did about them -- I hope these solutions will help people with identical problems and/or serve as examples to help address issues with addons we do not use.
I should also note that there are undoubtedly better ways of doing some of this -- I often took a brute force webpack approach because that's what I knew, but I believe embroider's packageRules might provide nicer solutions to some of these issues.
Before trying any workarounds, I would always update the problematic addon(s) to their latest version, so all of these descriptions and workarounds apply to latest published versions as of 5/14/2021.
ember-file-upload
ember-file-upload imports code from ember-cli-mirage that might not actually be included in the build -- see the discussion in this issue for more details. Since we don't use ember-file-upload's mirage utilities, we worked around this by stubbing out the import:
const emberFileUploadMiragePath = path.join(
path.dirname(require.resolve('ember-file-upload')),
'mirage'
);
// <snip>
webpackConfig: {
resolve: {
alias: {
[emberFileUploadMiragePath]: false,
},
},
}
ember-power-select
Despite having some code in @embroider/compat to support this addon, I found that it would not work with the embroider optimized flags all turned on, I think because even if embroider knows what @someComponent is, the or in {{component (or @someComponent "default-component")}} messes up its static analysis. So, I got my friend (that we'll see a lot more of), string-replace-loader to help out and came up with this wonderful bit of webpack config that definitely doesn't make me secretly feel dirty and/or naughty.
ember-concurrency
With embroider optimized flags (staticAddonTrees in particular), production builds of ember-concurrency fail with a bizarre error. The current theory is that it's a webpack bug, but fortunately the workaround is pretty straightforward.
ember-cli-page-object
ember-cli-page-object has a compatibility module that is meant to support older setups with old versions of @ember/test-helpers, or even from before that library existed. It exposes an API that is meant to more or less emulate @ember/test-helpers, and internally does some tricks with testing for modules at runtime that does not work with embroider optimized.
Since we know our code always has @ember/test-helpers present, we can use string-replace-loader to make that compatibility module statically import and re-export @ember/test-helpers.
ember-metrics
ember-metrics relies on some features of Ember's container and module loader that do not work under embroider optimized to dynamically look up its metrics adapters based on names specified in config/environment.js. Since I know the static list of adapters that our application uses, I wrote an initializer to statically import those adapters and then register them in the container using the names under which ember-metrics expects to find them:
import intercomAdapter from 'ember-metrics/metrics-adapters/intercom';
import mixpanelAdapter from 'ember-metrics/metrics-adapters/mixpanel';
export function initialize(application) {
application.register('metrics-adapter:intercom', intercomAdapter);
application.register('metrics-adapter:mixpanel', mixpanelAdapter);
}
export default {
initialize,
};
ember-changeset-validations
The whole ember-changeset/ember-changeset-validations/ember-validators bundle did not have dependencies set up and declared in a way that would work with embroider optimized, so some module aliasing and an appearance by my new best friend, string-replace-loader provided a workaround.
ember-validated-form
ember-validated-form does a lot of dynamic component invocations that embroider cannot statically analyze. I believe the fact that it uses curly component syntax and doesn't use this.foo vs @foo makes it even harder on embroider. The workaround involved soooooo much string-replace-loader, but it works!
Route-splitting miscellanea
At the end of this all, enabling route-splitting was a very pleasant anti-climax. It almost just worked, and I was SO EXCITED to see only the code I expected/wanted loaded in each window of our application. The couple of minor issues I had to address when enabling route splitting were:
- We had one route whose dynamic parameter was
:idand even though it didn't need to implement theserialize()hook (which is warned about in the documentation), when using the embroider router, passing a model instance to RouterService.urlFor() did not work. Evidently the code wanted the dynamic parameter to end with_idbefore it would look for anidproperty on the model instance, so I changed the dynamic parameter to:sheet_idand everything was hunky dory - We had one test that was doing
this.owner.lookup('controller:invention')before callingvisit(), and that was not returning the controller instance. I believe this was because that controller/route were lazy-loaded, so the code wasn't loaded and the factory wasn't registered with the container beforevisit(). I fixed this by shuffling the test code around to not do thelookup()until after thevisit()call, but this one has me kinda nervous that I have a lot of otherlookup()s like this that are only succeeding because they run after tests that have already caused the code for the lazy-loaded routes to load...but I haven't confirmed/investigated this any further yet.
Other tips, tricks, and gotchas
Here I've collected some gotchas that I ran into and overcame, and some tips and tricks for smoothing out the whole setup and developer experience of working in this bravely-embroidered new world.
Addon build failures
There is currently a bug in embroider that causes addon builds to non-deterministically produce incorrect output so that tests and the dummy app will not load or run. Until it is fixed, the workaround is to disable concurrency (set JOBS=1) in your environment anytime you're building addons (this does not have any effect on apps).
Because this not only produces a broken build but also poisons the embroider cache, I've applied a draconian workaround to all of our addons:
// ember-cli-build.js
'use strict';
const EmberAddon = require('ember-cli/lib/broccoli/ember-addon');
const { Webpack } = require('@embroider/webpack');
module.exports = function (defaults) {
let app = new EmberAddon(defaults, {
// Add options here
});
// https://github.com/embroider-build/embroider/issues/799
process.env.JOBS = '1';
return require('@embroider/compat').compatBuild(app, Webpack);
};
Out-of-memory errors in CI
Until this fix is released (which should happen in the next release after v0.40.0), embroider doesn't quite fully respect JOBS=1. With JOBS=1 it will run the entire build in a single process, but it still warms a worker pool whose size is number-of-cpus-minus-one which, in CI, can be a very big number. Spawning all of these processes and having them load up their code (even though they don't do anything) consumed enough memory that in some of our CI scenarios, there was not enough left over for our build/tests/etc.
I don't know of a good workaround. My super ugly one was to compile @embroider/webpack's latest master, check in a copy of ember-webpack.js to my source tree, and then manually copy it into node_modules to overwrite the existing v0.40.0 file at the beginning of the CI run. DON'T LOOK AT ME!!!!!
Addon-supplied webpack configuration
When using ember-auto-import pre-embroider, addons could supply a configuration to use when building their assets, including a webpack configuration. This could be very handy when addons included third-party libraries that they themselves added to the application bundle using ember-auto-import. As far as I can discern, embroider doesn't have a similar mechanism.
I think the whole idea of modularizing and then merging webpack configs is a little fraught because webpack wasn't really designed for modular configurations -- it all gets merged into a single configuration, so it's not hard to concoct scenarios where the different "modules" of configuration conflict and mess each other up.
That being said, in controlled environments like ours where the addons are private to our applications, and we have specific knowledge of how/where they will be used, such a mechanism can be very handy. So we ended up standardizing on addons putting their webpack configuration in /webpack.config.js so they could be imported and merged from consuming apps/addons.
For example, suppose we have an app, creatively called app that depends on addon-a and addon-b. This gist shows how we might put together and consume reusable webpack configs for this scenario.
Rebuild on addon changes
Pre-embroider, if an addon's index.js implemented isDevelopingAddon() to return true, then when running ember serve or ember test -s, the build watcher would watch that addon and auto-rebuild/live-reload when any changes were made to that addon. embroider uses a different mechanism, which is the environment variable EMBROIDER_REBUILD_ADDONS that can contain a comma-separated list of the names of addons to watch.
I like this mechanism much better because I don't have to modify code to change which addons are and aren't build-watched. But in the specific case of our monorepo with our app and addons, I always want to build-watch the in-monorepo addons. This is because our addons are not published, and are really only separate from the app for code organization and dev/test efficiency reasons. I could set EMBROIDER_REBUILD_ADDONS to those addons in my login configs or something, but I wanted a zero-config solution that would work for my whole team. So I settled on doing this little trick in all our in-monorepo addons' index.js files:
module.exports = {
name: require('./package').name,
};
// embroider doesn't respect isDevelopingAddon(), and we don't publish this
// addon, so always add ourselves to the list of auto-rebuild addons
process.env.EMBROIDER_REBUILD_ADDONS = Array.from(
new Set([
...(process.env.EMBROIDER_REBUILD_ADDONS || '').split(','),
require('./package').name,
])
).join(',');
This way anytime an app or addon runs a build that depends on this addon, it will add itself to the list of rebuild addons.
PLEASE DO NOT DO THIS IN PUBLISHED ADDONS!!!
Conclusions
At the end of this all, we had what we wanted (tree-shaking and code splitting across routes), and I was very excited. It was a good deal of work (3.5 weeks or so as my main focus), although a lot of that was taken up by investigations of embroider bugs, followed by bug reports, sometimes contributing fixes, and sometimes finding the workarounds described above. So hopefully that will pave the way for a faster/smoother experience for others.
embroider is moving fast, and I'm sure much of this article will be obsolete before long, but I hope it does provide some value in the meantime. I'd also love to hear from others -- your experiences with embroider, ideas to improve on the suggestions and methods I've provided, core team members telling me how I've missed the boat and misrepresented anything/suggested anything silly, etc. I hang out in the #dev-embroider channel on Ember Discord, and will happily discuss/answer questions/etc. here or there.
Top comments (2)
Hi, first of all, congratulations and great article! I've read the content from start to finish. I have a question and I was thinking if you could explain them to me.
Get all addons building with optimized flags, does it have a different configuration for optimized flag? Could you please point me to a documentation page to show these optimized flagsGet the app building with optimized flags, does it have a different configuration for optimized flag? Could you please point me to a documentation page to show these optimized flagsI have written a discussion here and I was thinking if you have time, your response is very much appreciated :)
Thank you!
Thanks for sharing your migration experience with embroider.