I have an ongoing project which uses 99% ESM (ecmascript modules) no build steps ecetera. With the rise of Deno (finally) I thought about how a browser could work with the same non centralized dependcy manager, (eg. You).
We already have cdns and esmodules, but with a far larger common js (CJS) ecosystem, thanks to node.js writing the rulebook in the interim. But some time later an official ecmascript standard was released, import export ❤️. To be fair to node.js, they started supporting it under a flag and nowerdays it's a working thing. But I put it to you that the damage has been done, we still see CJS style libraries everywhere.
It seems like slow progress for library authors to catch-up, that's also making a big assumption that they even care to target ESM directly, bundlers and tooling such as Babel and Typescript gave us the ability to transpile this functionality down to lesser EcmaScript, in some circumstances for the best support the resulting code uses require.
Detour
If we head over to pika.dev you will find a wonderful CDN that will find ESM supported libraries over at npm, provide a URL, distribution code and even Typescript types all in one request! it's a great tool for Deno and also frontend too.
Your brain on Pika
The trouble is that you will quickly start cycling through libraries degrading your original choice if that library isn't currently supporting ESM. An example, for some reason I want to use jQuery, I query pika and no, jQuery is found but doesn't support ESM, so now what?
Withdrawal
I might think of libraries similar to jQuery, maybe Zepto, is that thing still around? Nope, okay what about this one, "bobs-jquery-esm-fork"... So first I think, do I trust this Bob? Is he going to steal my 💰 money somehow, I do some digging, okay looks legit, is Bob keeping this fork up-to-date? Hmm it's the best option I have.
I could raise a ticket over on Microsoft GitHub, but I really neeeed jQuery in my life right now. Hmph okay let's do it.
Anger
You can see the problem with ESM, despite it's many advantages, perf and simplicity benefits, I should just go get a library and use it and not have to deal with this 💩.
Denial
What to do, what to do? Well I'm going to look for some tooling like webpack but in the style of ESM. Looks like pika built something called pika pack. What's that? Well
the glossy marketing website hadn't been created so I couldn't understand what the engineers where saying it does, but I understand one thing, superseded by Snowpack.
Snowpack, now that's a great name 💍 but is it marriage material? The docs are okay, 4 🌟. TLDR on this one, it's incredible but not for my project which is a pitty, Typescript has been a big part of my life even before it was cool. However Typescript support seems to be largely Babel powered and configuration is abstracted, this is probably great but not me, I'm a tinkerer and I believe that defaults don't cut it for long.
Acceptance Criteria
So what else? Without going down the realms of the obscure, let's just make something, it's not like I didn't explore and research before coming to the conclusion that I would like the following.
- Convert CJS to ESM
- Must have developer buy in and open source support
- Use npm or yarn to download dependencies just like Webpack workflows.
- Must not bundle (HTTP2 and 3 I'm thinking of you)
- Create a config file which specifies the dependencies I would like to 🧙♂️ magically transform into ESM.
- Ability to add CDN urls to this list and have them download and ESMify, this should cover edge cases where transforming didn't work out.
- Must accept both Typescript and JavaScript sources; transpiling to .mjs (tsc can't do this!)
- All ESM to sit in a flat directory structure just like modern node_modules, guess we should call it es_modules folder.
- The tool must focus only on dependencies for the browser but will try to help Deno users with node_module compatibility.
- must throw away unused CJS or development deps via runtime headless browser evaluation
- Generate .importmaps to stitch it all together
Not again!
Wow that's going to take me 20 years, fortunately Typescript to the rescue, the TSC compiler has some and I say that loosely, documentation on using the TSC compiler Api. This Api allows a developer to write up bespoke compiler programs using TSC under the hood, it sounds involved but it's not to bad really, you can write a compiler in maybe 30 lines. Anyway it has some interesting properties that make it ideal, for one, it has transformers. No not Optimus 🚚, I mean it introspects code you tell it to and allows you to change stuff. If I point my custom tsc program at a CJS module, I can have it take in JavaScript and typescript and have that all transform to export import syntax.
There is even a transform out in userland I can use, now all I need to do is impliment the rest of my wish list.
❤️ Hope you enjoyed! 🦄
🌟 Please star the repo if interested, it helps me out alot!🌟
Introducing Unpack
github.com/adam-cyclones/unpack
... but at the moment its broken because I am refactoring it into lib and cli packages.
Top comments (8)
neat! so if i'm understanding this correctly, you're going to be extending the TS compiler, which will look up all of your project dependencies and convert them to ES modules...then that would eradicate the need for any bundling at all!?
100% correct. And for those who say but thats lots of requests, I'm targeting HTTP2 servers that want lots of requests in smaller sizes. The exact opposite of a bundle.
very interesting, do you have a github repo for this? I started a recent a project with snowpack but hit the same issues as you and ended up just falling back to parcel.
Il get back to you soon about that. But I do need a name for it. Any ideas?
Here is a successful compile of all dependencies related to an example project currently nested in the repo.
look forward to it!
u know what they say, the two hardest things in programming is naming things and cache invalidation! 😜
Introducing Unpack
github.com/adam-cyclones/unpack
... but at the moment its broken because I am refactoring it into lib and cli packages.
The Unpack Repository
Milestone 1.0.0: cli hooked up to lib-unpack
What is the plan sailor?
Lib unpack needed a client to allow us to interact with it for development and your use.
There will be a config file client also. (Npm installable)
We (I but not lonely) are now outputting converted common.js converted to Esmodules into an es_modules folder, a .importmaps is also emitted to help future browsers support this natively, I will keep up with the spec, there is also a loader pollyfill in existence using system js under the hood. So I'm going to use that but optionally switch off the pollyfill when the time is right. Very exciting. What next, probably optimization of release profile and getting the client side working.
🥳