Using Git trailers, grep, and cherry-pick to generate unlimited combinations of features, stacks, and platforms

What would you say professional software engineers do? The job usually involves a lot more than writing code. Meetings. User feedback. Bug fixes. Design decisions. Choosing the right tools. Cost-benefit analysis. Team building. Delegation. The list goes on.

The job can vary significantly, depending on the role and level of experience, but at the end of the day, it all comes down to producing code which solves real problems and meets business needs. The effective output of software engineering can be measured as code committed, not by the number of lines or even the number of commits, but by the problems solved, features added, and systems integrated.

If you're a professional software engineer, there's at least a 99.99% chance you know what Git is, and you probably use it almost every day. The tangible work you produce is typically a series of Git commits and branches. The end result is the current version of your application.

What if you had a way to skip all of the time-consuming integration work, a way to pick and choose exactly the commits you need to produce your desired application? That is exactly what we're doing here.

Almost every user-oriented, data-driven app in existence shares much of the same core functionality, especially today, as users have grown accustomed to features like two-factor authentication, receiving emails, push notifications, dark mode, in-app payments, biometrics, cross-platform support, and everything else. For any serious application, users tend to expect all of this functionality at a minimum, and it can be extremely time-consuming to integrate all of it on your own in a manageable, extendable way, specific to your stack. Why should developers everywhere spend their time reinventing the wheel, over and over again?

Tools and technologies improve, standards change, and sometimes it is necessary to upgrade your stack and your codebase to keep up, but overall, the core functionality remains the same. Our mission with Molecule.dev and its open source tools is to give developers everywhere a way to skip reinventing the common core functionality and jump straight to customizing and building what really matters, without being locked into a single particular paradigm or ecosystem. Ideally, you'll be able to instantly add, remove, swap, and upgrade features, platforms, and maybe even eventually entire languages, and it will all work exactly as expected. (You can already do this with the current options!)

Introducing mlcl

To help make all of this possible, we've developed a command line tool called mlcl. It is designed around patterns and conventions already integral to nearly every developers' daily workflow, regardless of language or platform. Many of the commands resemble git commands, as it mostly uses git under the hood.

Choose combinations of the most popular battle-tested libraries, platforms, and features, depending on your individual needs, and mlcl generates your fully functional codebase by cherry-picking a series of carefully crafted commits from various branches, relevant to your selection.

While it initially produces very framework-like results, the output is not a framework. It is not limited to any single way of doing things, and you have full control over every line of code. Obviously, some structure and opinion is required, but we try to remain as agnostic as possible. The end result is a composition of well-defined, pure functions.

The biggest upside to the cherry-picking approach is that there is no need to devise some elaborate framework with a plugin system or anything like that. All that is necessary is good code and modular git commits.

Another upside is that every line of code has a purpose. There is no extra theoretical code or bloat for you to sift through. What you see is exactly what you get. The surface area is exactly as large as you need it to be, which helps you maintain a simple, predictable mental model.

How does it work?

A few approaches were considered before finally settling on the simplest one, which works with no extra setup and relies only on git.

Git has a handy feature called "trailers", which is just a line (or lines) of text at the end of git commit messages. Historically, only a Signed-off-by trailer was officially supported by using the --signoff (or -s) option, but a --trailer option was added roughly a year ago (since version 2.14) to officially support any trailer. (For older versions, you can manually add the line(s) in the proper format.) These trailers provide useful information about commits. For example, GitHub looks for a special Co-authored-by trailer, which allows you to attribute a commit to multiple authors, visible within GitHub's UI.

For codebases designed to work with mlcl, we use trailers to specify the language, library, platform, and/or feature relevant to the commit.

For example, we'll run a git commit command with messages and trailers which looks something like this:

git commit -m 'enable user authentication' --trailer Molecule='appLanguage=TypeScript, appRenderer=React, userAuthentication=Typical'

See more: Take a look at the commits throughout the various branches of molecule-api and molecule-app!

Each mlcl branch is created (as necessary) as object-hashes of the various combinations of selections. This means that as we add more options, the number of potential branches will grow exponentially. It also means that if you make the same selection as other people, you will clone the same branch when starting out.

It doesn't really matter which branch the original commits are in. There are only two requirements:

1. The author date for each commit must be in the necessary (working) order.

2. Each integration should be developed on its own, ideally within the branch for the minimal feature set. For example, for payment platforms, Stripe, Apple Pay, and Google Pay each have their own branch. If multiple are selected, they will be automatically combined together during assembly.

Note: Developers building on top of the resulting codebase(s) don't have to worry about any of this. You can build your app and commit your code however you want, unless you want to eventually make your own contributions. More on that later!

To assemble every codebase, mlcl runs a fairly complex git log --grep operation to find (and sort) all of the relevant commits for your selection, and then runs git cherry-pick for each one with custom Git merge drivers to automatically resolve any conflicts.

We're currently using only two custom Git merge drivers - one for JSON, and another for all other files using git merge-file's built-in --diff3 and --union options. We'll add more drivers as they become necessary.

When writing code for various options, it definitely helps to have enough experience with Git to know how it will attempt to merge. On rare occasion, placeholder comments and static code boundaries are necessary, but overall, Git is really smart with how it merges every commit. There are also probably some linting configurations which would help enforce mergeable code. For example, cherry-pickable items in an array should each be on their own line with trailing commas.

It's also worth noting that the Git history of each branch is rebased (overwritten) on a regular basis, but this is necessary and expected, as the Git history itself is the result we're after.

This approach seems to work very well and produces exactly the code we want. It also has the added benefit of forcing us to write clean, decoupled, modular code, such that every feature and integration can be added, removed, and/or swapped using simple git commands.

What's next for mlcl?

At the time of this writing, mlcl is very specific to Molecule.dev. One of the short-term goals is to generalize it, refine it, and add more commands to make it easier to go back and forth between branches, checkout the necessary commits to add features and fix bugs, and apply changes to all other relevant branches. It already has all of this, but it can be much more user-friendly. There may also be a more efficient way to pick all of the commits at once by passing a custom editor to git rebase instead of cherry-picking one by one.

Right now, the public version of mlcl queries the Molecule.dev API for the relevant commits to cherry pick, but the next major version will include all of the following additional commands (pulled and polished from the version we're using internally):

• mlcl select --show

Shows the current selection.

• mlcl select --feature=Value

Selects the necessary branch for the specified feature(s), while leaving the other selections intact.

For example:

  mlcl select --appThemes=Light

  mlcl select --appThemes=Dark

  mlcl select --appThemes="Light, Dark"

Or multiple features at a time:

  mlcl select --appUi=Tailwind --pushNotifications=Enabled --paymentPlatforms="Apple Pay, Google Pay"

• mlcl checkout <hash>

Checks out a known mlcl branch hash and updates the selection to match.

• mlcl rebase

Rebases the selection's relevant commits onto the current branch.

• mlcl rebase --show

Shows the selection's relevant commits which would be rebased onto the current branch.

• mlcl rebase --all

Rebases every known selection's relevant commits onto their respective branches.

• mlcl rebase --show-all

Shows every known selection's relevant commits which would be rebased onto their respective branches.

• mlcl rebase --all --continue=<hash>

If for some reason the original mlcl rebase --all failed, you can continue at a specific hash to avoid having to rebase every prior branch again.

• mlcl rebase --all --feature=Value

Rebases every known selection's relevant commits for the specified feature(s) onto their respective branches. Useful to update only the branches relevant to a specific feature (or features) so that no time is wasted rebasing unaffected branches.

• mlcl sync [remote]

Synchronizes every branch with the remote.

Some other useful additions might be things like:

• mlcl find for quickly finding exactly where certain code originated.
• mlcl branch with an interactive human readable list of each branch's features to quickly go from branch to branch.
• mlcl backup to make quick backups of repositories, as the original Git history is overwritten regularly.

What's next for Molecule.dev?

The future isn't 100% certain right now, but I would like to think that this is the beginning of something that can seriously raise the bar for both developers and end users. The long term idea is not only to save developers insane amounts of time and costs, but to also improve the way we think about building software, making it easier and more accessible for everyone to build and use high quality applications.

The hardest part of building the underlying tech is done. It took much longer than expected, but you can now assemble fully functional full-stack starter applications with the most common features, integrations, and platforms, tailored to your needs. Now that the foundation is in place and it works (better than expected!), adding more options should be relatively quick and easy.

Eventually (hopefully sooner rather than later), Molecule.dev will have a marketplace and a job board.

• As a buyer: Imagine having a vast selection of features and integrations to choose from, where you can instantly add, remove, and swap each one, knowing that you're getting high quality code tailored specifically to your needs.

• As a seller: Whether it's a UI library, an icon pack, templates, a database, a CDN, anything really... you'll have a place where you can scale your work (which ultimately boils down to git commits, right?) to any number of buyers.

There is a lot of potential here, and refining these tools and practices is an endeavor worth pursuing. If you disagree, please let me know! I would love to hear your thoughts and feedback.

Stay in touch!

Now that mlcl and assemble.molecule.dev is officially working, engineering blog posts like this one will be published more often, aiming for every Tuesday, or every other Tuesday if it isn't interesting/quality content.

At the very least, you can keep up with weekly update digests by signing up via assemble.molecule.dev and enabling the newsletter.

Molecule.dev is 100% bootstrapped and would like to stay that way, which means this can't be done without you.

Also be sure to:

• Share this blog post
• Follow @molecule_dev on Twitter
• Star all of our public repositories on GitHub

Thanks for reading!