I am the founder of lambdragon.com . And I have been answering emails non-stop since you published this. My inbox quite literally exploded :)
Why? Because I am working on this exact problem. As are many of the people who have left comments here. I'll talk about my own take in a second.
First, I want to take a second to acknowledge how great this piece of writing is. You managed to share the underlying insight with more people than we've managed to so far. This is quite remarkable.
I think this blog post of yours is probably the first arrow headed straight into an ocean covered in wildfire (apologies for the nerdy reference, but it is the closest image I can think of that makes justice to the reaction that this can potentially unleash). There have been many, many arrows before, but they seem to have missed.
So, in summary, I owe you a truck full of beer for this post. (Cc @biilmann
, @mojombo
, @ohadpr
, Ala Shiban, Bucky Moore , @ajcwebdev
- you are all witnesses ;) ).
So, why do I care so much? Well, I have a startup that is focused on precisely this. I created it 4 years ago. The original term was: “pure code”.
Just to prove this, you can still find the first prototypes here dev.decoupled.com/magic (this is pre-pulumi, pre everything, and that site was just meant for internal use… but still, it has good archeological value).
After many iterations and trying to use our platform to build a few big "real life" projects, we started uncovering some significant technical challenges. To name a few: Reliable closure serialization, transactional boundaries in async runtimes, reversible deployment processes, long running workflows, multi-runtime / multi-platform dependency management, etc.
Standing on the shoulders of giants (Pulumi’s closure serialization approach, Temporal/Cadence’s long running processes, Graal's multi-runtime JIT engine) and borrowing from academia when necessary for things like async transactional boundaries and type checking of heterogeneous codebases, we've spent he last two years solving all of these problems from the ground up so we can deliver a true "universal, automatically inferred runtime".
More recently we’ve shifted our focus to finding the best way to talk about such a monumental shift. Again, borrowing ideas and inspiration from Netlify, Github and others who have managed to bring about truly revolutionary changes by doubling down on simplicity, we're working on narrative that allows us to introduce this to the greater public, without having to lead them through the entertaining but costly process of understanding the fundamentals.
I know that pulumi/cloud faced this communicational challenge, and I’m sure serverless/cloud is about to discover the same: who is the audience here? Will they really be interested in understanding the “how”, or do they just want to reap the benefits?
Is this still infrastructure as code? Is it a new type of runtime? Are we talking about a new generation of frameworks?
And I can see you're doing a terrific job at figuring out how to place Temporal as a leading player here. I love their technology by the way,
Anyway. I just wanted to thank you for this article. So far it has cost me two days of sleep but once I get over that, I anticipate that I'll be extremely thankful ;)
When I look ahead, where my ambition vastly exceeds my ability, I see something like a truly universal virtual machine who's design would make Alan Kay proud (or is that just what the cloud/web ultimately is, anyway?), I see Armstrong's Entropy Reverser, something that can push-pull self-provisioning (self-REPLICATING?) programs through the fog or to the data, where even persistence becomes an emergent property of intentionally naively written programs, where everything is either machine driven or at least machine assisted and constantly trying to optimize itself, rewriting millions of lines of code and using test cases and type information to find suspected compatible alternative implementations, it's system-level-JIT, and all kind of things my lack of a computer science education probably prevents me from naming properly.
The only thing I know to call it for sure, is a revolution. I can't even fathom how many hours of productivity are spent solving and re-solving the same classes of problems. Why should developers even be asking if they should pick X or Y database? Why should they have to decide of their data should be column oriented or not? Why can't the runtime reoptimize these things on the fly, after all, it's my experience that whatever was true when we made the decision soon became less true as usage and requirements start to migrate. Are all your indices and views not somewhere in your source code already, hidden in plain sight?
Anyway, keep up the awesome work, I look forward to seeing lambdragon's full potential!
The only thing I know to call it for sure, is a revolution. I can't even fathom how many hours of productivity are spent solving and re-solving the same classes of problems
Absolutely!
All programming languages were designed with a set of requirements in mind. But, especially when it comes to popular languages like JS, there are notable omissions: Data, multi-runtime, long running processes, build targets.
These "missing requirements" have to be solved by adding scaffolding. And this is where most complexity comes from.
To solve this, we have 3 options:
The aforementioned scaffolding, which is what we do today with libraries and frameworks
Redesign the languages. This is tempting, and is actually what we did initially (we built a prototype language that had everything that was missing). But this is unrealistic since even in the best case scenario it would take a decade to penetrate. But, most likely, would end up adding to the long list of ambitious integrated languages that are somewhat dormant (like Dark Lang)
With a combination of idiomatic patterns (that don't break or escape the host language) + some new kind of technology that can "infer" the missing aspects from our code
The other thing here is to understand that the mental model that people have when coding is local: One runtime, one machine, one heap, etc.
If we can "virtualize" this local thinking so it can be layered on top of a more complex cloud, then we can give developers the ability to stay within this simple world.
Indeed, the second I saw this article I thought of you, Aldo.
I also recommend people give FSJam9 a listen around the 34 minute mark and my interview on Talking Serverless 38 to hear me talk about many of these ideas, projects, and infra as code solutions. I've been telling anyone who'll listen about this all year.
I'm now starting to realize that we completely skipped Infrastructure as code in the podcast episode. Yet that's where most of our work is going. You brought it up but then we just went down a rabbit hole.
The current craze for build tools and monorepos has distracted the whole industry from the actual endgame.
Build tools don't matter at all. What matters is leverage. Giving developers 10x more leverage. And that doesn't happen via a faster compiler or build caching :)
oh thats wild, I never even heard of lambdragon and yet I was able to help somehow - thank you so much for this heartfelt response!
re: who is the audience here, "show me how it works" vs "just give me benefits"
yes this is a constant (and tbh a little tiring) debate at Temporal too. Marketer instinct/conventional wisdom hews to the latter, whereas Developers often counterintuitively respond to the former.
I think this is tied to adoption stage. if you're working with early adopters, show everything, eliminate all magic. if you are working with late majority, hide it, they don't care.
You can also think about it as platform vs framework - platforms by definition "let non X do X", they aren't going to appreciate the sheer wizardry anyway, but they'll like the simplicity/speed. framework is a little more in userland control/responsibility.
re: what to call it: i like @dougmoscrop
's term Infrastructure FROM Code.
I remember when I first heard about Cadence via some Uber engineers, a long time ago. Someone said: You can Thread.sleep() for days.
And my mind literally exploded.
Because coming up with that simple idea as the solution to "workflows" is brilliant. It is the kind of insight that takes years to understand, and you can only arrive at that when you've already become an expert on workflow engines. Because you start seeing the essence of the problem clearly.
This goes back to the notion of "popular languages simply left important requirements out".
If we add them, we can solve complete categories of problems with zero cognitive overhead for developers.
But, just like Garbage Collection, which is transparent to almost everyone today, it is a messy affair under the hood and has taken years and the concerted work of some of the brightest minds in the industry to reach a point where we can reliably abstract ourselves from the problem.
Heap/Stack persistence is no different.
For example, the issue of dealing with persisted workflows across versioned logic requires some creative and prescriptive solutions.
For further actions, you may consider blocking this person and/or reporting abuse
We're a place where coders share, stay up-to-date and grow their careers.
Hey Shawn,
I am the founder of lambdragon.com . And I have been answering emails non-stop since you published this. My inbox quite literally exploded :)
Why? Because I am working on this exact problem. As are many of the people who have left comments here. I'll talk about my own take in a second.
First, I want to take a second to acknowledge how great this piece of writing is. You managed to share the underlying insight with more people than we've managed to so far. This is quite remarkable.
I think this blog post of yours is probably the first arrow headed straight into an ocean covered in wildfire (apologies for the nerdy reference, but it is the closest image I can think of that makes justice to the reaction that this can potentially unleash). There have been many, many arrows before, but they seem to have missed.
So, in summary, I owe you a truck full of beer for this post. (Cc @biilmann , @mojombo , @ohadpr , Ala Shiban, Bucky Moore , @ajcwebdev - you are all witnesses ;) ).
So, why do I care so much? Well, I have a startup that is focused on precisely this. I created it 4 years ago. The original term was: “pure code”.
Just to prove this, you can still find the first prototypes here dev.decoupled.com/magic (this is pre-pulumi, pre everything, and that site was just meant for internal use… but still, it has good archeological value).
After many iterations and trying to use our platform to build a few big "real life" projects, we started uncovering some significant technical challenges. To name a few: Reliable closure serialization, transactional boundaries in async runtimes, reversible deployment processes, long running workflows, multi-runtime / multi-platform dependency management, etc.
Standing on the shoulders of giants (Pulumi’s closure serialization approach, Temporal/Cadence’s long running processes, Graal's multi-runtime JIT engine) and borrowing from academia when necessary for things like async transactional boundaries and type checking of heterogeneous codebases, we've spent he last two years solving all of these problems from the ground up so we can deliver a true "universal, automatically inferred runtime".
More recently we’ve shifted our focus to finding the best way to talk about such a monumental shift. Again, borrowing ideas and inspiration from Netlify, Github and others who have managed to bring about truly revolutionary changes by doubling down on simplicity, we're working on narrative that allows us to introduce this to the greater public, without having to lead them through the entertaining but costly process of understanding the fundamentals.
I know that pulumi/cloud faced this communicational challenge, and I’m sure serverless/cloud is about to discover the same: who is the audience here? Will they really be interested in understanding the “how”, or do they just want to reap the benefits?
Is this still infrastructure as code? Is it a new type of runtime? Are we talking about a new generation of frameworks?
And I can see you're doing a terrific job at figuring out how to place Temporal as a leading player here. I love their technology by the way,
Anyway. I just wanted to thank you for this article. So far it has cost me two days of sleep but once I get over that, I anticipate that I'll be extremely thankful ;)
When I look ahead, where my ambition vastly exceeds my ability, I see something like a truly universal virtual machine who's design would make Alan Kay proud (or is that just what the cloud/web ultimately is, anyway?), I see Armstrong's Entropy Reverser, something that can push-pull self-provisioning (self-REPLICATING?) programs through the fog or to the data, where even persistence becomes an emergent property of intentionally naively written programs, where everything is either machine driven or at least machine assisted and constantly trying to optimize itself, rewriting millions of lines of code and using test cases and type information to find suspected compatible alternative implementations, it's system-level-JIT, and all kind of things my lack of a computer science education probably prevents me from naming properly.
The only thing I know to call it for sure, is a revolution. I can't even fathom how many hours of productivity are spent solving and re-solving the same classes of problems. Why should developers even be asking if they should pick X or Y database? Why should they have to decide of their data should be column oriented or not? Why can't the runtime reoptimize these things on the fly, after all, it's my experience that whatever was true when we made the decision soon became less true as usage and requirements start to migrate. Are all your indices and views not somewhere in your source code already, hidden in plain sight?
Anyway, keep up the awesome work, I look forward to seeing lambdragon's full potential!
Hey @dougmoscrop ,
Absolutely!
All programming languages were designed with a set of requirements in mind. But, especially when it comes to popular languages like JS, there are notable omissions: Data, multi-runtime, long running processes, build targets.
These "missing requirements" have to be solved by adding scaffolding. And this is where most complexity comes from.
To solve this, we have 3 options:
The other thing here is to understand that the mental model that people have when coding is local: One runtime, one machine, one heap, etc.
If we can "virtualize" this local thinking so it can be layered on top of a more complex cloud, then we can give developers the ability to stay within this simple world.
Indeed, the second I saw this article I thought of you, Aldo.
I also recommend people give FSJam9 a listen around the 34 minute mark and my interview on Talking Serverless 38 to hear me talk about many of these ideas, projects, and infra as code solutions. I've been telling anyone who'll listen about this all year.
Hey Anthony!
I'm now starting to realize that we completely skipped Infrastructure as code in the podcast episode. Yet that's where most of our work is going. You brought it up but then we just went down a rabbit hole.
The current craze for build tools and monorepos has distracted the whole industry from the actual endgame.
Build tools don't matter at all. What matters is leverage. Giving developers 10x more leverage. And that doesn't happen via a faster compiler or build caching :)
oh thats wild, I never even heard of lambdragon and yet I was able to help somehow - thank you so much for this heartfelt response!
yes this is a constant (and tbh a little tiring) debate at Temporal too. Marketer instinct/conventional wisdom hews to the latter, whereas Developers often counterintuitively respond to the former.
I think this is tied to adoption stage. if you're working with early adopters, show everything, eliminate all magic. if you are working with late majority, hide it, they don't care.
You can also think about it as platform vs framework - platforms by definition "let non X do X", they aren't going to appreciate the sheer wizardry anyway, but they'll like the simplicity/speed. framework is a little more in userland control/responsibility.
re: what to call it: i like @dougmoscrop 's term Infrastructure FROM Code.
I remember when I first heard about Cadence via some Uber engineers, a long time ago. Someone said: You can Thread.sleep() for days.
And my mind literally exploded.
Because coming up with that simple idea as the solution to "workflows" is brilliant. It is the kind of insight that takes years to understand, and you can only arrive at that when you've already become an expert on workflow engines. Because you start seeing the essence of the problem clearly.
This goes back to the notion of "popular languages simply left important requirements out".
If we add them, we can solve complete categories of problems with zero cognitive overhead for developers.
But, just like Garbage Collection, which is transparent to almost everyone today, it is a messy affair under the hood and has taken years and the concerted work of some of the brightest minds in the industry to reach a point where we can reliably abstract ourselves from the problem.
Heap/Stack persistence is no different.
For example, the issue of dealing with persisted workflows across versioned logic requires some creative and prescriptive solutions.