It already happened a few times in the history of computing that the level of abstraction the majority of us work on has been raised.
There are, and probably always will be, people that write and understand assembler. But most developers don't worry about the exact instruction they send to their CPU on a day to day basis. Most developers nowadays don't even worry about allocating memory manually. Syscalls to talk to our Operation System is the lowest level the majority of developers has in mind. Mostly it's not even syscalls but an API which the language runtime provides.
I think we are close to a point where the general level of abstraction of computing will be raised once more:
My prediction is that in the near future it will be normal to not think about which physical machine the program we are writing is running on.
We are moving to the cloud.
Of course this is not a new idea. I never have new ideas. But it took me a while to realise the implications of moving to the cloud.
Also please forgive me for using such a buzzword - Cloud is the best description for software with invisible hardware that I have heard of.
From a consumer's point of view this doesn't sound like a new idea. It has become totally normal to have all your files, all your data somewhere in the internet and access it from all your devices whenever you want.
People stopped caring where the cloud actually physically is already a long time ago (this is not completely true: here in Germany everyone is still worried and trying hard to find the cloud).
Our social and working tools have basically all moved already. The laptops and phones we own don't need to offer much. They are becoming commodity. They simply provide a window to interact with the cloud.
In some cases such as gaming the hardware you own is still critical but personal, high-end gaming PCs are also a thing of the past.
For consumers and workplaces the convenience of not having to worry about hardware is such a huge gain, it was an easy move. People just don't care about how technology works. They don't want to know. They don't want to bother. They want their services and tools to simply work and do their job. Let someone else do the maintenance.
Having control over your computer is simply a burden for most people.
But we developers like technology. We want to be in full control over our system and our data, right? I doubt it. Not the majority at least. We are also simply humans. We own or work for companies that try to run a business. If there is a more efficient way and the gain is big enough, we also move.
Our production systems are largely running in the cloud already. We rent virtual servers from Amazon, Google, Microsoft. We push static content to CDN services. More and more of the functionality we need is now available as a service and we don't have to manage them ourselves anymore:
databases, search engines, firewalls, message queues, file storage, load balancers, web servers, build servers, test runners, registries, secret management, user authentication, …
There is a service for basically every generic piece of software out there (if you see one missing, make sure you are the first one to build it!).
The only thing that makes our software unique is our own, custom business logic of how we connect the pieces together.
Surprisingly it is generally still the case that we express business logic as software in the same programming languages and runtimes we used when we were thinking about a physical machines. We put great efforts into taking the existing environments we have in the form of operation systems such as Windows and Linux, faking a virtual environment that is identical to the OS we have from the past and executing our business logic in there nested in unnecessary layers of indirection with OSes in OSes in OSes, ten layers deep.
What if we let go of the past, let go of the control and create efficient platforms suited for expressing our business logic?
If we want to tackle more complex problems, we need a solid foundation. We need to raise the level of abstraction. More complex business domains require us to be able to focus on them exclusively. We need to separate the work of building the foundation from the next layer.
The foundation is important and critical to get right. There will always be a need for great people to work on the foundation of computing. But the majority of problems developers are trying to solve today and an even bigger percentage of what we will have to solve in the future is not about technology in itself - it is all the problems in the world where technology has the potential to help us solve them. And there are plenty. We better get good at using technology effectively instead of fighting against the complexity we created through layers of unhelpful abstractions. Let's admit that it's time for a new abstraction. Let's use our existing languages, platforms and tools for what they are good at and build another solid, efficient layer as next abstraction.
A programming platform for this level of expression has different characteristics than lower-level platforms:
It is mostly glue connecting lower-level components. There needs to be a good API of primitives to talk to available components. The concept of starting and stopping the system is handled by the underlying platform which allows this layer to be much more dynamic, loading only what is needed at this time. Performance critical tasks are most likely handled by lower-level primitives.
There are many more properties to define for such as system.
Did I mention before that we are giving up control? And you know that software is eating the world?
If we don't want all power to end up in the hands of a selected few that run these new platforms, we better make sure we get this right.
Let’s not wait for the big infrastructure companies to come up with a platform. Let’s not wait for them to bind us to their specific ideas and their products. Let’s make sure we create a healthy system with a multitude of options and diversity of ideas.
It is great to see alternatives to the big cloud providers showing up. Shout out to netlify, fastly, DigitalOcean and friends! A healthy market needs competition.
Similar to the variety of operating systems and programming languages out there, we should aim for a variety of cloud platforms that are compatible with each other and share standards and protocols.
This is how technology like email and the web became this widely adopted.
The greatest efforts I see at the moment is the work happening under the umbrella of the Cloud Native Computing Foundation. We need more initiatives like cloudevents and openmetrics!
One of the next big steps I see is in developer tooling. Developers like to and need to be in control of their systems to test, monitor and debug them. It cannot be the right way that developers have to emulate a cloud on their laptop. Instead, let's move development environments into the cloud. And developers need good tools to be productive. A web interface is not enough. We should have realtime APIs on which we can build tooling on top of.
There is still a lot of room in this space waiting to be filled. Moving IDEs into the cloud is only a very first step.
Fellow developers, beware of the level of abstraction the problem you are trying to solve is on. Be ready to development in the cloud. Let's work together to make our next platforms a great place to work in.
Thank you.
Top comments (19)
A nice discussion I would love to have over a coffee! This is something I've been thinking about for many years. Everything will slowly move on the Cloud sooner or later, and it's a matter of time when our OS's will become cloud based. The devices we will use to access the cloud computer will act like a portal or a "shell" device. If something goes wrong with that device - no problem - access everything as it originally was on another device. This is how the future will look like, at least, this is how I imagine it to be. Good read :)
I don't see myself ever running anything less than a full Linux installation on my computer. Growing up outside of tech hubs, useful internet access is more scarce than oil in this world. And, I assure you that internet providers will keep it that way.
However, what I would like to see is a seamless experience where I, for example, boot up my cheap laptop at a coffee shop, work on something, and then have access to all that work on my home workstation. This can already be achieved but not seamlessly (Git, Docker, etc.)
Then I guess you didn't hear about the projects that Google, Facebook, and even SpaceX are all working on. The achievement of a true WWW is closer than we might think, whatever ISPs want/say. In the next decade or so Internet connection will not be the problem anymore, even if you are in the middle of the desert or the ocean. Now whether the offline system is from Microsoft, Linux based, or Apple, it doesn't matter, so you can still have your full-OS if you want. But most people don't care about a full OS, or a Linux based OS, they simply want to use their working tools / productivity apps / entertainment products etc.. and continue going with their life. They only want a simple workable environment, and something that works seamlessly everywhere / anywhere. The best way this can be achieved is if the system is cloud based.
Those projects by Google, Facebook and that Elon guy are all nice but if history is a teacher, it will be plagued by artificial throttles and pricing schemes that will continue to prevent the expansion of the cloud.
Think about how nobody has access to real cell phone service (at least in the US) since all service providers have artificial throttles and data limits. Remember, back when the internet wasn't quite as fast? We could, at least, use it endlessly. Once we it reached a speed that was truly useful, service providers capitalized on it and nullified the progress that was made.
Yeah, you have a great point here. I guess living in Europe has its own benefits on some things - sometimes. I don't know, I guess I have high hopes about things getting better on a global scale. Who knows, maybe one day in the future. But as you say, right now with all the limitations this is highly unlikely to happen anytime soon.
I agree with the overall sentiment of the post. In terms of making our software unique by connecting our business logic together, that logic has to be stored somewhere (e.g. the cloud). In the context of current tooling offered by AWS et al, we already have the tools to make our business logic connected through cloud functions, serverless, etc.
AWS has some great functionality. It's sad there is little alternative to Amazon in that space.
I also think we need better tooling to develop directly in the cloud.
Also, we need new runtimes that get rid of all the baggage of running directly on the OS.
What Cloudflare is doing is a super interesting step in that direction: blog.cloudflare.com/serverless-per...
Nicely written, I enjoyed reading it!
You really pointed out a change that I observe for a couple of years now in our company: We try to have all our dev, test, int and prod environments in the cloud, try to use terminal servers whenever possible, store data (even business critical) in the cloud (OneDrive, Azure, Google Drive, you name it, people use it) and try to get our customers on board (our ERP system runs on Azure).
Like you mentioned, we in germany aren't fully trusting cloud providers.
There is this one joke at the office:
Which basically means we are not able to operate if Azure servers are down!
At the beginning everyone was suspicious and investigated every bit of change.
Today it is exactly like you said: People don't care about it anymore!
When we developer have to setup our maschines for a new project and it takes more than an hour, we get upset. So we constantly search for alternatives to speed up the process and the concerns vanish.
I've taken the habit of running all development tools on Docker. My current work computer doesn't have Node or the JDK installed. I just spin up the containers. Once I'm done working on a project, I can remove the entire development environment from my computer with a few simple commands on Docker.
The real beauty of it comes when a new developer joins the team. They just have to clone the repositories and build the docker images locally. We have a team using Mac OS, Windows and Linux all to develop the same group of applications which are each containerized with Docker.
This is indeed a really nice way of setting up the dev environment independent of the OS and a smart way to get new colleagues up to speed within no time :)
Thank you, I enjoyed reading your perspective!
I just wanted to comment on this:
Can't agree more:
Usually on a Macbook, you'll tell your OS to download a browser, then, in the terminal of the OS set up your development environment (IDE, toolchain, etc).
I recently got a Pixelbook, and there is a switch of that relation - it is subtle but fundamental. The operating system is literally the browser: ChromeOS! So, as a developer, you want a terminal? Allright, your browser, the OS, will download and launch a container with a terminal! 🤯
TIL about OCCI - thanks Daniel :)
This looks to be heading in the right direction, along with commercial competition from the likes of Terraform, to providing a common cloud infrastructure API, with aspirations to move 'up the stack' and manage PaaS and SaaS, maybe FaaS deployment - good!
I'm also interested in the ongoing process of decoupling and standardizing the API used by that all important business logic: we've gone from libc + targeted binaries, via virtual machine APIs (JVM, CLR) and interpreted/JIT (Python, Ruby, ECMAscript), into containers (Docker, Solaris Containers, OpenVZ, BSD jail, ...), and Serverless/FaaS (AWS Lambda, Azure FaaS, Google Functions, Cloudflare Workers) where we can finally stop thinking about machines and OSes. Right now there are significant API differences across vendors (they need their lock in!) but I'm hoping this will resolve into a portable Serverless API. Early adopters can choose their vendor or use abstraction layers such as serverless.com/ although this provides a low common denominator ATM.
Sorry but this view that we (developers or people) don't care where our data is and where our tools are is terrible for privacy and convenience.
1) if development and tools move to the cloud, well now you have to have internet to write even a single line of code. What if the internet is slow? Too bad. What if the provider is down, under attack (GitHub faces these almost every quarter)? Too bad.
2) All our data is moving to the cloud recently with no regard for encryption or privacy and we've been eating that up because "oh we love new things" but now we're understanding just how much data these big companies have gathered about us and how they're using it to influence us.
In a free market, there will always be winners and losers and if the last 20-40+ years have shown anything, it is that companies get too big to be held accountable.
We simply cannot hand the keys over for the building blocks of software (tools) to the cloud.
Testing, sure. Development, there better be an offline alternative at the very least.
I agree that development and some testing environments must be available offline. The internet will not be reliable enough to truly live on the cloud within our lifetime (or ever). And, yea, not only do companies abuse our data but the bigger the company, the more developers placing security risks near your data.
Sounds well but doesn't work.
So, yeap, it's nice to have on early but then you project grew enough your definitely want to control and own it.
Have you been following AWS re:invent? Any takes on anything they've said or announced?
Not really. Looking forward to a nice summary on dev.to :)
However, AWS and friends taking over control is what I would like us to prevent. What I have seen so far, serverless is great but we need to go a step further than only wrapping existing tools in another layer I think.
My OS in cloud. Not acceptable for years, a nice debate though.
The Dark language is doing awesome things in this direction:
medium.com/darklang/how-dark-deplo...