Raúl Pedro Fernandes Santos

Posted on Jun 20, 2023 • Updated on Nov 18, 2023 • Originally published at borfast.com

Software engineering is broken

#softwareengineering #webdev

Introduction

A little over a year ago, a friend of mine hired an agency to build a web application for his business. The application is basically a simple online platform for medical appointments and consultations via video calls using Twilio. Since my friend doesn't have that much technical knowledge, he asked me to help him validate what the agency was proposing. When I read the proposal, I was shocked. Not only because of the value they were asking but even more so due to the technological choices. It was clear to me right away that they were the typical agency where no one knew anything other than Javascript and MongoDB (thanks, Freecodecamp! 😠) - in other words, they know very little about proper software engineering.

They mentioned React, Strapi, MongoDB and GraphQL as the foundational technologies for the application. This made absolutely no sense:

Strapi is a framework to build CMS, not a framework or library to build a full-fledged web application.
MongoDB is a document store but this type of application will obviously be based on relational data and thus, it needs a relational database.
GraphQL is meant for scenarios where you have a lot of scattered data and you need to easily be able to fetch random pieces of data from random places.
React might be a good fit for large teams building very large and very complex frontends but this is a fairly simple application and the agency building it is very small.

In summary, none of the tools they chose were meant to solve the problem at hand, which was building a very specific and very well-defined type of web application. Sure, any of those tools could be used to build the application, just as you can also use a spoon to cut your potatos and a fork to eat your soup - but does it make sense? Just because you can doesn't mean you should.

All of those choices added unnecessary complexity, which meant unnecessary time spent building the application, unnecessary electricity spent and the accompanying carbon footprint, unnecessary money spent by my friend, and ultimately even unnecessary frustration for the developers who are working on it. Yes, "working", not "worked", because they still haven't finished it and are way beyong their original estimation of three to four months.

The problem

Software engineering is broken. We, as an industry, have distorted it so much that what used to be serious work but still possible to enjoy, is now a source of pain, frustration and burnout. A pale shadow of the proper discipline it used to be.

It has become the norm to over-engineer any project from its inception, applying tools and solutions for problems we don't have and probably never will. Even the simplest things have become convoluted. For example, I cannot remember the last time I worked at a company that did not require an insane amount of effort or had a an excessive level of complexity simply to get a local environment ready for me to work, let alone doing the actual work.

The saddest part about this is that we seem to glorify this unnecessary complexity. Somehow we convinced ourselves that we need design patterns everywhere. That gigantic dependency trees are normal. That overly complex infrastructure is acceptable. And that to build a simple blog it is necessary to use libraries and / or frameworks that add several megabytes of code to what the visitors' browser has to download. That's what passes for good software engineering in these strange times.

There are way too many "software engineers" who should not work writing code because they contribute negatively to the ecosystem they're in. And let's not forget the end result of their work, which can negatively affect the world. Think "emergency response systems," or "hospital patient management." But they think they are great at it because some tools made it too easy for them to write a "hello, world!" What's worse, these pseudo-programmers became so large in numbers, that they now think they are the ones who are right.

It didn't help that the whole world has settled on the erroneous idea that "anyone and everyone can learn to 'code'." This is true to some point but not everyone will make a good software engineer. Unfortunately, in this post-modern era, we are forced to accept that every opinion is as valid as any other, no matter how ridiculous or unfounded. So we end up with tons of people who write spaghetti code, making a mess of the world.

More shockingly, many of those are able to write spaghetti code in a beautiful way, so we take what they build and put it on a pedestal - simply because it's complex and hard to understand. To me, complex code is a sign of bad code. Nowadays it seems to be a sign of glory. Maybe I'm getting old.

I'm not sure exactly how we got here but I think it's a combination of at least three things:

The "anyone can code" culture, which led to too many pseudo-programmers contributing negatively to the ecosystems.
Marketing folks pushing buzzwords to everyone's mind.
Script kiddies wanting to use the cool tools from the big guys because they all want to be the next Zuckerberg.

These things combined led to the problems I mention above, among other things.

To be fair, there are lots of other people out there who, like me, think this has become ridiculous and are doing what is in their power to fix it and go back to a simpler and happier life. In the frontend world alone:

There are many projects trying to create a "better React" from scratch to make it simpler and more light-weight.
The most loved frontend framework is Svelte which tries to massively simplify how frontends are built.
Several well-established communities tried to get rid of the insanity that are SPAs by creating a saner approach to modernizing web applications:
- Ruby on Rails -> Hotwire
- Phoenix/Elixir -> LiveView
- Laravel/PHP -> Livewire
- Django/Python -> Unicorn and Tetra
- Even the React world came up with Server Components

But other parts of the "stack" are not roses, either. Complex design patterns for object-oriented languages have become the norm. Kubernetes and micro-services are used everywhere. People are learning (wrongly) that MongoDB should be used as the database for everything. AWS/GCP/Azure is being used when something like Linode or Digital Ocean would be enough. Everyone is shooting themselves on the foot at every opportunity.

The vast majority of companies are small. Your company is most likely a small company. What huge companies like Google and Facebook do does not apply to you. Not only does it not apply, trying to replicate what they do on your smaller company is actually detrimental. You do not have the same problems they do, and certainly not at the same scale. And even if you did, you don't have the amount of money or people to absorb the complexity of the systems and tools they use. It's incredibly counter-productive to adopt, for example, micro-services and Kubernetes in a company that has 6 or 7 engineers. Such a company does not have the type of problem these tools solve, nor do they have the capacity to absorb the complexity that comes with said tools. And yet, this is exactly what many small companies do.

I started challenging these choices because I'm tired of suffering unnecessarily and feeling frustrated with the way I have to work due to poor technology choices everywhere. There is a saner way to work, so I decided to put my thoughts into a few blog posts. Maybe I can contribute positively to this change. I'll link them here once they're published: