DEV Community: Raúl Pedro Fernandes Santos

Job boards for remote work

Raúl Pedro Fernandes Santos — Wed, 24 Jul 2024 11:11:47 +0000

One of the biggest perks of working in software development is the ability to work from anywhere in the world. A lot of people tell me they don't know how to find a remote job or project, though. Since I have worked remotely for the vast majority of my career, I figured I'd share some of the websites I use to find remote full-time or part-time jobs, as well as freelance projects.

Unfortunately, the two best websites I knew, Stack Overflow Jobs and Github Jobs, have been closed down in 2022 and 2021, respectively. :(

Luckily for us, there are many others!

Top three recommendations

1. Climatebase

My current top recommendation is Climatebase.

It's not the job board where you will find the most jobs but I believe it is where you will more easily find jobs that will allow you to put your skills to good use.

Climate change is by far the single most urgent challenge humanity is facing, and I believe we should all try to do what we can to fight it. If we can get a remote job in the software engineering world that also happens to be doing something positive about this, I'd say we should do it.

2. Key Values

The second place goes to Key Values.

It is also not the place with the most jobs but it has a unique approach to job searching that helps you find jobs that share your values. It makes me wonder why this isn't the standard in every job board. I find the idea of finding a job by matching values to be absolutely genius.

3. Wellfound (formerly AngelList Talent)

Probably the best job board after Stack Overflow Jobs and Github Jobs in terms of the amount of jobs posted. Lots of young startups and most of them are in the VC game, which might be something you want to take into account, since the ultimate goal of the game is to sell the company, at which point, all the promises that were previously made are tossed out the window.

Remote-focused job boards

These websites are focused on remote work. They may also have other types of jobs but their bread and butter is remote.

Non-remote-focused job boards

These are not focused on remote work (they do have ads for remote work) but are still quite good.

https://otta.com/ - Very interesting job board with a very nice way to filter the things that matter the most to you. They also send out regular alerts for jobs that match your preferences.
https://portfolio.joinef.com - There are usually a lot of very interesting positions on this one.
https://hnhiring.com/ - An aggregator of job posts from Hacker News "whoishiring" account
https://airtable.com/app53PsYpHxJW61l3/shr303f0nPKns4n45/tbl8QszTVi0TT7ur8 - A list of other interesting job boards.
https://hired.com - You create a profile and companies can find you and request an interview.
https://www.honeypot.io - Similar to Hired.
https://landing.jobs - Similar to Hired but to be completely honest, I've had nothing but bad experiences with them, both as a job seeker and as an employer. And they focus a lot on the Portuguese market, which means lower rates.
https://authenticjobs.com
https://relocateme.eu
https://www.monster.co.uk
https://www.simplyhired.co.uk

Talent matching

These sites actively find projects and match them with developers after a pre-selection process.

https://www.toptal.com - Great culture and work-life balance
https://x-team.com - Great culture and work-life balance
https://turing.com
https://www.gun.io
https://www.vettery.com

Freelance work

A good way to get some more income with some side projects, or if you're getting started it can also be a good way to find simple projects to gain some experience with.

https://www.upwork.com - One important thing to keep in mind is that some employers demand that the worker installs Upwork's software, which is basically spyware: every 5 minutes or so, it takes a screenshot of your screen(s) and a picture through your webcam.
https://www.guru.com

Tips and articles about salary negotiation and interviewing:

These are great articles to read with lots of super valuable insights. If you want the most valuable ones, here are three basic rules:

Never, ever tell a recruiter how much you want as a salary; let them give you a number, otherwise you will probably leave lots of money on the table.
Always try to negotiate the salary. No one will give up on you if you ask them if they are willing to pay a little more. If they are that close-minded, you probably don't want to work with them anyway.
Never sign an NDA or non-compete agreement.

Header photo by Kristin Wilson on Unsplash

Why we chose Elixir

Raúl Pedro Fernandes Santos — Tue, 02 Jul 2024 08:03:00 +0000

Some time ago, I worked with a team to rebuild a company's internal web application, which was based on a very outdated version of Symfony, and was no longer salvageable for several reasons.

After some time debating which technologies we should use, we decided to go with Elixir and Phoenix. In short, these tools gave us the productivity, stability, safety, and scalability (the company was planning on opening up the application to the public, with a new API added to the mix, so future performance was a bit of a concern) that seemed appropriate for the company's plans.

A not-so-technical external consultant that was helping out with other parts of the business asked us to explain this choice, since he had never even heard of Elixir, so we wrote down a few of the reasons that led us to the decision, in a way that anyone with just a little bit of technical background could understand.

I haven't been writing much here lately, and I thought it could make a good blog post for anyone else considering these tools. This was not edited or made more readable for the web, it is just a bullet-list dump of the reasons we wanted to convey at that time, so here you go.

It is hard to distill in a condensed version several months of conversations and research about a technology, ultimately leading up to the decision of using it for a project. I tried to summarise everything here but it's still a long read.

Elixir has rock solid stability, which it gains from the BEAM VM (the Erlang underlying platform). Erlang was created by Ericsson for its telecommunications platforms, and to this date is still a very popular choices for mission critical software. Elixir inherits not only the BEAM VM's stability, it also has all of Erlang's OTP libraries and middleware available to it (in the same manner that languages that run on the Java VM usually have access to Java packages).

The fact that it is a functional language with immutable principles, along with its pattern matching capabilities and error handling mechanics, means that the resulting code is typically much simpler, easier to reason about, and more robust than the typical result from an object-oriented or imperative language. In turn, this means less bugs, easier maintenance, easier onboarding, etc.

Being a functional language also makes it less popular than other languages. But also because of that, it attracts more people who actually know how to write good software, instead of folks who did a 6-week "bootcamp" and were tricked into thinking they now know everything they need to be good software engineers. This also has an effect on the ecosystem around the language: the quality of Elixir packages is significantly higher than what is found in other more popular languages. In other words, even though there's a smaller pool of people to hire from and libraries to use, the average quality is significantly higher. We don't get as many low-quality developers, or libraries that are inneficient and riddled with bugs and security holes. On top of it, the Elixir community is well known for its friendliness and support, something we have experienced first-hand over the past year.

Elixir has amazing asynchronous and parallel processing capabilities (in its default configuration, the BEAM VM automatically uses all available CPU cores, so we get true parallelism, not just concurreny), granted by very lightweight processes (you can easily have millions of them running in a single simple machine) and associated functionalities that are very useful. These processes are completely isolated, share no variables / memory, and communicate with each other only through message passing. This allows for safe and easy concurrency and parallelism because there are no locks and no race conditions.

These processes are also meant to be discardable: "let it crash" is a guiding principle for writing good software, meaning that if something goes wrong, it should not fail silently while everything keeps running seemingly without a problem. The BEAM VM makes this super easy and transparent because it takes care of automatically restarting a process that terminates abnormally, while still generating logs, warnings, or whatever we need to detect the problem. In other words, Elixir is highly fault-tolerant by default.

An example of why these processes are useful: we don't need separate infrastructure (message broker, task queue, task workers) to run asynchronous tasks as we would traditionally do, along with all the monitoring necessary to keep it running. Elixir's processes and supervision trees make it very easy and very efficient to have background tasks, like what we would use cron jobs for, or task queues (for example, for sending emails, or periodically fetching data from third-party services). This is simple to do with Elixir alone and works just fine but if we want to go further, there is a package called Oban which can use the same database as our app (no external broker required), and implements a lot of functionality, like scheduled tasks similar to cron jobs, automatically retrying failed tasks with backoff algorithms, a dashboard with metrics and controls for the queues and tasks, and many other niceties.

The BEAM VM is also extremely efficient in computing resource usage, which means we spend less on infrastructure and, perhaps more importantly, reduce our environmental impact.

Though not as extensive as in other more popular languages, Elixir has many libraries available, the number of which grows by an order of magnitude or two when you consider OTP as well. You'd be surprised by how much is covered by the available packages. Due to not being one of the most popular languages, libraries for specific third-party services may not be available but we've found that for these, we usually only want a very limited subset of its functionality, and we can easily implement something ourselves. For example, there is no official library for accessing OpenAI's API but the only thing their official libraries do is make a few HTTP requests. I would even argue that I'd rather not add yet another dependency to my projects if all it gives me is a way to make HTTP requests.

Elixir and Phoenix are known for low latency when handling requests, which is critical for providing good UX.

On top of that, Phoenix has built-in support for web sockets, which it uses in its Liveview library. It's hard to describe how amazing Liveview is to someone who hasn't used it yet. One of the best things about it is how easy it makes it to implement an application that behaves like an SPA without requiring the insanity of a full blown frontend framework. It is able to load data in the background (including loading data asynchronously after loading the basic structure of the page, a.k.a. hydration for a fast "first contentful paint") and maintaining a bi-directional communication channel with the backend to modify data, get real-time notifications of any changes, and modify the DOM in a super efficient way (it's so efficient that other frameworks are copying the methods used by Phoenix). This would allow us to create a web app that could be used as a PWA, saving us a ton of time and money until the need for a native mobile app would arise.

Phoenix uses a component-based approach for building web UIs, similar to what React and other popular frontend frameworks do, allowing for good code organisation and reuse.

Phoenix promotes a project structure that separates business logic from data access logic, and from web or API logic. As we talked about before, it is trivial to add API logic to a project that was started with only a web layer that renders HTML.

Both Elixir and Phoenix provide good telemetry and observability capabilities. Phoenix has a built-in dashboard that provides invaluable information and metrics about not only the application performance and health, but also the underlying BEAM VM.

Phoenix is well known for contributing significantly to productivity and developer happiness.

Bonus: This thread on Reddit is chock-full of goodness about Elixir and Phoenix.

Please don't use React

Raúl Pedro Fernandes Santos — Mon, 06 Nov 2023 08:52:57 +0000

This is the second post of my "Software Engineering is broken" series. In the introduction post, I described a friend's medical appointments platform where an agency proposed using React - a perfect example of how our industry often makes poor technological choices based on hype rather than actual needs. Today's post expands on why React was a poor choice for that project, and examines the broader issue of React's overuse in web development.

You should stop using React. In fact, you probably should have never used React in any of the projects you used it on. But before you pull out your sawed-off shotgun and shoot me, hear me out. Also, what I'll say here also applies to other tools like Vue and Single Page Applications in general, not just React.

React is an impressive piece of technology and it helped push forward the concept of components for web UIs, which is great. Or at least that's what lots of people will say.

But components are far from having been invented by React. Nothing ever stood in the way of organising our server-rendered HTML in a similar manner (most of the good projects I saw during my career did something along those lines) and the manipulation of DOM elements has been possible since anyone cares to remember.

The true revolution from React was the scale. The virtual DOM allowed complex manipulations of the DOM to be done more efficiently at a time when doing them directly was slow. I'll emphasise that again: scale. This is what eludes most people who use React. It was never meant to be used in blogs and static websites, it was meant for frontends that did very heavy DOM manipulations, either in number of operations, number of nodes manipulated, or even both. Frontends that would have been unthinkable became achievable.

So yes, React is impressive and inspired a lot of improvements in web technology we've seen since its release.

Unfortunately, React is also bloated, slow, complex, ignores and reinvents foundational web technologies, it completely negates search engine indexing (and, consequently, SEO), and it duplicates the effort to build a web application because it forces you to have two full software development projects - frontend and backend - with all the costs (not only financial) that come along with each one.

More importantly, the vast majority of projects are much simpler than the scenarios React was designed for, and choosing it ends up causing more harm than good.

So why do so many people swear by React?

React probably became popular due to a combination of a great idea and the usual eagerness to adopt technologies from Big & Cool companies, as if these tools were made to solve the same type of problems small companies have. A lot of people also say that React is "easy," which is a distorted interpretation of the concept of "easy", but probably still influenced a lot of junior developers to get into it.

Interestingly, whenever I ask someone about their motives to use React, they don't really have a concrete reason. Some people are humble enough to admit that "it's what everyone else was using and I just went with it." But more often, people adopt a defensive stance and shield themselves behind flawed arguments.

After reading the agency's proposal for my friend's project, I replied suggesting some alternatives that would be better choices, including telling them to not build a Single Page Application. I proposed that instead of React, they could use simpler tools, like plain HTML and Javascript, and enhance it with something like HTMX and/or Alpine.js. After all, the UI for the application was very simple and had no need for React or to be an SPA at all. If they really wanted to use that type of framework and build an SPA, then they could at least go for something like Svelte or Solid. Or even better, use Phoenix Liveview, Ruby on Rails and Hotwire, or even (forgive me) Laravel's Livewire.

As expected, the agency replied in a very defensive manner and stood by their proposal of using React. Also not surprisingly, they used flawed arguments to defend their choice.

Popularity

They said, and I quote:

Because we're talking about the most popular and most used technologies in the market, versus libraries with no expression and no community. This point is important to guarantee that you can find people who can work on the product, even after we deliver it.

This point is actually based on a valid concern and it is indeed something that should usually be taken into consideration when picking a software tool. The idea is that by using something that is popular, there will be more resources available for that tool. Namely, people to hire and community for support. It was one of my main concerns for my friend's project, because I knew it was not something he had thought about.

But here's the thing: HTMX, Alpine and Svelte are not libraries "with no expression and no community" - quite the contrary!

Let's look at Svelte as an example. According to the State of JS survey, Svelte is in the fourth place for awareness and usage (only the three behemoths, React, Angular and Vue ranked higher, which is to be expected). But it comes first in terms of interest, while React comes in fifth place. And Svelte comes in second place in terms of retention, with only 1% point difference from Solid in the first place, while React is in the fourth place.

HTMX may not be as popular as Svelte but anyone who pays attention to the web development world knows that it has been gaining some serious popularity over the past couple of years and people are saying very good things about it.

I obviously didn't have these numbers in my head when I proposed the alternatives for my friend's project, and I am not a front-end developer, but I still knew these "facts" - so how come an agency that works in this world every single day, did not?

But that's not all. Sure, there are more people who know React than people who know HTMX and even Svelte. And if HTMX was a complex beast that required a lot of time to master, I would agree that it would not be a good choice. But as it turns out, HTMX is super simple to use and would have given them everything they needed to build the project.

More importantly, HTMX does not reinvent the wheel or add a ton of new concepts and techniques with subtle exceptions that need to be mastered and remembered; it reuses foundational web technologies, and because of that, anyone who knows HTML and Javascript will be fully up and running with HTMX in a matter of minutes - much, much less than it would take anyone to learn React.

The big difference here is: in order to work with React, you also have to know HTML & JS, but just because you know HTML & JS, it doesn't mean you know React.

Think about it as a Venn diagram:

So if the concern here is about being able to hire people to continue working on the project, it would make more sense to use HTMX than React, because surely, there are more developers who know HTML and Javascript than those who know React.

Simplicity

Another popular argument for React, and one that the agency did not forget to include, is that it is very simple.

In my arguments, I said React was too complex for the project. Here's what they responded:

About complexity, it is dangerous to interpret it that way. In reality, React helps you structure a project in a more robust way that avoids that same complexity. Otherwise, we'll have a chaotic project without rules for the code structure, and even though it may seem "simpler and lighter," it will become harder to manage.

They misunderstood my point and were mixing a couple of different topics: code structure, and complexity.

React doesn't provide any standard code structure. There may be popular ways to organise source code in a React project but that's by common agreement from the community, not something we can or should attribute to React, and even less something imposed or required by it. A robust code structure will depend, first and foremost, on the developer and good programming principles. React will not do anything for the project structure if the developer doesn't know what they're doing. The question is: what is "well structure code?" If we ask this question to 10 developers, we may get 11 different answers - even within the React world. The main point is to follow good principles, and that does not depend on the tools you use.

As for the complexity that I mentioned, that's not related to the structure of the code but with the concepts and subtleties that React introduces beyond the foundational web technologies, and that one must master in order to use it properly: routing, virtual DOM, JSX, hooks, class-based components versus function-based components, component life-cycle and all its events that you must understand, state management (there are full courses and several libraries for this part alone), among other things. None of this exists on the foundational web technologies and they are all extra things any developer has to learn if they want to work with React, even if they already know HTML and Javascript. And I'm not even going into the insanity that is brought in by the constantly-changing React and Javascript world, as well as things like Webpack, Babel, Parcel, etc. The real complexity comes from these things, not the code structure.

I fully agree that sometimes it is necessary to add complexity to a project because the benefits justify it. However, it is necessary to keep in mind that the introduction of any tool is always a trade-off between simplicity and functionality: to add the latter, we lose some of the former. This is valid for any line of code, even our own. For this reason, it is only justifiable to add tools to a project when there is a concrete problem for which a given tool is the best solution and brings benefits that outweigh the cost. If the agency could show me concrete problems that React was going to solve in my friend's project, problems that could not be solved in a simpler way, I'd be the first to tell them to use React - but of course they couldn't show me any.

Sure, React is simple - if you never work on a serious production-ready project with it. Anyone who has worked seriously with React for some time will have the same opinion: it quickly becomes a headache.

Time and cost

One of the last points the agency made in favour of React was that by not using React, the whole project would take much longer and cost much more.

Update on February, 2024: after more than a year, the agency still has not finished the project, which they initially estimated to take 3-4 months, and already cost 4 times the initial agreed upon budget.

This was probably the most obvious point showing that they had no clue about what they were doing.

It was also one of the reasons why I mentioned the complexity issue, and why I defend simpler solutions. Any project that involves React will automatically take more time and be more expensive because of the points I mention in this article.

It's a fallacy to say that React saves time because the reality is exactly the opposite. React may seem simple for what it proposes and in its basic idea, but it's not simple at all when compared with well-written Javascript or TypeScript, or even something like HTMX.

There's also the not-so-subtle detail of backwards compatibility. Most React releases introduced breaking changes, forcing people to make deep changes or even rewrite their frontend projects. This sounds like the exact opposite of saving time. Compare that to foundational web technologies that were created dozens of years ago and still work unchanged to this date.

Unfortunately, in the Javascript world, hype is everything, and the mass hysteria in favour of React made it a sort of self-fulfilling prophecy. It is very hard to convince its fans that, even though it's technically impressive, it is a bad choice for the majority of projects people work on.

But, as discussed above, if you have a much larger pool of developers potentially available to continue a project that uses HTMX than you would have for a project that uses React; if there's no "React tax" caused by all the hype surrounding it; and if you'll have a much simpler project that will also be much easier to maintain - doesn't it stand to reason that it will cost much less to not use React?

Having examined the organizational and development costs, let's look at perhaps the most direct impact on users:

Performance & Size

React invented the virtual DOM, which was revolutionary for its time. But it can also lead to very slow performance. Not only that, it is like a full application running in your browser.

The library size alone and how much JS code you have to write for your frontend, along with all the stuff you have to include for browser compatibility, polyfills, and even CSS-in-JS. Holy hell, you simply wanted to serve a blog page and suddenly you're forcing your users to download megabytes of code! And this is considered normal? How did we get here?

When you're a developer working on a beefy machine, sure, you don't notice it, but when you're a user on a mobile phone or less powerful device, it can easily become apparent how sluggish it is.

Look at GMail and Reddit. The "rich" versions of their UIs, though containing more features than the plain HTML versions, are so much "heavier" and sluggish, especially in older machines. Is it really worth it?

To put this in perspective, let's look at some numbers:

A typical React application starts with a base bundle size of ~120KB (minified + gzipped)
Add React Router: +20KB
Add a state management solution like Redux: +15KB
Common UI component libraries: +30-100KB

Compare this to:

Alpine.js: ~15KB total
HTMX: ~10KB total
A basic HTML/CSS/JS solution: ~0KB additional weight

These differences become especially significant on mobile networks or in regions with limited bandwidth.

What about when Javascript is unavailable?

Because websites built with React are rendered on the browser, and thus require Javascript code to be executed, if JS is unavailable for some reason, you'll have a few problems on your hands.

First, your SEO will suffer because search engines don't do so well with Javascript, meaning your website won't be properly indexed and thus, your ranking will drop significantly.

Lastly, there are other scenarios where JS may not be available - either by choice, or necessity. And there are perfectly valid reasons for this to happen, like if you're a journalist covering dangerous news and want to do all you can to secure your browser; or when you're using a computer that forcefully configured your browser to disable Javascript; or perhaps you live in a remote location in a third-world country and only have a low-powered device of which you need to conserve the battery; or maybe you simply can and choose to live in a world without JS and disable it altogether because you don't want to be tracked (yeah, it's not enough to fully block tracking but it's something).

If you can and know how to set up a Node server, you can use server side rendering to somewhat mitigate this with server side rendering (something that has existed and has been the default since the dawn of the web, by the way; it is not exclusive to JS frameworks) but at that point React loses a lot of its appeal.

The future

I started writing this article back in 2020 (yes, I'm a very slow writer), and fortunately, since then, the frontend world seems to have gained some sense and more and more people are speaking up against the insanity of React and SPAs. Some examples (which I will update as I find more supporting articles):

As I mentioned before, the idea of components is a good one. Fortunately, we now have many other, better options to use similar components, including in languages other than Javascript.

In fact, you might not even need components at all. If you feel the need to organise your code that way, sure, go for it. If you are working on a complex application and will need to reuse components in lots of places, absolutely, go for it. Need to share components with other projects or teams? Go for it. But if you don't have a problem that components would solve, you can save yourself some time and pain.

My first suggestion would be to go back to foundational web technologies and the web platform: HTML, CSS, and lightly sprinkle some plain Javascript to add interactivity where the web platform doesn't provide what you need. There are libraries - not full-fledged frameworks - that can easily add all the functionality you probably need for the vast majority of the projects you will work on.

Start by taking a look at HTMX and Alpine.js. They both allow you to build web applications that are rendered on the server, while still being able to easily modify the UI in reaction to user actions, load data in the background, use websockets and Server Sent Events, etc. And all of this with simple augmentations to the HTML language - no bloated and complex framework required.

If you can use Elixir, Ruby, or even PHP on the backend, try Phoenix Liveview (by far my favourite), Rails and Hotwire, or Laravel and Livewire. These tools allow you to build a reactive frontend, with bi-directional real-time state synchronisation, without having to pay a huge complexity price. Unfortunately, I don't know of an equivalent for Django. There are a few things but they're not part of the framework, require a lot of manual wiring, and are just not as mature as the other ones.

If for some reason you still want to use a more complex JS framework, then try Solid or Svelte. They are much simpler and much more performant than React, Vue, or Angular. There's also Stencil and Lit, which are quite interesting due to being both based on Web Components standards, meaning they work on all modern browsers and are probably more future-proof than other options (then again, who am I kidding? This is the Javascript world, nothing lasts more than two weeks).

React can be an acceptable choice for some scenarios but people should learn to evaluate the specific needs of the project and weigh the trade-offs before jumping onto the cool kids train. If it solves a specific problem you have that can't be better solved by another tool, go for it. Otherwise, you'll be better off with something simpler.

Header photo by Lautaro Andreani on Unsplash

Software engineering is broken

Raúl Pedro Fernandes Santos — Tue, 20 Jun 2023 07:00:00 +0000

Introduction

A little over a year ago, a friend of mine hired an agency to build a web application for his business. The application is basically a simple online platform for medical appointments and consultations via video calls using Twilio. Since my friend doesn't have that much technical knowledge, he asked me to help him validate what the agency was proposing. When I read the proposal, I was shocked. Not only because of the value they were asking but even more so due to the technological choices. It was clear to me right away that they were the typical agency where no one knew anything other than Javascript and MongoDB (thanks, Freecodecamp! 😠) - in other words, they know very little about proper software engineering.

They mentioned React, Strapi, MongoDB and GraphQL as the foundational technologies for the application. This made absolutely no sense:

Strapi is a framework to build CMS, not a framework or library to build a full-fledged web application.
MongoDB is a document store but this type of application will obviously be based on relational data and thus, it needs a relational database.
GraphQL is meant for scenarios where you have a lot of scattered data and you need to easily be able to fetch random pieces of data from random places.
React might be a good fit for large teams building very large and very complex frontends but this is a fairly simple application and the agency building it is very small.

In summary, none of the tools they chose were meant to solve the problem at hand, which was building a very specific and very well-defined type of web application. Sure, any of those tools could be used to build the application, just as you can also use a spoon to cut your potatoes and a fork to eat your soup - but does it make sense? Just because you can doesn't mean you should.

All of those choices added unnecessary complexity, which meant unnecessary time spent building the application, unnecessary electricity spent and the accompanying carbon footprint, unnecessary money spent by my friend, and ultimately even unnecessary frustration for the developers who are working on it. Yes, "working", not "worked", because they still haven't finished it and are way beyond their original estimation of three to four months.

The problem

Software engineering is broken. We, as an industry, have distorted it so much that what used to be serious work but still possible to enjoy, is now a source of pain, frustration and burnout. A pale shadow of the proper discipline it used to be.

It has become the norm to over-engineer any project from its inception, applying tools and solutions for problems we don't have and probably never will. Even the simplest things have become convoluted. For example, I cannot remember the last time I worked at a company that did not require an insane amount of effort or had an excessive level of complexity simply to get a local environment ready for me to work, let alone doing the actual work.

The saddest part about this is that we seem to glorify this unnecessary complexity. Somehow we convinced ourselves that we need design patterns everywhere. That gigantic dependency trees are normal. That overly complex infrastructure is acceptable. And that to build a simple blog it is necessary to use libraries and / or frameworks that add several megabytes of code to what the visitors' browser has to download. That's what passes for good software engineering in these strange times.

There are way too many "software engineers" who should not work writing code because they contribute negatively to the ecosystem they're in. And let's not forget the end result of their work, which can negatively affect the world. Think "emergency response systems," or "hospital patient management." But they think they are great at it because some tools made it too easy for them to write a "hello, world!" What's worse, these pseudo-programmers became so large in numbers, that they now think they are the ones who are right.

It didn't help that the whole world has settled on the erroneous idea that "anyone and everyone can learn to 'code'." This is true to some point but not everyone will make a good software engineer. Unfortunately, in this post-modern era, we are forced to accept that every opinion is as valid as any other, no matter how ridiculous or unfounded. So we end up with tons of people who write spaghetti code, making a mess of the world.

More shockingly, many of those are able to write spaghetti code in a beautiful way, so we take what they build and put it on a pedestal - simply because it's complex and hard to understand. To me, complex code is a sign of bad code. Nowadays it seems to be a sign of glory. Maybe I'm getting old.

I'm not sure exactly how we got here but I think it's a combination of at least three things:

The "anyone can code" culture, which led to too many pseudo-programmers contributing negatively to the ecosystems.
Marketing folks pushing buzzwords to everyone's mind.
Script kiddies wanting to use the cool tools from the big guys because they all want to be the next Zuckerberg, without understanding the problems these tools were designed to solve.

These things combined led to the problems I mention above, among other things.

To be fair, there are lots of other people out there who, like me, think this has become ridiculous and are doing what is in their power to fix it and go back to a simpler and happier life. In the frontend world alone:

There are many projects trying to create a "better React" from scratch to make it simpler and more light-weight.
The most loved frontend framework is Svelte which tries to massively simplify how frontends are built.
Several well-established communities tried to get rid of the insanity that are SPAs by creating a saner approach to modernizing web applications:
- Ruby on Rails -> Hotwire
- Phoenix/Elixir -> LiveView
- Laravel/PHP -> Livewire
- Django/Python -> Unicorn and Tetra
- Even the React world came up with Server Components

But other parts of the "stack" are not roses, either. Complex design patterns for object-oriented languages have become the norm. Kubernetes and micro-services are used everywhere. People are learning (wrongly) that MongoDB should be used as the database for everything. AWS/GCP/Azure is being used when something like Linode or Digital Ocean would be enough. Everyone is shooting themselves in the foot at every opportunity.

The vast majority of companies are small. Your company is most likely a small company. What huge companies like Google and Facebook do does not apply to you. Not only does it not apply, trying to replicate what they do on your smaller company is actually detrimental. You do not have the same problems they do, and certainly not at the same scale. And even if you did, you don't have the amount of money or people to absorb the complexity of the systems and tools they use. It's incredibly counter-productive to adopt, for example, micro-services and Kubernetes in a company that has 6 or 7 engineers. Such a company does not have the type of problem these tools solve, nor do they have the capacity to absorb the complexity that comes with said tools. And yet, this is exactly what many small companies do.

Returning to my friend's project: over a year later, what started as a simple medical appointments platform has become a cautionary tale of modern software development gone wrong. The unnecessary complexity introduced by poor technological choices has led to delays, increased costs, and frustrated developers - exactly the problems I've outlined above. Through this series of posts, I hope to show that there's a better way: one that focuses on choosing the right tools for the job, embracing simplicity where possible, and remembering that our goal is to solve real problems, not to chase the latest trends. Perhaps by examining these issues one at a time, we can start shifting our industry back towards more sensible, maintainable, and enjoyable software engineering.

I'll link each new post in this series as they're published:

Please don't use React - how the frontend world became unnecessarily complicated
MongoDB everywhere and not knowing relational databases - exploring how NoSQL became the default choice regardless of data needs
Micro-services and Kubernetes - when splitting your monolith creates more problems than it solves
AWS/GCP/Azure when Linode or Digital Ocean would be enough - the hidden costs of over-engineering your infrastructure
DevOps - how a culture of collaboration became a buzzword for complexity

Header photo by Hasan Almasi on Unsplash

The best desktop for programming in 2022

Raúl Pedro Fernandes Santos — Mon, 21 Nov 2022 23:04:57 +0000

A friend recently asked me for some tips to buy a new desktop computer for programming and what I think would be the best choice for components. I started typing away a few things but quickly found myself deep in the proverbial rabbit hole, and since this is a question I am asked fairly frequently, I figured I'd expand a little on what I wrote him and create a blog post.

Besides, black friday 2022 is almost upon us, and even though I find the concept abslutely abhorrent (that's a long topic for another day), many people may find it a good time to get some of this stuff. Also, it helps with SEO.

What and why
- Disclaimer
CPU
- CPU cooler
Motherboard
- Chipset
- The actual motherboard
RAM
Storage
Graphics card
Power supply
Case
The end!

What and why

Why a desktop instead of a laptop, you ask? When it comes to work, laptops are more common than desktops nowadays but even though I love the mobility a laptop grants me, I still want to have a desktop as my main machine. It just makes more sense to me because I can individually upgrade or replace any part as I want or need, drastically reducing my environmental impact and not having to pay for an entire new machine.

And since this is a work machine, I always try to avoid the flashy "gaming" stuff with ridiculous LED lights and absurd price tags. Also, I never go for the high-end stuff because it's too expensive and I just don't need it. On the other hand, I don't want to get the cheaper stuff, since that would make the machine slow and it wouldn't last much.

Currently, my main desktop has these parts in it:

AMD Ryzen 5 2600 CPU with 6 cores and max clock of 3.4 GHz
Gigabyte AB350 3 motherboard
32 GB of RAM DDR4 Kingston HyperX Fury
Asus Radeon RX580 graphics card with 8GB of RAM
Around 5 TB of storage in multiple Samsung SSDs (one 840 EVO, three 870 EVO)

All components except for the three 870 EVO SSDs are a few years old but it's still a very solid machine. The graphics card was launched in 2017 and I bought mine in 2018. It's a good card and I still play games with it without any problem. The CPU is also from 2018 but I honestly don't feel the need to upgrade.

What I suggest next is what I would buy for myself if I were building a new machine from scratch right now, November 2022. Keep in mind that my main focus is software development, not "gaming" (whatever that degenerate verb means). I do play some games from time to time but they're not as heavy on the GPU as the big AAA titles. My main concern is having plenty of RAM and a decent CPU that can easily run multiple processes without choking.

I also suggest that you check UserBenchmark to see for yourself what's out there and find your sweet spot between price and power.

Disclaimer

You will find links to Amazon for the products I suggest. If you buy it from them, I'll get a small fee at no cost to you.

I never used affiliate links on this website and I actually removed the ads I used to have here many years ago but I decided to give this a try and see if I can earn a few €€ from it.

That said, I would strongly suggest you support a local shop and buy from them instead, if you can (they're usually more expensive).

CPU

My recommendation would be AMD, since it tends to be slightly cheaper than Intel for the same power and have more cores for the same price-point.

They recently announced their new Ryzen 7000 lineup of CPUs along with a new socket AM5, and while this means the CPUs and motherboards will be more expensive due to the novelty factor, I still recommend them because if I'm building a new machine, I want it to last as much as possible.

That said, I'd probably go for the Ryzen 5 7600X (buy it on Amazon), based on their "Zen 4" architecture. It's one of the more affordable ones at around €350 but still packs 6 cores / 12 threads, and a clock speed of 4.7 GHz which can be boosted up to 5.3 GHz. My current one also has 6 cores but a lower clock speed and like I said, it's more than enough for what I do.

If you don't mind paying an extra €100 for a couple more cores, slightly more speed, and a tiny bit more L1 and L2 cache, you can go for the Ryzen 7 7700X (buy it on Amazon). Depending on how heavy your CPU usage is, or if you're not planning to get a graphics card (all Ryzen 7000 CPUs have graphics capabilities integrated) it may be worth the extra cost.

Yes, you could even go for a Ryzen 9 7900X or 7950X but frankly, I don't think the extra cost is worth it, and it has the added negative point of spending more energy, which in turn means more money spent on electricity and the equivalent added environmental impact. It also requires more cooling, and though the cooler I suggest is enough, it will probably be more noisy.

CPU cooler

The Ryzen 7000 series don't come with a cooler, so you need to get one separately. Both the Ryzen 5 7600X and the Ryzen 7 7700X have a TDP (Thermal Design Power) of 105W, so you need a cooler designed to handle at least that much. If you decide to get a more powerful CPU, you need to check that the cooler you get is enough for it.

We want coolers to be silent, especially when building a machine for work. Usually, larger fans can displace more air and thus need to rotate less to cool down their targe. This also makes is tricky, though, because coolers nowadays are huge beasts that take up some considerable real-estate inside the computer case and can even interfere with other components, more commonly the RAM modules.

The two coolers I mention should fit any of the motherboards I suggest without any issues but keep this in mind if you pick different components and always check the cooler's dimensions to make sure you have enough clearing for whatever is around the CPU socket.

Noctua has been one of my favourite brands for coolers for a long time because they haven't fallen prey to the "gaming" BS, so their parts are sober, focus on quality, and don't have any unnecessary RGB lights or fancy designs. Their website also doesn't look like it was designed for 5-year olds, and you can actually get useful information on it. But more importantly, their coolers are really good, they are silent and last a very long time.

The NH-D15 (Amazon) is a good one for the setup I'm proposing and should even give you enough room to upgrade to a Ryzen 9, if you want.

Another good option would be the bequiet! Dark Rock Pro 4 (Amazon) and, surprisingly, is actually cheaper than the Noctua.

In case you're wondering, no, you don't need a water cooler for these CPUs. Maybe if you want to overclock them, but even then, the two coolers I suggested should be able to handle it - but I don't recommend overclocking, so do it at your own risk. In any case, if you want to splurge on a fancy toy, the be quiet! Pure Loop (Amazon) or the be quiet! Silent Loop 2 (Amazon) seem like good choices. I'd save the money for something else, though.

Motherboard

The motherboard you get needs to take the CPU into account, of course. Not just the brand, though - the socket type (AM5, AM4, etc), the chipset and RAM type also have to be considered.

The socket is just the physical socket where you insert the CPU. The chipset is what controls the flow of data between all the components, like the CPU, the GPU, drives, RAM, etc. Different chipsets have different characteristics. For example, some may support more SATA ports than others.

Ryzen 7000 supports DDR5 RAM and socket AM5. DDR5 is pretty new, and the Zen 4 and socket AM5 are the latest architectures from AMD, so if you buy a motherboard that supports them, it means it's more "future-proof" and a machine based on them will last longer.

Chipset

The Ryzen 7000 works with these 4 chipsets: X670E, X670, B650E and B650.

The main difference between the X670(E) and the B650(E) is that the X670(E) has more PCIe lanes, and it also supports more USB and SATA ports.

The difference between the ones with E at the end and the ones without, is that the "E" version supports PCI-Express 5.0 for both the graphics (x16 socket) and NVMe drives, while the non-E version only supports PCIe 4.0 for the graphics card and it is optional for the NVMe drive (the board manufacturer decides, I guess?).

See the table below, which is a simplified version of the full table from AMD's page linked above:

Chipset	Graphics	NVMe	USB 10 GBps	USB 20 GBps	Max SATA ports
X670E	1x16 or 2x8 PCIe 5.0	1x4 PCIe® 5.0 plus 4x PCIe® GPP	12	2	8
X670	1x16 or 2x8 PCIe 4.0	1x4 PCIe® 5.0 plus 4x PCIe® GPP	12	2	8
B650E	1x16 or 2x8 PCIe 5.0	1x4 PCIe® 5.0 plus 4x PCIe® GPP	6	1	4
B650	1x16 or 2x8 PCIe 4.0	1x4 PCIe® 4.0 (PCIe® 5.0 Optional)	6	1	4

I would go for the B650 because I don't think I'll need more than 4 SATA ports and the fast USB ports are not that important, but again, I'd like to future-proof my machine as much as possible, so broader PCIe 5.0 support would be nice. The B650E it is, then.

The actual motherboard

After looking at MSI, Gigabyte, Asus and ASRock, I think the best value for money would be an MSI MPG B650 EDGE WIFI (Amazon). For a little bit more (~€50) you can also get the MSI MPG B650 CARBON WIFI (Amazon) which, compared to the EDGE, supports 2 PCIe x16 slots (as opposed to only 1), 4 M.2 slots (only 3 on the EDGE), and a few more USB ports.

The ASRock B650E PG Riptide WiFi (Amazon) is also a nice choice. If you want to throw some money out the window, the Gigabyte B650E AORUS MASTER (Amazon) is also good but it costs a lot more than the other two.

All of these include bluetooth 5.2, audio and wireless networking, so you won't need to buy additional parts for those.

RAM

The AMD Zen 4 architecture only supports DDR5. All Zen 4 / AM5 motherboards support it, so that's one less concern. Other than that, you have to pay attention to two numbers: data rate and CAS latency.

In very simple terms, the data rate is the memory "speed" in MHz, and it is usually represented as a number after the DDR designation, such as DDR4-3600 or DDR5-6200. Usually, higher is better but your motherboard and CPU have to support it.

The CAS latency, sometimes abbreviated as CL, is basically the amount of time it takes for the memory circuit to initiate an operation. Obviously, you want this to be as low as possible.

The motherboards I suggested support dual-channel (meaning you should install RAM modules in pairs), non-ECC, un-buffered memory, up to DDR5-6600.

With that in mind, I'd get a set of 2x16 GB Kingston FURY Beast DDR5-4800 CL38 (Amazon) or 2x16 GB Corsair Vengeance DDR5-5200 CL40 (Amazon).

With the current prices, 32 GB is probably the sweet spot. More or less than that and you'll probably be paying more per GB than it's worth.

Storage

The choice for this part is easy. For years now, Samsung's EVO line has been the best value for money with really good performance that outshines pretty much all competition.

Just grab a 1 TB Samsung 970 EVO Plus NVMe M.2 (Amazon) - don't go for the 980, despite the higher number, it's actually slower than the 970 EVO Plus.

They're both PCIe 3.0, though, which is a shame since the motherboard supports PCIe 5.0, which is 4 times faster - from 32 GB/s to 128 GB/s. Samsung has the 990 PRO (Amazon), which is PCIe 4.0 at 64 GB/s, already double the speed of PCIe 3.0 - but also double the price. Might be worth it, if you have the spare budget.

PCIe 5.0 drives are still not available for mere mortals but once they are, you'll be able to upgrade, if you really want to squeeze all the juice from your machine.

If you'd rather have a more traditional SATA drive, the 1 TB Samsung 870 EVO (Amazon) is your choice. The QVO should have similar performance but it's supposed to last less due to the inferior storage cell technology. Keep in mind these will be significantly slower, though, typically less than 1 GB/s.

One note about speed: I'm mentioning the maximum supported speeds of the PCIe bus but the actual speed of the SSDs will certainly be lower. For example, according to UserBenchmark, the 970 EVO Plus' sequential read speed goes up to 2.3 GB/s, while the 870 EVO only goes up to 468 MB/s.

Graphics card

If you don't want to use the integrated GPU from the CPU, you can get a dedicated (or "discrete", as they're called nowadays) graphics card. I never go for the high end ones because they're ridiculously expensive (thanks again, "gaming").

I tend to prefer AMD's GPUs because they have officially supported open source drivers in the Linux kernel. Many people say they have terrible experiences with the open source Radeon drivers but so do many others with NVidia's drivers. My experience is that AMD's open source drivers have worked much better for me than NVidia's ever did (or even AMD's closed-source drivers, for that matter).

The Asus Dual Radeon RX 6600 8 GB (Amazon) is probably a good choice, as is the Powercolor Fighter AMD Radeon™ RX 6600 8GB (Amazon).

Keep in mind, though, my benchmark is not AAA games, since I don't find most of them interesting at all. For those, you would probably be better served with a more powerful GPU, like the Gigabyte Radeon RX 6600-XT Gaming OC 8GB (Amazon)

Power supply

You power supply unit (PSU) needs to be able to cope with the demands of the rest of your hardware. Without a dedicated graphics card and one of the CPUs I suggested, a 450 W PSU would probably be enough. That said, I would recommend getting a beefier PSU because eventually, you may want to upgrade to a more powerful CPU or add a dedicated graphics card, and in that case, 450 W simply won't be enough, so I'd go for at least 650 W, which definitely covers all the components I suggest here.

You can also use a power supply calculator to have an idea of how much power you'll need for your components.

Another thing to keep in mind is cable management. A modular or semi-modular PSU helps tremendously in this because it allows you to disconnect the cables you don't use. This has more advantages than it seems at first: better airflow and thus, better cooling; less cluttered inside of the case, which makes handling the innards of your machine easier; more room for other parts; in the unlikely event that you want to temporarily remove the PSU, you don't have to disconnect and then reconnect the cables from all the components, you just need to do it directly on the PSU, which is a lot easier.

With that in mind, my suggestion goes to the be quiet! Straight Power 11 750W Platinum (Amazon). This should give you more than enough power even for a better dedicated graphics card than the ones I suggested but if you think that in the future you may want a more powerful graphics card or a beefier CPU, you can go for the 850 W version (Amazon) or even the 1000 W one (Amazon). The price difference between the 750W and the 1000W is "only" ~€50, so if you're not sure, perhaps it's worth spending a little bit more but ensuring you won't have to switch the PSU after a few months or a year.

Case

Since this is a work machine and not a toy, I recommend simple cases that prioritize silent operation and good air flow, and do away with the useless RGB lights and transparent panels that only make it more expensive without adding any real value.

With that in mind, the be quiet! Pure Base 600 Black (Amazon) is a really nice case without costing a fortune.

If you want an even more silent case and with a USB-C port, the be quiet! Silent Base 802 (Amazon) is a greant choice.

And if you don't mind spending a bit more for what I think is probably the best case around, get the Fractal Design Define 7 (Amazon in white, and in black.

The end!

That's it, you have all the necessary components to assemble your machine. Would you do anything differently? Let me know in the comments. If not, then it's time to grab those parts and have fun building your new desktop computer!

How to fix Code42 / Crashplan "Unable to sign in. Unknown error." on Linux (missing libuaw.so)

Raúl Pedro Fernandes Santos — Sat, 16 Jul 2022 22:49:14 +0000

Crashplan, Code42's backup utility, was recently upgraded to version 10.0.0 on Linux and as soon as I upgraded it, it stopped working.

I use Crashplan for some backups and recently, the GUI app said there was an upgrade available. It looked like it automatically downloaded the newer version, upgraded itself, and then it shut down. When I opened it again (the GUI), it would sit there for a minute or two and then complain it couldn't connect to the local service. I tried restarting the service manually but it kept crashing.

I tried upgrading manually but was still spitting out the same error.

Finally I uninstalled and reinstalled it, which seemed to finally allow the service to run and the app seemed to be able to connect to it but then the login was failing with a "Unable to sign in. Unknown error." message.

Digging into the logs, I found that it seems to be missing a native library:

Native library (linux-x86-64/libuaw.so) not found in resource path (lib/com.backup42.desktop.jar:lang)

Unfortunately, I couldn't find anything about this library anywhere, which led me to think it's a proprietary library, meaning they released this new version without it. Great job! :P

Asking Code42 for help only resulted in a generic response saying my version of Linux (Linux Mint) is not supported.

It took me some time to find a solution and since the problem recently happened again when it automatically upgraded from 10.0.0 to 10.2.0, I decided to document it here so I don't have to search for it again.

Apparently Code42's Crashplan doesn't bother to detect other Linux distributions other than Red Hat Enterprise and Ubuntu, even if it's a distribution based on one of those, like CentOS or Linux Mint. It should install a specific library file depending on the detected operating system. Maybe I'm missing something here but I'd say that logic dictates if it needs to detect the OS in order to install the right file, if it can't detect the OS, the installation should fail, or maybe ask the user to manually select which distro they want to be considered.

I've been meaning to switch to Kopia for a while now, so maybe this is the final push I need to finally move away from Crashplan, which is a shame. I used to love the service back in the "Crashplan for Home" days.

Until then, here's how to fix this.

Download the tarball containing the installation files and decompress it - it should create a directory named "code42-install". Then open a shell in that directory and follow these guidelines:

# Extract the installation files so we can access the missing file.
$ gzip -dc CrashPlanSmb_10.2.0.cpi | cpio -i

# Stop the Crashplan service
$ sudo /usr/local/crashplan/bin/service.sh stop

# Copy the missing library file. The actual file path depends on your Linux version. For me it's ubuntu20 but you may need to replace it depending on which Linux version your system is based on. Look inside the nlib directory to see what other distributions are supported.
$ sudo cp nlib/ubuntu20/libuaw.so /usr/local/crashplan/nlib
$ sudo chmod u+x /usr/local/crashplan/nlib/libuaw.so

# Restart the Crashplan service
$ sudo /usr/local/crashplan/bin/service.sh start

That's it. Maybe my next article will be on how to do backups with Kopia...

Header photo by benjamin lehman on Unsplash

How to override and share git configuration options between multiple directories

Raúl Pedro Fernandes Santos — Sun, 28 Feb 2021 14:12:03 +0000

Sometimes I have a set of projects for which I want to override a few git configuration options. For example, I have a directory with all the git repositories for my work projects. For those work repositories, I want to use my company's email address and GPG signing key instead of my personal ones.

The obvious way to do it is to set the relevant git configuration options in each of the directories but if there are many projects, it becomes a tedious process. And if I ever want to change something, I'll have to do it all over again.

Luckily, there's a better way: Git Conditional Includes!

This allows us to add configuration files conditionally, depending on the folder in use. In other words, we can tell git to use different configuration options depending on the repository path.

For example, let's say I keep all my work projects under ~/myworkprojects. All I need to do is:

Edit my global git configuration in ~/.gitconfig and add:

[includeIf "gitdir:~/myworkprojects/"]
    path = ~/myworkprojects/.gitconfig

Edit ~/myworkprojects/.gitconfig and add all the work-specific configurations I want. For example:

[user]
    email = myemail@mycompany.com
    signingkey = ABC123DEF456...

And that's it!

Now all the repositories inside ~/myworkprojects will use my work email and signing key, while anything outside that folder will retain the default configuration. And if we have other folders with multiple repositories for which we want to override configuration options, we can add more includeIf settings in ~/.gitconfig.

We can even define paths that match any directory that ends with our give pattern. An example from the docs:

[...] the pattern foo/bar becomes **/foo/bar and would match /any/path/to/foo/bar.

On the other hand, if we don't add a trailing slash to the gitdir keyword, it will not match sub-folders recursively. For example: [includeIf "gitdir:/foo/bar/"] would match /foo/bar/baz and /foo/bar/boo but [includeIf "gitdir:/foo/bar"] would only match /foo/bar and nothing else inside it.

One other detail to keep in mind is: includeIf includes the file we specify as if we copied and pasted its contents. This means it should be set after the option(s) we want to override in ~/.gitconfig. Otherwise, they will be overridden again.

For example, if our .gitconfig looks like this:

[includeIf "gitdir:~/myworkprojects/"]
    path = ~/myworkprojects/.gitconfig
...
[user]
    email = myemail@mycompany.com
    signingkey = ABC123DEF456...

The [user] options in ~/myworkprojects/.gitconfig will not take effect because they would be overridden later in ~/.gitconfig.

On the other hand, if we do this:

[user]
    email = myemail@mycompany.com
    signingkey = ABC123DEF456...
...
[includeIf "gitdir:~/myworkprojects/"]
    path = ~/myworkprojects/.gitconfig

We will be doing the exact opposite: we first specify the [user] options and then we override them with the values from the included file.

Since the goal is to override the options set in the global config, I just pop the includeIf at the end of ~/.gitconfig to make sure nothing else will get in the way.

Basic rules for software deployment

Raúl Pedro Fernandes Santos — Tue, 22 Sep 2020 14:35:25 +0000

Now and then someone asks my opinion on what is the best way to deploy code to a server. There's a lot to be said about this subject, so I usually end up disappointing the inquirer because I rarely have a simple answer for them. Like so many other things in life, the best way to do it depends on the exact situation. The scale you're operating at, the type of servers you use, the type of application you're deploying, the level of security required, the infrastructure you're using, etc.

What I usually do is convey a few basic rules that I always follow in every scenario.

Automated testing

The first rule is also the one that most people overlook or outright ignore but it's also probably the most important: there has to be some sort of automated testing that runs before anything else and stops the deployment process if anything goes wrong.

Unit testing, integration testing, functional testing - do whatever you want but never, ever deploy code that isn't tested in an automated way.

I can't stress enough how important this is. It gives you peace of mind because you know that if a bug is introduced somewhere, there's a very good chance it will get caught before reaching production (assuming you're doing your tests correctly).

Most people think this is the biggest benefit of testing and it is a huge benefit - but there's something else I consider an even greater benefit, even though it's a much more subtle one.

Not being burdened with the worry of breaking something on the live servers gives developers people peace of mind. By removing this weight from the developers' shoulders, they can be more productive because they can manipulate their code freely without (as much) fear of breaking everything.

We don't write tests (only) to catch bugs before they hit production - we write tests so we can refactor and iterate faster.

There's a lot more to be said about automated testing but the bottom line is: do it, no excuses.

Automate all the things

This is another very common mistake: updating production code by logging into a server and running git pull. That's a bit like playing Russian roulette with more than one bullet in the pistol chamber.

Automating everything means that at most you push a button to trigger a deployment. From that point on, everything has to happen automatically. A very common setup is to have a hook of some sort on the code repository and when code is pushed, a deployment is triggered and a script is executed.

Why is this important? For a very simple reason: we are humans. Humans make mistakes whereas a deployment script is deterministic: it will run the same way every single time. If your deployment consists only of getting a few changed files from your code repository, it's not a big deal. But as you add more steps to this, the probability of making a mistake, even on something simple, gets higher.

For example, a long time ago I worked on a project where the project lead insisted that we manually deployed our code. He said that way we would feel more responsible and would be more careful. I didn't want to take any chances, so I wrote a script to do it for me but most of my colleagues were doing it manually. They did it dozens of times without a problem but one time one of them forgot to run the database migrations and several hours passed before anyone noticed there was a problem. By then there was a ton of corrupted data and we all had a super fun time recovering from the event.

So do yourself a favour and resist the temptation of doing things manually. It's far too easy to mess it up even in simple scenarios. Write a script to automate your deployment from the start of your project, even if it's just a simple git pull. It will eventually grow and if you do it, you will never realise how thankful you should be that you did it.

Deploying the code

This is where things get murky because it depends on what you are developing. For example, if you're deploying a piece of Python or Ruby, you can have a simple script that runs git pull or something similar and restarts your web server or long-lived processes. But if you're building a Java application, it's a bit more involved, because a new .jar file has to be built and whatnot. If you're using a database you may need to run some database migrations. You probably need to compile and publish some static files. There are a lot of variables.

I used to have a Fabric or Capistrano script that contains all the steps I would do if I were manually deploying the code. That script is triggered by TravisCI, CircleCI , Drone.io, Jenkins or whatever Continuous Integration tool I'm using when I push code to my repository (usually to a specific branch, like master). It connects to the servers and executes a series of pre-programmed steps to fetch the new code from the repository, publish static assets, run database migrations, restart servers, rotate DNS for multiple server redundancy, etc.

A lot of people do a simple git pull but the way I used to do it was running
git fetch origin first followed by git reset --hard HEAD. This way I guarantee that if anyone committed the capital sin of manually going into the server and changing live code, those changes are gone.

Nowadays I prefer to use Docker instead of scripts that change things on servers. The CI tool does the job of building a container image and pushing it to a repository. Then it takes care of updating the running services on the servers or a hosted service like Google Cloud Run, Google Kubernetes Engine, AWS ECS, etc.

Single source of truth

There should be a single "source of truth" for your code. This means that anything coming from outside the repository should be ignored, discarded and not permitted on the production servers.

If someone decides to patch a bug directly in the live code, they should get 5 lashes for each line of changed code. If they do that and then don't apply the same changes on the repository, then that's another 20 lashes :-)

That's because the next time the code is deployed, the patch will be overwritten and the bug will be back. Or if you use the plain git pull method, the deployment may crash because there are untracked changes in the server's copy of the repository.

Final thoughts

I hope this gives you a few ideas. Again, there's a lot to be said
about this topic and it entirely depends on your specific situation,
so it's impossible to come up with "the best way" to deploy software.

If this is a topic that interests you, I strongly recommend you read Zach Holman's post titled "How to deploy software". It's one of the best articles I ever read about it. It's long but worth every minute.

What about you, how do you deploy software? What are your basic rules?

8 reasons why I rarely sign an NDA

Raúl Pedro Fernandes Santos — Wed, 02 Sep 2020 22:08:30 +0000

Now and then, somewhere along the recruitment process for a job, the recruiter will hit me with an NDA. In case you don't know what it is, NDA stands for Non-Disclosure Agreement. It's an agreement between two parties meant to protect confidential information they wish to share.

The first problem I have with this is that some recruiters don't mention an NDA from the start. They just assume I'll sign it. If I don't want to sign it, the recruiter and I will have wasted all our time up to that point. For this reason, it only gets worse the later in the process this detail is brought to light.

A job interview should be a friendly conversation about if and how the employer and I can enter a mutually beneficial professional relationship. An NDA makes things less than friendly and mostly only gets in the way. The following are the reasons why I rarely sign one.

1. Legal jargon

The language used in most NDAs is meant for lawyers. I'm sure you've seen the sort of text: inscrutably dense and full of legal mumbo jumbo that sometimes not even lawyers understand.

I'm not a lawyer, so this means that even if I think I understand what's written, I may well be making a costly mistake. I never sign anything that I'm not 100% sure I fully understand.

2. Too much liability

Some NDAs almost feel like a threat instead of an agreement. They have clauses that say I will have to pay tens or hundreds of thousands of Euros in damages in case of a breach. Sometimes it's a tiny startup that doesn't even have a product yet and thus have no market value. Some say they will be the sole arbiters of how much damage I have caused. Some say things I'm not even sure how to describe or interpret. For example (emphasis mine):

The Parties expressly agree that that any breach or threatened breach of this Agreement will cause not only financial harm to the Disclosing Party but also irreparable harm for which monetary damages would not be sufficient remedy for any breach of this Agreement, and that in addition to all other remedies, the Disclosing Party shall be entitled to specific performance and injunctive and other equitable relief as a remedy for any such breach, and the Receiving Party further agrees to waive any requirement for the securing or posting of any bond in connection with such remedy.

How can a mere threat of breach cause as much harm as an actual breach of confidentiality? And keep in mind how easy it can be for someone to twist what someone else says and make it sound like a threat.

This is ripe for abuse and I don't want to feel that I almost have to fear for my life if I were to accidentally reveal something.

3. Too broad

Some NDAs are too broad in what they try to cover, both in scope and reach. To start (emphasis mine):

"The term “Proprietary Information” means, to the extent previously, presently or subsequently disclosed by or for Discloser to Recipient [...]"

They want to punish me if I reveal things they already told me, and they want my consent for this. If they fear I divulge their secrets, how about not telling me anything until I sign the document?

But the real problem is usually the endless list of generic things they don't want me to disclose. I wouldn't find it odd if it included my own name as something they didn't want me to disclose.

For example (again, emphasis mine):

[...] all financial, business, legal and technical information of Discloser or any of its affiliates, suppliers,
customers and employees (including information about research, development, operations, marketing, transactions, regulatory
affairs, discoveries, inventions, methods, processes, articles, materials, algorithms, software, specifications, designs, drawings,
data, strategies, plans, prospects, know-how and ideas, whether tangible or intangible, and including all copies, abstracts,
summaries, analyses and other derivatives thereof), that is marked or otherwise identified as proprietary or confidential at the
time of disclosure, or that by its nature would be understood by a reasonable person to be proprietary or confidential.

Wow, I wonder if they could have included anything else in there. This particular one is even bold enough to include marketing and articles. In theory, those are things meant to be public but they are concerned that I may disclose them.

And who decides what a "reasonable person is" when the time comes to decide whether something is proprietary or confidential?

I usually take this as two possible signs:

the company doesn't know what they're doing and are trying to look more professional, or perhaps they are genuinely trying to protect their value but shooting themselves in the foot in the process, or;
they want a weapon they can use against me in the future.

In any case, I don't want to sign something like this. I don't want to fear to sneeze and get sued because the droplets I expelled may contain some sort of secret.

4. Non-compete clauses

These clauses usually state that I will not work for a competitor during a period of X years (I've seen it go up to 10!)

First of all, what is a competitor? Let's imagine I sign such an agreement with Google or Microsoft. Those companies do business in almost any area you can imagine. This means I would be preventing myself from working in software engineering for that amount of time. Why would I want to do that to myself?

But this doesn't make sense even with smaller companies. Let's say a small startup approaches me. They want to hire me, so we sign an NDA for a job interview. In the end, we conclude we're not a good fit and we part ways amicably, wishing each other good luck. Later, another company shows interest in hiring me. We're a match made in heaven but they happen to have a project that may be considered a competitor to the first company. I wouldn't be able to accept a job there.

They want to protect their ideas, I guess, but the value is in execution, not in the idea. Barring something exceptionally novel, ideas are a dime a dozen. Anyone could come up with a ton of great ideas but only a few companies are actually able to bring them to fruition. That's not because they ferociously protected their ideas with draconian agreements. It's because they executed them better than everyone else.

5. Non-solicitation clauses

These usually say that I will not be hired by, or try to hire any of the company's current employees for a period of Y years (typically from 1 to 5 years).

If their employees are so dissatisfied working there that they want to leave, they should consider changing something. Companies should retain their employees by making them feel valued, not by using legal shackles. Some say it's a way for a company to protect the investment they made in that person. I can understand that but again, they shouldn't do it by shackling their employees. Instead, offer them retention bonuses, give them better working conditions, listen to them and make them feel valued. It's not my fault they're not doing these things.

As for not being hired, how am I supposed to know that someone I never met and is trying to hire me, used to work at that company I signed an NDA with? I don't meet everyone from the company at a job interview. Sure, I can stalk them and try to find out but I don't want to keep track of all the NDAs I sign (see more on that below). Besides, concealing that information is trivial.

But above all, I don't want to skip a great job opportunity because of a 30-minute conversation from years before, which didn't even lead to anything, but for which I was forced to sign an NDA.

6. I don't want the burden

If I start signing all NDAs people ask me to, I will soon lose track of what I signed. If I wanted to make sure I wasn't in violation any of them, I would have to keep track of every detail of every NDA I signed and keep it in mind during every conversation I had.

I don't have time for that and even if I did, I would never want to spend it that way. Besides, it's obviously not going to work because no one can remember that much detail.

7. There's usually no big secret

In most job interviews, the information shared by the interviewer is not a big secret - nor should it be. The interviewer should tell me what the company does, not how they do it. I don't need to know their secret sauce to decide whether I want to work there. Granted, sometimes it is useful or even necessary but most of the time, it's not.

Something even more common is that there's no secret at all. The interviewer does not reveal any secret or does not have anything secret to tell me because whatever the company does is not a secret.

This means that for most cases, there should be nothing to protect and an NDA is unnecessary. Still, they require it of me. It becomes a barrier to dialogue rather than something useful. A thorn in our newly formed relationship.

8. Unenforceable

It is my belief that a lot of NDAs are simply unenforceable, especially when they are overly broad.

Proving that I divulged confidential information can also be tricky, to say the least. I can think of dozens of ways in which I could spread that information or use it to my advantage without it ever being traced back to me.

Finally, unless a company has a legal presence in my country, it's going to be extra hard for them to sue me if I breach the agreement, so why do they even bother?

That said, I never breached an NDA and thus, never faced legal action for doing so. This means I may be completely wrong about all this. Lots of people seem to agree with me, though. If you search the web for this topic, you will find many blog posts and articles saying the same thing.

The 6 reasons why I will consider signing an NDA:

I believe most people have good intentions when asking me to sign an NDA but are inexperienced or haven't thought it thoroughly. Because of this and all the points above, I won't sign most NDAs. Usually, the potential for negative consequences for me far outweighs the positive things that can come out of it.

That said, sometimes I will sign an NDA, depending on a few things:

The job is of special interest to me. If it's not something special, I won't even bother.
I am told upfront that there will be an NDA for me to sign.
The information considered confidential is defined and scoped in a very clear and unambiguous way.
The clause that specifies remedies and damages does not feel like a threat to bankrupt me or ruin my life.
There are no restrictive clauses on my future work.
There are no restrictive clauses on me hiring, or being hired by people who work at the company.

If all these check out, I will consider signing the NDA. Otherwise, it's going to be a "thanks, but no thanks."

Others who also oppose NDAs

As I mentioned before, many other people share this opinion that most NDAs are unnecessary, unfair, and even dangerous. Here are a few of them, in no particular order:

How about you? Do you sign NDAs? What do you think about them? Leave your thoughts and comments below.

Invalid HTTP_HOST header errors in Django and Nginx

Raúl Pedro Fernandes Santos — Wed, 19 Aug 2020 00:38:17 +0000

Do you have a Django application running behind Nginx and getting lots of Invalid HTTP_HOST header errors, even though you already configured ALLOWED_HOSTS correctly?

It's a fairly common problem but it's also simple to fix.

TL;DR: remove the default_server option from your server configuration block so that Nginx doesn't reply to all HTTP requests for any Host header, and place a new server block with default_server returning a 444 status (or, if you're not using Nginx, a 422 status).

Why does it happen?

Note: my explanation assumes you already configured ALLOWED_HOSTS correctly. If you haven't, do it and then check back here to make sure you're doing the Nginx part correctly.

To put it simply, Nginx is proxying to your Django application requests that should be either denied (or maybe proxied elsewhere, if, for example, you have multiple backends).

Many people tend to think this is a problem in their Django application, maybe because Django has an ALLOWED_HOSTS setting that determines for which domain names it should accept requests, but in reality, it is most likely due to a misconfiguration of the Nginx server block that proxies requests to your Django backend.

Some background

Computers communicate on the internet via IP addresses, not domain names. When a piece of software in a computer tries to communicate with another computer via a domain name, like your browser trying to open a website for you, it first needs to translate the domain name to an IP address. This is what the Domain Name System, or DNS, does. For example, if you point your browser to www.borfast.com, it will first ask the DNS what IP address is serving that domain name, will get back a response like 151.101.1.195, and then it will be able to communicate with that IP address.

Very old HTTP servers could only serve a single domain from each IP address. In other words, there was a one-to-one match between domain names and IP addresses. Obviously, this wouldn't scale well, as it would require a separate network interface for each domain, drastically increasing the cost of hosting multiple sites, as well as other problems. To overcome this, a way to serve multiple domains from the same IP address was devised: name-based virtual hosts.

Multiple domains could now be served from a single IP address but the server would need to be explicitly told which domain was being requested. This required HTTP clients, like browsers, to specify exactly which domain name they were trying to reach and thus, the Host HTTP header was born.

Using this header, which became required in HTTP/1.1, the HTTP client would still communicate via the IP address but would also send that extra bit of information with the request, so that the server knew which domain to serve.

"What does this have to do with the error I'm seeing?"

Glad you asked!

This is the part that may be confusing for some people: your server may get HTTP requests that are not meant for your domain name because the header is set by the client and can be set to whatever the client wants.

The simplest explanation is that someone is testing your server or application to see if they can gain unauthorized access or cause some other issues, so they send HTTP requests with specially crafted Host headers to see what they can do. This is a very deep topic, so I won't get much into it but keep in mind it could be a problem.

Another possible explanation is that somehow, someone got a wrong or stale DNS response for a domain and now is trying to reach that domain via your server's IP address, when that domain is actually hosted elsewhere, on another server, with another IP address.

"OK, but I configured Django's `ALLOWED_HOSTS` to only serve my domain; why is it still causing problems?"

Going back to what was said above, this is not a problem with Django but Nginx.

Your Nginx configuration probably looks like this or its equivalent:

upstream django_app {
    server unix:///path/to/yoursite/uwsgi.sock;
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name yourdomain.com;
    ...

    location / {
        uwsgi_pass  django_app;
        include     /path/to/yoursite/uwsgi_params;
    }
}

There are many ways to proxy requests from Nginx to Django and this is just one of them. Yours will probably look different, hopefully with HTTPS thrown into the mix, but the key part is the default_server bit. If you don't have the default_server option, then the server block is probably the only one (or at least the first one) in your Nginx configuration, and Nginx considers it the default.

When Nginx receives a request, it has to determine which server block to use to process the request. It uses the Host header for this and it tries to match it with the server_name option. If the Host header does not match any server_name, or if the header isn't present in the request, nginx will use the default server to process the request.

This means that any request without a Host header or with a header that doesn't match the server_name will be served by this default server block and proxied to the Django backend (in this case, via the uWSGI socket declared in the upstream django_app block).

The header is then passed from Nginx to Django and Django checks it againts the ALLOWED_HOSTS setting. When it's not there, Django throws the Invalid HTTP_HOST header error.

By the way, I imagine this may also be a problem with name-based virtual hosts in Apache, though I never faced this problem with Apache nor have I ever had to fix it, so I don't know for sure.

How do you actually fix it?

Since a lot of people tend to look at the problem from a Django-based perspective, they also try to fix the problem there, sometimes doing things that can even be dangerous, like suppressing Django's security warnings or setting ALLOWED_HOSTS to accept any host or domain name.

Unless you know very well what you are doing, you should absolutely not do either of these things. Warnings in logs are telling you that something is wrong. Suppressing them is rarely a good idea. For example, if someone is trying to hack your server using a host header attack, you may not know about it until it's too late, and you may not have any information for a forensics investigation. The ALLOWED_HOSTS setting in Django ensures that if the value of the Host header doesn't match what you're expecting, Django will not process it, so this should also be set appropriately.

As explained above, your problem is actually the Nginx default server proxying requests to Django that should not be allowed through, so what you need to fix is Nginx's configuration.

Taking the example configuration from before, fixing it is quite simple and has actually been documented in Django's docs for quite some time:

upstream django_app {
    server unix:///path/to/yoursite/uwsgi.sock;
}

server {
    listen 80; // <--- Remove the default_server
    listen [::]:80; // <--- Remove the default_server
    server_name yourdomain.com;
    ...

    location / {
        uwsgi_pass  django_app;
        include     /path/to/yoursite/uwsgi_params;
    }
}

// Add this server block
server {
    listen 80 default_server;
    listen [::]:80 default_server;
    return 444;
}

You have now declared a default server that does nothing and should catch any requests that are not meant for your domain.

Instead of declaring a "catch all" default server, some people prefer to explicitly match the desired domain inside the server block and only serve the requests meant for that specific domain. Like so:

upstream django_app {
    server unix:///path/to/yoursite/uwsgi.sock;
}

server {
    listen 80;
    listen [::]:80;
    server_name yourdomain.com;
    ...

    if ($host !~* ^(yourdomain.com)$ ) {
        return 444;
    }

    location / {
        uwsgi_pass  django_app;
        include     /path/to/yoursite/uwsgi_params;
    }
}

This approach means more configuration text in each server block but if you don’t want to be dependent on a single default server for some reason, it's a better alternative.

In either case, you're all done! You should now stop seeing those pesky errors in your logs and/or emails.