I recently published an article about my 16-year-long journey as a co-founder and sole developer who built and still maintains the entire business suite of a global aviation company with €30 million in revenue. With so much responsibility and so little room for mistakes, my focus gradually shifted towards maintainability and security. But there's one thing that I haven't explained as a takeaway in my previous article, and that's the role of trust.
So let's talk about trust, and why no longer trusting software is a problem in our tech-driven world. The reason is simple: Every security and privacy measure is solely based on trust. Isolation, end-to-end encryption and zero knowledge are just promises made by a developer or company. If they decide to log your passwords and transfer your private keys, they can just do it, and most of the time you won't even notice that it happened. Every piece of software, even a browser extension, is arbitrary code that can compromise your computer and put your private information at risk. In other words, there is no such thing as secure software. When we talk about secure software, then we basically talk about trusting the developer or company to keep their promise. And this trust fades with the heavy use of AI and the lack of responsibility that comes with a focus on velocity, instead of quality. A famous recent example is Proton's heavy shift towards AI.
Companies like Proton exist because trusting big tech companies seems questionable these days, especially when you process information that someone else entrusted to you. It doesn't matter what these companies say or what's written in their terms, in the end, they will very likely process every single piece of information you input into their products. From their perspective, it's rational, because whoever has access to the most information has a clear market advantage. In terms of AI, you must assume that every prompt and every file you upload will be used to make better models, which also means that your own ideas are easier to copy.
We now know why Proton exists and we also know that security is based on trust. So the real question is, can you rely on a company like Proton that calls its developers losers when they want to act responsibly and take some days to verify the security of a new feature?[1] Can you trust a company that forces developers to let AI generate more code than they can process and verify, and might even require them to push to production without review? I can't. Behind closed doors, Proton probably isn't the only company that sacrifices product quality and security for feature velocity. And I see and feel the consequences of such decisions every day when I use software, because almost everything I use is somehow broken. For example, in 6 of the last 10 services, changing the email address did not work. Has macOS ever been so unstable? IntelliJ IDEs have become less reliable since JetBrains started focusing on AI. Have you recently tried the Docker Image Management UI? Even by today's low standards, I haven't seen such a broken UI in a very long time. In my opinion, understanding and taking ownership are core requirements for developers, but today this no longer seems to be the case. I don't want more software or features no one asked for. I want my software to be reliable and secure, but I don't feel that this expectation is aligned with the priorities of these companies.
While AI plays a major role in the current situation and ongoing development, it is not to blame. Don't get me wrong, LLMs make too many subtle and dumb mistakes and they obviously don't know what they are doing. Running them on your host without a controlled sandbox basically means exposing your entire computer to something that can wipe your entire disk, because it did not understand what you meant or it interpreted remote information as a prompt to perform an action. Every AI acts on its own reasoning, which represents an average, to follow one specific goal. Your prompt is just context. For example, when you ask an LLM to serve static HTML files LOCALLY, it will very likely bind a port to 0.0.0.0, because that's what most people seem to do. LLMs don't understand conflicts or consequences, which means you cannot trust AI.
AI has many flaws, but we must blame the people who trust AI too much, not AI itself. Building software mainly by hand means making decisions and changing them consciously and eventually feeling responsible for your work. It also means that you work at the speed you can process and understand information. But when you build with AI, you subconsciously delegate this responsibility and often exceed your information processing capabilities. And if you don't even review or understand the result, you blindly trust something that doesn't even understand what you say or what it did. And that's the fundamental problem in the current state of software, and the main reason why I no longer trust software, especially not software that was written 100% by AI.
This doesn't mean that you shouldn't use AI. It also doesn't mean that you should write every single line of code by hand. But it means that you must slow down and find a way to feel connected to and responsible for the software you build. It also means that you should not blindly trust some new tool or extension and install it on your computer with your main user's privileges. Especially AI must run in a controlled and isolated environment, while every output must be verified.
Before you install software, make sure you can trust the developer or company, and isolate the software when there are doubts. You can never eliminate trust completely, but you can reduce the risk. Start with the reputation of the company and its core people. How much AI do they use? Do they review their code and understand what they are building? Read the source code yourself if it's available, and look for independent audit results. Question feature promises and check whether they are realistic. And look at the business model: How does the company make money, and what is its goal?
[1] https://www.404media.co/manifestos-fear-and-loser-thinking-inside-protons-ai-reckoning/
Btw, if you haven't read my article about my developer's journey, you can find it here:
https://dev.to/hazington/how-i-solo-built-and-maintained-an-entire-business-suite-for-a-eu30-million-company-for-over-a-2fk3
Top comments (0)