DEV Community: Mustafa ERBAY

System Architecture is a Bit About Paranoia

Mustafa ERBAY — Sat, 09 May 2026 11:50:07 +0000

Recently, a series of OOM-killed errors in the AI generation pipeline running on my own VPS took me back to the old days. I once again saw how a sleep 360 command could wreak havoc on a system and the cost of a simple mistake. This situation made me realize that system architecture is, in fact, a bit about "paranoya."

For me, this state of "paranoia" is like a way of life built on anticipating worst-case scenarios, accepting that anything can go wrong, and taking precautions accordingly. While it might sound a bit negative, my 20 years of experience in the field have repeatedly proven why this approach is indispensable.

Roots of Paranoia: Past Scars

This "paranoid" mindset isn't an empty delusion; it's a result of bitter experiences I've lived through and learned from. Over the years, I've seen many systems crash unexpectedly, slow down, or become completely inaccessible. These incidents formed the foundation of my architectural approach.

I remember once, at a major Turkish e-commerce site, the database server completely locked up in the middle of a critical campaign. I'll never forget the helplessness and panic of that moment. I've experienced similar, though smaller-scale, crises on my own VPS; for example, my disk filling up to 100% on April 28th. Such events taught me how crucial it is to ask, "what if?"

Incidents on My Own VPS

I manage over 13 Docker containers on my own server. Sometimes, when even one starts behaving unexpectedly, I see a domino effect, with others also getting swapped out. These scenarios show how each part of the system interacts with one another and how the weakest link can affect the entire system.

⚠️ VPS Overload and OOM

One of the most classic scenarios I've experienced on my own VPS is out-of-memory (OOM). Sometimes I've encountered situations like kcompactd using 92% CPU, or sshd being unable to accept new connections. This always reminds me how critical it is to monitor resources and know the limits.

Once, I noticed that Docker's build cache had reached 33 GB, and unused images were taking up 23 GB. My server disk was 100% full, and I couldn't even SSH in. This situation painfully taught me that even a simple full disk could cripple the entire operation. Since that day, I regularly run the docker system prune -a command.

Knowing That Everything Can Break

As a system architect, my most fundamental principle is to accept the fact that everything, absolutely everything, will break one day. This could be a hardware failure, a software bug, or a simple human error. What's important is knowing that these breakdowns will happen and how we build a resilience mechanism against them.

I once experienced state corruption in my GitHub Actions runner due to _work/_temp directories. The pain of deleting those directories and having to rebuild the entire pipeline showed me how fragile even automation systems can be. Such incidents explain why redundancy and fast recovery mechanisms are so valuable.

Resilience and Fault Tolerance

This "paranoid" perspective drives me to focus more on the concepts of resilience and fault tolerance when designing systems. Planning how the system will remain operational if a component fails is one of the most critical steps in architecture.

For example, this blog's Astro build process sometimes consumes 2.5 GB of RAM, pushing the total system RAM to 7.6 GB and resulting in an OOM. In such a situation, I add a preflight resource guard to the pipeline to check resources first. If resources are insufficient, I defer the operation and switch to polling-wait mode. Last month, when I typed sleep 360 and got OOM-killed, I had to activate this polling-wait mechanism.

Cloudflare Cache Strategies

Even when using Cloudflare, this "paranoia" comes into play. Astro's default max-age=0 wasn't providing the performance I wanted for static content. Therefore, I implemented an override on Nginx to define longer cache durations for specific paths. This is a matter of making a trade-off between content freshness and performance, and consciously managing that trade-off.

location /_astro/ {
    expires 1y;
    add_header Cache-Control "public, max-age=31536000, immutable";
    proxy_pass http://localhost:4321;
}

In this example, I set a 1-year cache duration for static assets starting with _astro. This reduces unnecessary origin hits at the CDN layer, thereby improving performance and lightening the load on my server.

Security and Constant Vigilance

One of the most prominent areas where "paranoia" is evident in system architecture is security. I always assume that attackers will try to find the weakest point in the system. Therefore, I try to take precautions not only against known vulnerabilities but also against potential risks.

In recent years, I've seen many times how critical CVEs can be. Even on my own system, I've tracked potential risks like CVE-2026-31431 and tried to close a possible vulnerability by blacklisting kernel modules like algif_aead. Such proactive steps strengthen the system's overall security posture.

ℹ️ Proactive Security Measures

Steps like blacklisting kernel modules or disabling unnecessary services, while often seeming like minor details, can prevent major security incidents. Remember, the best security measure is to prevent an incident from happening.

Runner Economics on My Own VPS

To avoid exceeding my GitHub Actions quota, I use a self-hosted runner on my own VPS. While this provides a cost advantage, it also requires me to constantly track updates and security patches. This situation allows me to view "paranoia" as a form of cost optimization and risk management tool.

Even when setting up my AI-powered content pipeline, I noticed errors occurring when slashes (/) were used in tags or when the publishDate field wasn't a quoted string. Even Turkish-specific details like the dotted-i character problem show that unexpected issues can arise at every layer of the system. These kinds of "quirks" are part of my constant vigilance and thinking about every detail.

Paranoia or Professionalism?

So, what exactly is this "paranoia" in system architecture? For me, it's not about being constantly anxious or expecting a disaster at any moment. Rather, it's about understanding the inherent complexity and fragility of systems and taking conscious steps to minimize these risks. We could call this risk-aware design.

This is a kind of engineering perspective. Just as an engineer building a bridge designs with scenarios like earthquakes, storms, or excessive loads in mind, we also strive to make our systems resilient against potential failures. I'm not ashamed of my own mistakes; on the contrary, last month when I typed sleep 360 and got OOM-killed, I learned from that error and developed the polling-wait mechanism. This is about asking, "what can I do to prevent this from happening again?" instead of just shrugging it off as "it happens."

For me, system architecture is a bit like this: being constantly vigilant and designing with the knowledge that everything can go wrong. Without this "paranoia," none of my side projects like hesapciyiz.com, spamkalkani.com, or islistesi.com would run so stably. Have you ever had such "paranoid" moments? I'd love to hear about them in the comments.

That Meaningless Stress After a Deploy

Mustafa ERBAY — Sat, 09 May 2026 09:20:12 +0000

Recently, I deployed a small CSS change for this blog. Normally, it's a simple tweak, just shifting a few pixels, but after hitting git push, that inexplicable tension settled over me again. It was as if I'd deployed a critical banking system; the question of "what if" started swirling somewhere inside.

This feeling is familiar to me; I've experienced it after every deploy for 20 years. Automatically, my hand reaches for the tail -f /var/log/nginx/access.log command, and I open the Cloudflare dashboard in my browser to check cache hit ratios and error logs. Even if everything appears fine, I remain vigilant for a while longer.

Symptoms of That "What If" Feeling

This tension that arises after a deploy is a situation many of us are familiar with. Sometimes it manifests as a minor twitch, other times as a mild paranoia lasting hours. There are even times when I wake up in the middle of the night with an urge to check, wondering, "Did I forget something?"

I don't just experience this process with large projects. Even on my own VPS, in an environment where I manage over 13 Docker containers, I feel this way after a simple configuration change. Despite everything being automated, one still thinks, "What if, just maybe?"

Past Painful Experiences and Triggers

At the root of this "what if" feeling are, I believe, the painful experiences we've had in the past. Those moments are deeply etched in our brains and are triggered with every deploy. For me, some of these triggers are very clear.

On my own VPS, I experienced this feeling most intensely on April 28th. I had deployed a new container, and the next morning, the Pipeline-health monitor sent a "DEGRADED" email. I saw the system was choked with kcompactd %92 CPU; it couldn't even accept SSH connections. The helplessness at that moment, and the hours of debugging that followed, explain the reason for this tension.

⚠️ Docker Disk Fire

Once, again on my own VPS, I experienced a Docker disk fire. The disk filled to 100% due to 33 GB of build cache and 23 GB of unused images. All my applications went down instantly, requiring urgent intervention. Such incidents are among the most significant reasons that reinforce that 'what if' feeling after a deploy.

There were also times when my Astro build consumed 2.5 GB of RAM, pushing the system's 7.6 GB RAM to its limits and causing an OOM (Out Of Memory) error. Or the pain of deleting directories inside _work/_temp on a GitHub Actions runner... All these scenarios have repeatedly shown me that a system can react unexpectedly. That's why, no matter how prepared I am, that meaningless stress lingers with me for a while.

The Balance of Risk and Control

This situation is essentially a reflection of risk management and the need for control. The uncertainty of what consequences our developed or managed systems might cause when they go live confronts us with this stress. Even if we use robust tests, automations, and monitoring tools, the "production" environment always holds its own surprises.

Striking this balance—finding a way between the desire for fast deploys and the goal of risk-free deploys—is often challenging. Sometimes we compromise on certain controls to gain speed, and we might pay the price later. On the other hand, trying to perfect everything also slows down the process.

My Coping Mechanisms

Over the years, I've developed my own methods to cope with this stress. While it hasn't completely disappeared, I've managed to reduce its impact. Automation and comprehensive monitoring are at the forefront of these methods.

I've set up automatic deploy processes with GitHub Actions. Every change is automatically pushed to production after passing tests. With Prometheus and Grafana, I monitor every corner of the system, and with Alertmanager, I receive instant notifications for anomalies. For pipeline reliability, I've specifically implemented preflight resource guards; these check if system resources are sufficient before a deploy.

💡 Small and Frequent Deploys

Instead of large, monolithic deploys, I prefer small, atomic changes. This narrows the scope of a potential problem and makes rolling back much easier. When an issue arises, it becomes much simpler to pinpoint what changed.

Rollback mechanisms are vitally important to me. When a deploy is found to be problematic, I need to be able to revert to the previous stable version with a single command. This sense of security somewhat alleviates that initial moment of stress. Furthermore, I'm not ashamed to make mistakes. Last month, when I wrote sleep 360 and got OOM-killed, I told myself, "this too was a lesson," and switched to a polling-wait mechanism. Learning from my self-created problems helps me be more careful in the next deploy.

The "It Happens" Philosophy and Acceptance

Ultimately, a certain amount of risk and uncertainty is inherent in this line of work. There's no such thing as a perfect system; there can always be a vulnerability, a bug, or an unexpected interaction. Accepting this truth, embracing the "it happens" philosophy, reduces the pressure on me.

Of course, this is not a state of complacency. On the contrary, it constantly pushes me to build better, more resilient, and more secure systems. There are times when I implement kernel module blacklists (like algif_aead for CVE-2026-31431) as part of CVE mitigation; this is also part of the job. I learn from every mistake, every problem, and enter the next deploy better prepared.

ℹ️ Self-Hosted Runner Economics

To avoid exceeding GitHub Actions quotas, I use a self-hosted runner on my own VPS. This both reduces costs and gives me more control. However, it also brings its own operational overhead. Every decision has a trade-off.

This constant state of vigilance has, I suppose, become a part of my profession. Perhaps this situation is a source of motivation that drives us to build better systems. It's not about striving for perfection, but a continuous cycle of improvement and learning.

Do you also have similar experiences after a deploy? How do you cope with that "what if" feeling? What's the first thing you do after a deploy? I'd love if you could share in the comments; perhaps we can learn from each other.

The Hidden Dependency Hell of Cloud-Based Microservices

Mustafa ERBAY — Sat, 09 May 2026 09:13:21 +0000

The Hidden Dependency Hell of Cloud-Based Microservices: A Guide to the Way Out

Cloud-based microservice architectures have become an indispensable part of modern software development. While they offer benefits such as flexibility, scalability, and rapid development, the complexities they bring along also can't be overlooked. One of the most insidious and exhausting of these complexities is the situation we can call "hidden dependency hell."

This situation arises when different services in the system develop invisible, vague, and hard-to-manage dependencies on each other. Although these dependencies aren't noticed at first, over time they erode the stability of the system, make debugging impossible, and turn adding new features into something close to torture.

Sources of Hidden Dependencies in Microservices

Hidden dependencies can find their way into microservice architectures for various reasons. Understanding these reasons is the first step to getting to the root of the problem and producing solutions. Typically, the pressure for rapid development, lack of documentation, or sloppy choice of communication protocols between services lead to these kinds of issues.

Another important source is services becoming indirectly dependent on the inner workings of one another. For instance, one service might expect another service to return a specific piece of data in a specific format. If that format changes, an unexpected chain of errors can follow.

ℹ️ What Is a Dependency?

In the context of software development, a dependency is when a component (a service, a library, etc.) needs another component in order to function. This need can be a direct call, or it can occur through an indirect data flow or a shared resource.

Symptoms and Effects of Hidden Dependencies

The presence of hidden dependencies usually shows itself when sudden and unexplained errors appear in the system. A small change in one service can cause big problems in unexpected places. This situation leads to a serious loss of motivation and a drop in productivity for the development teams.

Such dependencies also negatively affect the overall stability of the system. The debugging process turns into something like a labyrinth because of the difficulty of finding which service caused the issue. New deployments come to be done with the worry of when the next big problem will erupt.

Debugging Difficulty: You have to inspect more than one service to find the source of the problem.
Slow Development Cycles: It's hard to predict the possible effects of making a change.
Low System Stability: Unexpected errors and crashes happen more often.
Increasing Operational Costs: More resources are needed for troubleshooting and maintenance.

How Do You Get Out of This Hell? Paths to a Solution

To escape hidden dependency hell, you have to take a proactive approach. This requires being careful in many areas, from architectural decisions all the way to daily development practices. One of the most effective methods is to make communication between services explicit and standardized.

Using an API Gateway creates a centralized point of control by preventing services from communicating directly with each other. Thanks to this, dependencies between services become more visible and easier to manage. Additionally, event-driven architectures can prevent these kinds of problems by encouraging loose coupling between services.

💡 The Role of the API Gateway

The API Gateway is a mediator that receives all client requests and ensures they're handled by the relevant services. Thanks to this, clients aren't aware of the architecture of the services and dependencies between services are managed better.

Communication Patterns and Standardization

How communication between microservices is done plays a critical role in managing dependencies. Using standardized communication protocols such as RESTful APIs and gRPC allows services to understand each other more easily. These standards help dependencies become more explicit and predictable.

It's also important to clearly define the data formats and messaging schemas the services use. These definitions should be documented and versioned. That way, when one service changes a data format, the other services can adapt to the change or be made aware of it.

Traceability and Observability

Being able to monitor the behavior of all services in the system and the interactions among them is one of the most effective ways to detect hidden dependencies. Observability tools such as centralized logging, distributed tracing, and metrics collection provide valuable information about the overall health of the system.

Thanks to these tools, you can track how a request travels across multiple services, easily understand which service is causing latency or where the error started. This lets you quickly diagnose problems caused by hidden dependencies.

⚠️ The Importance of Observability

Observability is a property that lets you understand the internal state of the system by observing it from outside. In microservice architectures, this property is vital for proactively detecting and solving problems.

Other Methods of Managing Dependencies

Dependency Injection: Providing the dependencies a service needs from the outside makes services more independent.
Circuit Breaker Pattern: When a service repeatedly fails, it prevents the system from crashing by blocking calls from other services to that service.
Service Discovery: Lets services find each other dynamically, which helps reduce static dependencies.
Regular Refactoring: It's important to regularly review the architecture and make improvements aimed at reducing dependencies.

Conclusion: Continuous Effort for Healthier Microservices

The flexibility and speed brought by cloud-based microservices, when not managed properly, can lead to growing complexity and to problems like "hidden dependency hell." Escaping this hell isn't possible with a one-time fix; it's possible only through continuous effort.

Making your architectural decisions carefully, standardizing inter-service communication, using observability tools effectively, and regularly reviewing your system will help you reach healthier, more stable, and more manageable microservice architectures. Remember, a well-designed microservice architecture forms the foundation of your future growth and innovation.