DEV Community: Alan Bonnici

Software Defectrums: The End of an Era?

Alan Bonnici — Thu, 25 Jun 2026 12:58:13 +0000

What I mean by "Defectrums"

Throughout this article I use the term Defectrums as an umbrella for every category of software error. That includes logic errors, runtime errors, dependency and supply-chain vulnerabilities, security vulnerabilities, and specification misalignment, where the software does exactly what it was told to do, just not what it was meant to do. If a category matters to you that I have not listed, mentally add it; the argument that follows applies to all of them.

Software is deterministic, people are not

Start with a deceptively simple observation: software is deterministic. When code executes, it behaves identically every single time. Take a routine that branches on a person's recorded sex: if the value is M it follows the male branch, every single time; if it is F it follows the female branch, every single time. No exceptions, ever.

But watch what happens the moment someone arrives whose record is neither M nor F. The machine does not improvise and it does not panic. It does exactly what it was told to do, which may be nothing useful at all, or something nonsensical, simply because the developer never imagined that input. The determinism held perfectly. The defect was ours: a failure of foresight, baked into the code and executed faithfully forever.

That is the whole problem in miniature, and it matters more than it first appears. If software is deterministic, then in principle every defect in it is knowable, reproducible, and fixable. A flaw is a fixed point in a fixed system. The reason software remains stubbornly broken is not the machine. It is the chain of humans who wrote the code, the compiler, the operating system, and the libraries beneath it. The determinism is in the silicon. The unpredictability enters through us.

Hold on to that idea, because it is the spine of everything that follows: a deterministic system should, eventually, be made flawless. The only question is whether we can remove enough of the human variability to get there.

The making of a programmer

It was not always this messy, because it was not always this widespread or complicated.

In the beginning, computing was a niche industry. Computer engineers were the only people who programmed and operated the mainframes and minicomputers, and the cost of buying and running those machines put them far beyond the reach of any hobbyist. Those machines also had limited storage and processing power, which kept the complexity of any single program in check. And because computer time was both slow and expensive, code was typically desk-checked and reviewed line by line before anyone dared run it. Scarcity enforced a discipline that abundance would later erode.

The microcomputer changed everything. Initially pooh-poohed by mainstream media and the technology establishment as non-serious gadgets, toys that would stay confined to hobbyists while "serious" computing remained the domain of mainframes, these machines quietly put programming into the hands of an ever-growing audience. Ordinary people began writing their own programs, then sharing and selling them, which drew in more buyers, which created more programmers, in a snowball that never stopped rolling.

VisiCalc is the classic example. The 1979 spreadsheet from Software Arts was the "killer app" that drove Apple II sales; many people bought the computer specifically to run it. As John Markoff observed, the machine was effectively being sold as a "VisiCalc accessory." Every computer that entered a home widened the exposure further. Dad's machine became the kids' games console the moment he stepped away, and some of those children started dabbling in code of their own. Programs multiplied to cover every topic imaginable: recipe managers, household budgets, astronomy charts. If you could think of it, someone had probably already written it.

The incumbents had lost. Any hope of keeping the role of the programmer, later rebranded the software engineer, narrow or formally licensed was gone. As the user base grew exponentially, so did the flood of ideas. It was, genuinely, a case of let a hundred flowers blossom. Microcomputers went on to take over the office, and through networking, the internet, and mobile computing the proliferation has become almost comical. Pick the most niche subject you can imagine, knot tying, say, and you will still find an astonishing number of apps, sites, and videos devoted to it alone. Now multiply that by every hobby, profession, and passing curiosity on earth and you begin to sense the scale. Some of this software is specialised, some visual, some gamified. You get the gist.

The downside is that not all software is created equal. Where scarcity once forced rigour, abundance rewarded speed. Quality is bounded by the programmer's skill, by time-to-market pressure, by whatever testing strategy survived the deadline. Plenty of programs written for an audience of one have taken on a life of their own and ended up serving millions. And software is never self-contained. It runs on an operating system, leans on packages to do its work, and each of those components in turn depends on libraries and tools built by someone else.

XKCD's "Dependency", comic #2347, captures this perfectly. Consider FFmpeg, " a complete, cross-platform solution to record, convert and stream audio and video. " Tens of thousands of software programs, including browsers, applications, packages, libraries, and operating systems, and billions of devices, including TVs, streaming boxes, games consoles, and phones, rely on it internally. A defect in one such building block does not stay local. It propagates silently into everything stacked on top.

We have watched this happen, twice in recent memory. In 2021, Log4Shell, a single flaw in Log4j, a Java logging library few outside the trade had ever heard of, opened a remote-code-execution hole in millions of servers overnight and set off a global firefight that ran for months. In 2024 the xz-utils backdoor came even closer to catastrophe: an attacker spent years patiently earning the trust of a burned-out volunteer maintainer, then slipped a hidden backdoor into a compression library threaded through the entire Linux ecosystem. It was caught almost by accident, by one engineer who noticed a login was running half a second slower than it should. That is the XKCD cartoon made real, and the near-miss is more frightening than the hit.

The human in the loop

So the problem with software is the human. We are intelligent but not omniscient; we cannot foresee every possible outcome and guard against it in advance. And even if we imagine a perfectly written program, the compiler that built it, the operating system that runs it, and the libraries it depends on are never guaranteed to be flawless. Any link in that chain can be the weak one.

We have accepted this so completely that essentially all software ships with a warranty disclaimer like the one below. Imagine the engineers behind an apartment block, a bridge, a tunnel, a train, or an aircraft offering their customers anything remotely like it.

    THE SOFTWARE IS PROVIDED AS IS WITHOUT ANY WARRANTY WHATSOEVER, INCLUDING BUT NOT LIMITED TO ANY WARRANTY OF FUNCTIONALITY. YOU RECOGNIZE THAT THE AS-IS CLAUSE OF THIS SOFTWARE LICENSE AGREEMENT IS AN IMPORTANT PART OF THE BASIS OF THIS SOFTWARE LICENSE AGREEMENT, WITHOUT WHICH THE COMPANY WOULD NOT HAVE AGREED TO ENTER THIS SOFTWARE LICENSE AGREEMENT. THE COMPANY AND THIRD PARTIES DISCLAIM ALL WARRANTIES, EXPRESS, IMPLIED, OR STATUTORY, REGARDING THE SOFTWARE, INCLUDING ANY WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, LACK OF VIRUSES, TITLE, AND NONINFRINGEMENT. NO REPRESENTATION OR OTHER AFFIRMATION OF FACT REGARDING THE SOFTWARE SHALL BE DEEMED A WARRANTY FOR ANY PURPOSE OR GIVE RISE TO ANY LIABILITY OF THIRD PARTIES WHATSOEVER.

That disclaimer is not just legal boilerplate. It is an industry-wide admission that we expect our products to be flawed. And those flaws are not academic. Criminal groups and individuals exploit them every day to steal data, install remote-control backdoors, hold systems to ransom, and otherwise turn other people's defects into money.

That acceptance is beginning to change. The European Union's Cyber Resilience Act, in force since December 2024 and biting in full from December 2027, drags software toward the same product-liability regime we already take for granted in bridges and aircraft. Manufacturers of connected products and software sold in Europe will have to build in security, manage vulnerabilities over a product's lifetime, and report the ones being actively exploited, or face real penalties. The "as-is" escape hatch that the industry has hidden behind for years is, slowly, being welded shut. The pressure to make software genuinely defect-free is no longer only moral or commercial. It is becoming law.

With a little help from AI

Here is where the story turns. The same determinism that makes defects inevitable also makes them tractable for a machine that can read code faster and more patiently than any human team.

This is not a brand-new dream, either. For decades, formal methods have let us mathematically prove that a piece of software meets its specification: the seL4 microkernel, for instance, carries a machine-checked proof of correctness, and tools like TLA+ have caught design flaws in systems running at planetary scale. The catch was always cost. Proof at that rigour is so labour-intensive that it stayed locked inside aerospace, chip design, and a handful of safety-critical kernels. What AI changes is the economics, holding out the prospect of proof-grade scrutiny applied to ordinary, everyday code. Anyone interested in diving deeper should read the 1996 Fast Company article They Write the Right Stuff by Charles Fishman, a portrait of the NASA shuttle software team that achieved near-perfect code through sheer discipline, and a reminder that the price of such perfection has always been the thing AI promises to lower.

AI systems built for code can already review software and surface flaws that the original developers missed. The capability is no longer hypothetical. In 2024, Google's Big Sleep, a collaboration between Project Zero and DeepMind, became the first AI agent to find a previously unknown, exploitable vulnerability in real-world software, a memory-safety bug in SQLite, the database engine embedded in billions of devices. Anthropic's Mythos model was reported to identify vulnerabilities in highly sensitive US government systems within hours during a testing exercise. The capability proved sharp enough that, in June 2026, the US government forced Anthropic to pull both its Fable 5 and Mythos models from general availability over national-security concerns. OpenAI, meanwhile, released GPT‑5.5‑Cyber as part of its Daybreak programme, a model explicitly built to find vulnerabilities, validate them in a controlled environment, and develop and test patches.

These frontier models are strikingly good at the full loop: quickly surfacing security flaws, generating working exploits to prove their reasoning, and then rewriting the code to close the hole. It is not hard to imagine a near future in which buggy code is the exception rather than the rule. The "CI" in a CI/CD pipeline could read like this:

Developer commits code written with the help of AI coding agents → an agentic CI pipeline generates scripts to test the code for Defectrums → AI analyses the Defectrums → AI rewrites the code to resolve them → feedback is returned.

I can even imagine the pipeline running itself in reverse: an AI agent that discovers a vulnerability in, say, a package your solution depends on, and then initiates and leads a development cycle to fix it, always under human direction.

But there is a catch worth naming. Today, the very models that find and fix flaws also introduce them. Independent testing in early 2026 found that leading AI coding agents reproduced decade-old security mistakes, with the large majority of their pull requests containing at least one vulnerability. And the tool that can patch a flaw is, mechanically, the same tool that can weaponise it. That dual-use reality, defence and offence sharing one engine, is exactly why governments are nervous, and it shapes everything about the future below.

There is a deeper irony hiding in here. We are proposing to police a perfectly deterministic system with a profoundly non-deterministic one. A large language model does not behave identically every time; ask it the same question twice and you may get two different answers. So who verifies the verifier? An AI's fix can be confidently wrong, can introduce a fresh regression, or can quietly paper over a symptom while leaving the real cause untouched. For now the answer has to be a human, or a second model, reviewing the work, which means the loop never fully closes. We may be trading a world of human defects for a world that needs constant, vigilant checking of the machines we hoped would do the checking.

The Defectrum-free future

So, can we ever reach a world where the only updates we receive are the welcome ones, new features and a fresh interface, rather than yet another urgent patch? I believe it is a realistic possibility. I also believe it will take a very long time, for reasons that have little to do with the cleverness of the AI.

The sheer volume of existing software is astronomical, and we will never fully know who is running what, or where. Thousands of packages are no longer maintained, or are kept alive by a handful of volunteers, the unpaid maintainers in that XKCD cartoon, who lack the time, the money, or both to revisit code they wrote years ago. The dependency problem, whether in code or in the wider supply chain, has to be addressed link by link. The first organised attempts already exist: Google's CodeMender and OpenAI's "Patch the Planet" both point AI squarely at finding and fixing flaws in widely used open-source code. They are a start rather than a solution, but they show the direction of travel.

Then there is the question of who pays. Most of this fragile, forgotten code belongs to no one with a commercial reason to repair it, which is precisely why it rotted in the first place. So why would anyone fund the cleanup? The uncomfortable answer is that the large platforms increasingly have to. Their own products are built on the same open-source foundations, so a hole in a common library is a hole in their stack too; defending themselves means defending the commons. Regulation sharpens the incentive further, as liability regimes like the Cyber Resilience Act turn a neglected dependency into a balance-sheet risk rather than someone else's problem. That covers the popular, widely used components. The genuinely orphaned code, used by a few, owned by no one, watched by nobody, is the part the market will always be slowest to reach, and it is where public funding or shared industry consortia may be the only answer.

Then there is the embedded layer. Enormous amounts of software sit inside hardware that is connected to the internet but locked in a cabinet and never touched again, forgotten, unsupported, or simply left alone out of fear of breaking something that currently works. Below even that lies silicon-level fragility, the Spectre- and Meltdown-class flaws that no software rewrite can fully erase.

AI agents are not perfect, which means human oversight is still necessary. This is the same loop I argued earlier never quite closes, and it is also where I expect the answer to be found. As the models improve, I believe single agents will give way to committees of them, working seamlessly together, cross-checking one another through a quorum-based consensus mechanism, with the human holding the casting vote. A panel that has to agree is far harder to fool than any single member of it. The honest caveat is that a committee can still share a blind spot if its members were trained alike, so oversight grows lighter over time without ever quite reaching zero.

The honest summary of where we stand is the one every engineer reaches for eventually: it's complicated.

Picture the far horizon, though. Suppose these AI Defectrum agent committees become free to use, modify, share, build upon, and learn from everyone's code. Suppose they are woven into every development environment, and suppose all existing software has eventually been refactored or replaced. Does the cybersecurity role then fade away?

In the short term, almost certainly the opposite. The likeliest near-future scenario is a barrage of attacks, as criminal groups get their hands on the same offensive AI capabilities and race to exploit the vast backlog of unpatched, vulnerable software before defenders can reach it. The attacker only needs one open door; the defender has to close all of them. For a while, the asymmetry favours the attacker.

Will cybersecurity ever be written into history?

No. And the reason takes us right back to where we started.

We can make a deterministic system flawless, in principle. We cannot do the same to the human using it. Software may one day be Defectrum-free, but as long as a person sits at the keyboard, there remains a non-deterministic, fallible, persuadable entity in the loop, one that can be tricked, phished, and socially engineered no matter how perfect the code behind the screen.

The era of buggy software may well be ending. The era of the human, gloriously and dangerously unpredictable, is not. And so cybersecurity does not disappear. It simply moves up the stack, from defending the code to defending the one part of the system we were never able to make deterministic: ourselves.

How to install Plex on a Synology NAS

Alan Bonnici — Thu, 11 Jun 2026 07:00:00 +0000

Introduction

In my Ubuntu Plex guide, I documented each action from first principles and kept the write-up beginner-friendly on purpose. I am taking the same approach here.

The Ubuntu guide and this Synology walkthrough point to the same broad destination, but they take different roads. On Synology, Plex is still (almost) the same Plex, but the OS expects you to think in terms of Package Center, Shared Folders, system internal users, and DSM-controlled permissions rather than package repositories and cron jobs.

I have a Plex Pass, and certain functionality — such as remote streaming and hardware transcoding — requires this type of licence. After July 1, 2026 the price of a Lifetime Plex Pass is increasing. This guide is narrated on the assumption that you have one.

Prerequisites and Assumptions

You should have a Synology NAS running DSM and an administrator account. Plex's own NAS guidance states that not every NAS is supported and that CPU power is often the biggest limitation. If you plan to rely on hardware transcoding, verify support beforehand. Plex's NAS compatibility list is the source of truth for predicting Direct Play, software transcoding, or hardware-assisted transcoding capabilities.

Besides installing Plex directly on the NAS, another way to run it is within a Docker container on your Synology. If you would like a video on that approach, drop me a note.

This guide also assumes that you know how to create Shared Folders on your Synology.

Timeline at a Glance

The broad installation flow is straightforward, even if the wording inside DSM changes slightly from one release to another.

The Installation: PMS Available through Synology

You can opt to install the version of PMS that came bundled with your Synology. This is normally not the latest version. You will find it listed in Package Center under the Contributor section.

The Installation: PMS from the Plex Website

If you decide to go with the PMS package directly from Plex, you need to find out which version is compatible with your NAS.

Go to Control Panel → Info Center → General tab and take note of the:

DSM version
Model name
CPU

AI can help identify compatibility, but you need to verify its output. Below is what Google's Gemini returned. As you will see, some of the information is not correct.

Go to https://www.plex.tv/media-server-downloads, select the Plex Media Server tab, and from the drop-down list select your Synology. There are two Synology DSM v7 entries. Choose the one that matches your DSM version. Synology changed package handling in DSM 7.2.2, so Plex had to create separate installers for DSM 7 and DSM 7.2.2+.

Plex may offer multiple packages for different architectures. Choose the one that matches your NAS CPU.
Download the .spk package to your computer.
In DSM's Package Center, click the Manual Install button, select the package you downloaded, and follow the installation process. Keep the defaults.

Installation Successful!

At the start of the installation you have to accept a warning explaining that the package is from a third-party developer and is not verified by Synology. At the end of the installation, a dialogue box appears with the next steps. You cannot copy the text in the box (pity), but you can use a screen-capture tool to save a copy and reference it during the next steps.

PlexMediaServer Application Folder

Once installed, Plex on DSM 7 uses the PlexMediaServer account and data structure. Plex's own documentation states that the Synology DSM 7 data directory lives under /volume1/PlexMediaServer/AppData/Plex Media Server, and its database-repair guidance refers to file ownership under DSM 7 as PlexMediaServer. That is the account you will be granting access to in DSM's shared-folder permissions.

If you navigate to this folder, you will see a notice telling you not to place media files there.

Keeping Plex Up to Date

Unlike the Ubuntu installation — where you can uncomment a line in the apt sources file and let the system update Plex alongside other packages — Plex on Synology does not update automatically. When a new version is available, you will see a yellow up-arrow indicator in the Plex web interface (accessed through your browser). To apply the update, you download the new .spk from the Plex website and install it via Package Center → Manual Install, just as you did the first time.

There are community scripts that automate this process as a scheduled task on the NAS. If you would like details on setting that up, leave me a comment.

Plex Media Folders

The installation process creates a system account called PlexMediaServer. To include media within PMS, this account must be specifically given access rights to your media folders.

There are two types of folders that I normally give Plex access to:

View once and delete
Long term

When creating a Shared Folder, Synology's wizard guides you through the parameters you associate with the folder and the files within it. Remember that certain parameters add considerable overhead; enable them only if you have a genuine need. I will comment on the points I think are important in this context. If your file system is not Btrfs, your settings will vary.

It is far more efficient to control read-only and read-write access by limiting the PlexMediaServer account directly rather than by setting restrictions at the Shared Folder level described below.

Set Up Basic Information

If you would like to manage media from other computers on your network, leave Hide this shared folder in "My Network Places" unchecked.
When you delete a file, should it be retained in Synology's recycle bin?

Enable Additional Security Measures

How should the media you add be stored:

Skip — Creates the shared folder with no extra security layer. This is the standard choice for general-purpose storage and has the least overhead.
Protect this shared folder by encrypting it — Encrypts the folder's contents at rest using an encryption key. You must mount it with the key or passphrase to access files, and it adds performance overhead.
Protect this shared folder with WriteOnce — Applies WORM (Write Once, Read Many) protection. Files cannot be modified, deleted, or renamed for a set retention period. This is generally irreversible for the retention window.

Configure Advanced Settings

Enable data checksum for advanced data integrity — Turns on file self-healing and data scrubbing to detect and repair silent data corruption. The overhead can hurt performance.
Enable file compression — Compresses stored data to save space. It is greyed out here because it depends on data checksum being enabled first (both rely on the Btrfs file system). Most media content is already compressed, so the gain is minimal.
Enable shared folder quota — Sets a maximum size limit (in GB or TB) for the folder to cap how much storage it can consume.

Configure User Permissions

The last screen of the Shared Folder Creation Wizard allows you to set the access rights different accounts have on the shared folder.

Because the PlexMediaServer account is a system account, you need to select it from the System internal user list. From here you can set the access this account will have on that shared folder:

RO means that the media cannot be deleted from within Plex
RW allows you to delete the media using the delete option in Plex

Create a Sensible Folder Structure Before Adding Libraries

Plex's installation and library documentation both emphasise proper structure. Do not point libraries at the root of a drive or volume; create a dedicated share with subfolders such as Movies, TV Shows, Music, Other Videos, or Photos, and point Plex at those. Plex explicitly warns against using a drive root as a library folder because it causes problems.

A tidy Synology-native layout might look like this:

/volume1/PlexMedia/Movies
/volume1/PlexMedia/TV
/volume1/PlexMedia/Other

If your media lives on another volume, replace /volume1 with the correct volume path for your NAS. A dedicated Plex share also makes permission management easier.

If you have media that lives in its own pre-existing shared folder, you need to edit that folder's permissions to grant the PlexMediaServer account the appropriate access rights — RO if you want to ensure the media cannot be deleted from within Plex.

Name Your Files So Plex Can Identify Them

Plex relies on file and folder naming to match your media against its online metadata databases. If files are named inconsistently or ambiguously, Plex may mismatch them, pull the wrong artwork, or fail to identify them entirely. Getting this right from the start saves a lot of manual corrections later.

Movies

The recommended format is:

Movie Name (Year).ext

For example:

/volume1/PlexMedia/Movies/Gladiator (2000)/Gladiator (2000).mkv
/volume1/PlexMedia/Movies/The Shawshank Redemption (1994)/The Shawshank Redemption (1994).mp4

Each film lives in its own subfolder. The year is important because it helps Plex distinguish between films that share the same title (e.g. remakes).

TV Shows

The recommended format is:

Show Name - sNNeEE - Episode Title.ext

Where NN is the two-digit season number and EE is the two-digit episode number. For example:

/volume1/PlexMedia/TV/Breaking Bad/Season 01/Breaking Bad - s01e01 - Pilot.mkv
/volume1/PlexMedia/TV/Breaking Bad/Season 01/Breaking Bad - s01e02 - Cat's in the Bag.mkv

The season subfolder is optional but keeps things tidy.

Key Points

Always include the year for movies.
Always include the sNNeEE pattern for TV episodes — Plex depends on it.
Avoid stuffing quality tags, release group names, or other metadata into the filename. Plex does not need them and they can confuse the scanner.
Plex's own naming guides are at support.plex.tv/articles/naming-and-organizing-your-movie-media-files and support.plex.tv/articles/naming-and-organizing-your-tv-show-files.

Open the Plex Web App and Finish First-Run Setup

You can complete setup by opening a browser and navigating to:

http://<local-nas-ip>:32400/web

Associate Plex Libraries with NAS Shared Folders

Select Add Library and choose the type of media that will be stored there.
Specify the folder on the NAS where that media is located. You can associate multiple folders with a single library, but they must contain the same type of content.
Repeat for other media types.

Adding and Viewing Content

To add media to your PMS, simply copy files into the appropriate folder. Plex will normally detect and index new media automatically, making it appear in the menu.

If this does not happen, you can trigger a manual scan via Settings → Libraries → Scan Library Files.

Once indexed, the media can be played by selecting it.

Performance Tuning for Synology

Synology performance with Plex is mostly about avoiding unnecessary transcoding. NAS devices are typically processor-limited, transcoding is CPU-intensive, and some lower-powered ARM-based devices have transcoding disabled entirely. So the first performance improvement is not a setting at all: keep media in formats your clients can Direct Play whenever possible.

If your NAS appears in Plex's compatibility list as supporting hardware-assisted transcoding, and if you have Plex Pass, enable Use hardware acceleration when available under Settings → Server → Transcoder. Hardware acceleration makes transcoding faster and allows more simultaneous streams, though it can reduce quality or compatibility in edge cases. For the remaining transcoder settings, a sensible starting point is to leave Transcoder quality on Automatic.

Deleting Empty Folders

I have Plex running on both Ubuntu servers and my NAS (DS1525+). From an operational perspective, I have not noticed any difference in performance or quality. One function that is missing from PMS on Synology is the ability to automatically delete directories that become empty when their content is removed.

It is not a critical issue, but it would be convenient if emptied folders were automatically cleaned up, as happens on the Ubuntu version.

Accompanying Video

https://youtu.be/TD_zG7f0UpM

When Your CPU Dies: My Journey with a Defective Intel Core i7-13700K

Alan Bonnici — Tue, 09 Jun 2026 07:00:00 +0000

Introduction

My perfectly stable homelab server gradually descended into chaos — random crashes, segfaults, system hangs — and how months of troubleshooting, community support, and hardware swaps eventually led to one conclusion: the CPU itself was defective from the factory.

If you're experiencing unexplained instability on an Intel 13th or 14th Gen system, this story might save you months of frustration.

Quick Symptom Checklist

If you're seeing multiple of these symptoms, you may be affected:

Random segfaults in unrelated applications
System instability appearing after months of stability
Crashes or hangs under light load or while idle
VMs freezing while still marked as "running"
Issues persisting despite PSU, RAM, or OS checks or changes

The Setup

In November 2022, I built a homelab server with the following components:

Component	Model
CPU	Intel Core i7-13700K (Raptor Lake)
Motherboard	ASUS PRIME Z690-P D4 (Intel Z690, ATX, DDR4)
RAM	Patriot Viper Steel DDR4 3600 MHz C18, XMP 2.0
Cooler	Cooler Master Hyper 212 Black Edition
PSU	Corsair RM-850e
OS	Proxmox VE

The system ran Proxmox VE 7, later upgraded to v8, hosting a mix of Windows, Ubuntu, and Debian VMs, plus a couple of LXC containers. For over a year, everything was rock-solid. NUT (UPS monitoring), LAN bonding, automated backups — all worked flawlessly.

The Trouble Begins: A Timeline

August–September 2024: First Signs of Instability

After upgrading VirtIO drivers from 0.1.248 to 0.1.262, I started seeing x86/split lock detection errors. The entire Proxmox host would crash, taking down all VMs and containers.

Even though I didn’t fully trust that theory, I pointed a finger at the VirtIO upgrade — it was the only change I had made.

What the community said:

"Don't think VirtIO would make the whole host crash... this seems more like a hardware issue with storage or memory"
"Split lock is an indication of your VM doing some weird stuff... generally related to very buggy software/OS or faulty hardware — bad memory, bad CPU, bad power supply"
Suggestions: Run memtest, stress-test the CPU, check thermals, check fans

I disabled split lock detection as a workaround. The crashes continued.

September 2024: Testing Everything

Following community advice, I:

Ran PassMark memory tests — no errors
Updated the motherboard BIOS - I was one version before the latest
Ran stress-ng on the CPU — no errors
Replaced the PSU with a spare one I had.

October 2024: Stability... or So I Thought

The issue appeared to disappear.

On 17 October 2024, I reported back to the Proxmox forum that the system seemed stable. I waited a full month before posting to be sure. I assumed—incorrectly—that the component I had changed was the culprit.

The stability was temporary—the degradation was progressive.

December 2024 – January 2025: New Symptoms Emerge

New problems surfaced:

A Windows Server 2022 VM would hang during backup restarts
QEMU guest agent fs-freeze/fs-thaw commands would time out
VMs appeared "running" but were unresponsive
Console showed improper shutdown states

I tried:

Switching backup modes
Changing machine types (q35 → i440fx)
Updating QEMU

Nothing resolved the issue.

February 2025: The Penny Drops

While continuing to seek help on Reddit, someone pointed me to the Intel 13th/14th Gen instability issue. The symptoms matched perfectly:

Random crashes under varied workloads
Segfaults in unrelated binaries
System hangs after running for days
Progressive worsening over time

It was suggested that I run the Intel Processor Diagnostic Tool (IPDT) (https://www.intel.com/content/www/us/en/support/articles/000005567/processors.html).

This tool is Windows-only. Intel should have created a utility that is OS-agnostic making it something you could run from a bootable USB drive.

Having to jump through hoops to perform a test for the company's product is counterproductive.

Stress tests like mprime / Prime95 didn’t expose the issue.

March 2025: Intel RMA Process

Armed with the information, I hopped over to the Intel Community (https://community.intel.com/) and asked about the process.

At https://www.intel.com/content/www/us/en/support/articles/000057098/processors.html I found all the information to start the process.

You need to provide:

Processor number
ATPO
FPO

If you no longer have the box, you’ll need to remove the CPU and clean off the thermal paste to read the markings.

When the Intel support agent asked me for the following additional information (from Intel's email):

Motherboard: Please include the model and any relevant details.
Power Supply Unit (PSU): Model, wattage, and manufacturer.
Dedicated Graphics Card: Model and manufacturer, if applicable.
Thermal Solution Details (Air Cooler or a Liquid Cooler):
Operating System Details:
Have you tested this processor on a different motherboard, if yes, on which motherboard was it tested?
Have you tested your current motherboard with a different compatible processor, if yes, which processor have you used?

The process was excellent.

I live in Malta, an island at the edge of Europe, where cross-border logistics usually take time.

Key dates:

16 March 2025: Case opened / CPU information provided / collection process initiated
18 March 2025: DHL collected the defective CPU
20 March 2025: Replacement CPU delivered by DHL

Thermal Paste

Make sure you have thermal paste on hand. If you need to read information off the CPU (and using the machine while your case is being processed) and when you mount the new CPU on your motherboard.

Replacement CPU

The replacement CPU arrived in Intel-branded packaging.

Post-Replacement: Problems Resolved

After installing the replacement i7-13700K, all previously experienced problems disappeared. Two months after replacing the CPU, the system returned to the rock-solid stability it had enjoyed during its first year of operation.

The Bigger Picture: Intel's Raptor Lake Defect

My experience was not isolated. It was part of one of the largest CPU reliability crises in recent computing history. Earlier in my troubleshooting process, I had seen references to CPU-related problems, but:

CPU tests passed
in over 40 years I had never encountered a time-delayed CPU defect

What Went Wrong

Intel's 13th Gen ("Raptor Lake") and 14th Gen ("Raptor Lake Refresh") desktop processors suffered from a fundamental defect that caused progressive, irreversible degradation. The issue affected high-performance SKUs — primarily the Core i5, i7, and i9 K/KF/KS variants with the 8P+16E core configuration.

Root Cause: Vmin Shift Instability

On 25 September 2024, Intel employee Thomas Hannaford posted the official root cause analysis on the Intel Community forums (https://community.intel.com/t5/Blogs/Tech-Innovation/Client/Intel-Core-13th-and-14th-Gen-Desktop-Instability-Root-Cause/post/1633239)

Intel localized the problem to a clock tree circuit within the IA core that is particularly vulnerable to reliability aging under elevated voltage and temperature. These conditions lead to a duty cycle shift of the clocks, causing system instability.

Intel identified four operating scenarios that lead to Vmin shift in affected processors:

1. Motherboard Power Delivery Exceeding Intel Guidance

Mitigation: Intel Default Settings recommendations for 13th/14th Gen desktop processors. It is common practice for motherboards to exceed these settings and they have done so for years. It is normally set by default.

2. eTVB Microcode Algorithm Issue

The Enhanced Thermal Velocity Boost algorithm was allowing i9 desktop processors to operate at higher performance states even at high temperatures.

Mitigation: Microcode 0x125 (June 2024)

3. SVID Algorithm Requesting High Voltages

The microcode's Serial Voltage Identification algorithm was requesting high voltages at a frequency and duration that caused Vmin shift.

Mitigation: Microcode 0x129 (August 2024)

4. Elevated Core Voltages During Idle/Light Activity

Microcode and BIOS code were requesting elevated core voltages especially during periods of idle and/or light activity — exactly the condition a homelab server experiences most of the time.

Mitigation: Microcode 0x12B, which encompasses 0x125 and 0x129, and addresses elevated voltage requests during idle and/or light activity periods

Intel confirmed that mobile processors and future product families (Lunar Lake, Arrow Lake) are unaffected by the Vmin Shift Instability issue.

Manufacturing Oxidation (Early Units)

For some early 13th Gen processors (manufactured in late 2022 — exactly when I purchased mine), there was an additional manufacturing defect involving oxidation. Intel confirmed this was identified internally in late 2022 and addressed in production by early 2024, but on-shelf inventory with the defect may have persisted into early 2024.

The Critical Point: Damage Is Irreversible

Once a processor has been exposed to excessive voltage for long enough, the damage to the clock tree circuit is permanent and cannot be repaired by any software update. Microcode patches can only prevent further damage on CPUs that haven't yet degraded — they cannot restore already-damaged processors.

This is why my system's stability gradually worsened over time, and why replacing the PSU only appeared to help temporarily.

Why My Homelab Was the Perfect Victim

Looking at Intel's four identified scenarios, my Proxmox homelab hit the worst-case combination:

Scenario 4 (idle/light activity): A homelab server spends most of its time in light-load or idle states — exactly when the faulty microcode was requesting the highest inappropriate voltages
Scenario 3 (SVID high voltage requests): The constant power-state transitions of a virtualization host (VMs starting, stopping, idling) triggered frequent SVID voltage requests
Early manufacturing (oxidation): Purchased November 2022, squarely in the window for the oxidation manufacturing defect
Always-on operation: Running 24/7 meant maximum cumulative exposure to the damaging conditions

The May 2025 microcode update (0x12F) specifically addressed "systems continuously running for multiple days with low-activity and lightly-threaded workloads" — a near-perfect description of a homelab server.

Intel's Response: A Timeline

Date	Action
Late 2023 – Early 2024	Community reports of instability begin accumulating
April 2024	Intel recommends motherboard manufacturers use "Intel Default Settings"
July 2024	Intel officially acknowledges elevated voltage as the cause
June 2024	Microcode 0x125 released — fixes eTVB algorithm issue
August 2024	Microcode 0x129 released — addresses high voltage requests
August 2024	Intel announces 2-year warranty extension (3 years → 5 years) for affected SKUs
September 2024	Intel identifies root cause (clock tree circuit / Vmin Shift)
September 2024	Microcode 0x12B released — "final" fix encompassing 0x125 + 0x129 + idle voltage control
October 2024	Intel confirms the voltage issue was the sole root cause
November 2024	Class action lawsuit filed in San Jose, California
May 2025	Microcode 0x12F released — addresses edge cases in systems running continuously for multiple days with light workloads

Warranty Extension

Intel extended the warranty from 3 years to 5 years for all affected boxed 13th/14th Gen desktop processors (https://community.intel.com/t5/Mobile-and-Desktop-Processors/Additional-Warranty-Updates-on-Intel-Core-13th-14th-Gen-Desktop/m-p/1620853).

Key points from the announcement:

The extension applies to new and previously purchased processors
Coverage applies to all customers globally
The warranty eligibility period starts on the original purchase date and does not reset if Intel provides a replacement
Intel committed to supporting all customers experiencing instability symptoms through the exchange process

Affected Processors

The instability primarily affects desktop processors with the 8P+16E Raptor Lake silicon:

13th Gen (Raptor Lake):

Core i9-13900K/KF/KS
Core i7-13700K/KF
Core i5-13600K/KF

14th Gen (Raptor Lake Refresh):

Core i9-14900K/KF/KS
Core i7-14700K/KF
Core i5-14600K/KF

Lower-power variants (non-K, mobile) were less commonly affected but not entirely immune.

Lessons Learned

For Users Experiencing Instability

Don't assume it's software. If your system was stable for months and gradually becomes unstable, hardware degradation is a real possibility — especially with Intel 13th/14th Gen CPUs.
The symptoms are deceptive. The crashes manifest as memory errors, storage corruption, split lock violations, segfaults in random binaries — anything that looks like "something else" is broken. This is because the CPU is making computation errors.
Run the Intel Processor Diagnostic Tool / equivalent. Download it from Intel's support site. If your CPU fails, you have clear evidence for an RMA. Remember that a Pass is not a sign that your CPU is not impacted.
Don't waste money replacing other components first. I had a perfectly good PSU to replace but didn't have spare RAM, storage and a motherboard lying about. While testing RAM and storage is reasonable, be aware that a degrading CPU can make other components appear faulty.
Check your warranty status. Intel extended the warranty to 5 years for affected 13th/14th Gen desktop processors. If you purchased after October 2022, you likely have coverage through at least 2027.
Update your BIOS/microcode. If your CPU hasn't yet degraded, microcode 0x12B (or newer) can prevent the excessive voltage that causes damage. Check your motherboard manufacturer's website for the latest BIOS. A replacement is ultimately the best solution if you are eligible.

Conclusion

What started as random crashes turned out to be a widespread hardware issue.

The community played a critical role in identifying the root cause.

People who dedicate time to maintaining forums and helping others are a key part of the ecosystem—thank you.

If you're reading this because your 13th or 14th Gen Intel system is acting up take action. The fix exists, the warranty coverage is there, and Intel's replacement process works. Don't spend months chasing ghosts like I did. The clock is running out and these CPUs will eventually no longer be covered for replacement.

The Zero-Day Exploit Clock Is Ticking

Alan Bonnici — Thu, 04 Jun 2026 10:12:58 +0000

A Personal Introduction

The notion of breaking into a system and making it do something its developers never intended has existed since the early days of computing. My own introduction to computing came through hacking into games — disabling or increasing the number of lives to compensate for my slow reflexes.

Using TD (Turbo Debugger, from Borland — shipped alongside Turbo Assembler and Turbo C/C++), I would painstakingly try to identify the memory location that held the lives counter. Once found, one could either alter the value directly or place a NOP (No Operation) over the conditional jump. It was a task that required patience, determination, and, many times, reams of sprocketed continuous-feed paper annotated to help decipher what the code did.

Discoveries were rarely kept to oneself. You shared them with others on BBSs (Bulletin Board Systems), the dial-up communities that served as gathering places for the technically curious long before the web existed.

BBSs were not just a place to share results; they were a primary source of knowledge. Text files on cracking techniques, annotated memory maps for popular games, and tutorials on x86 assembly circulated freely between boards. You would dial in with a modem, download a collection of .TXT and .NFO files, then spend hours offline studying them. It was a decentralised, informal education system: no textbooks, no courses, just collective knowledge passed between pseudonyms.

The motivation was curiosity. There was no financial incentive and no criminal intent. It was intellectual challenge for its own sake — the digital equivalent of picking a lock just to prove you could.

From Annoyance to Destruction

That harmless hacking eventually evolved into something darker: worms, trojans, and viruses. Initially, these programs were little more than an annoyance, displaying messages on screen or slowing machines down. But the lever was gradually notched towards the harmful and destructive. Rather than simply popping up a message, these programs began deleting files and formatting hard drives.

Early examples like the Brain virus (1986) were relatively benign; it even included the authors' names and phone number. But by the early 1990s, destructive payloads had become common. The Michelangelo virus (1992) overwrote the first hundred sectors of a hard disk. CIH/Chernobyl (1998) went further, attempting to corrupt BIOS firmware and render machines unbootable.

If you are interested in watching a YouTube video of mine from 19 years ago demonstrating the operation of a destructive virus called “Casino de Malte”, head over to https://youtu.be/wiLZAEMsofM. Your hard disk’s fate was determined by the outcome of a game.

The Commercialisation of Hacking

The internet, cryptocurrency, and the dark web created an environment in which malicious actors could come together and organise themselves into profit-seeking entities. Their business model: penetrate systems, steal data, hijack operations, and then seek payment from the victim to rectify the mess they created.

It proved extraordinarily lucrative. Cybercrime is now measured in the trillions. Cybersecurity Ventures estimated that global cybercrime damage would reach $10.5 trillion annually by 2025 — a figure that, if treated as an economy, would make cybercrime the third-largest in the world after the United States and China.

These are not lone hackers in basements. Modern cybercriminal organisations operate with corporate structures — complete with HR departments, customer service teams to “help” victims pay ransoms, software development lifecycles, and even employee performance reviews. Groups such as LockBit and Conti have operated as Ransomware-as-a-Service (RaaS) platforms, licensing their tools to affiliates in exchange for a percentage of the ransom collected.

The Ecosystem: States, Criminals, and Brokers

Today, the malware market is made up of three distinct layers:

State-backed groups — Advanced Persistent Threat (APT) actors funded by nation-states for espionage, sabotage, and geopolitical advantage. Examples include Fancy Bear (Russia), Lazarus Group (North Korea), and APT41 (China).
Private criminal groups — Organisations that carry out penetration attacks, deploy ransomware, and operate data-theft schemes for profit.
Exploit brokers — Companies and individuals that trade in hacking knowledge. Firms such as Zerodium openly advertise bounty prices for zero-day exploits: up to $2.5 million for a full Android exploit chain, for example. This grey market means that vulnerabilities are sometimes sold to the highest bidder rather than reported to the vendor for patching.

The success of these operations often depends on what are known as zero-day exploits: vulnerabilities in software that are not yet publicly known and for which no patch or mitigation exists. The term “zero-day” refers to the fact that the vendor has had zero days to fix the problem before it is exploited.

To complicate matters further, some attacks require chaining together multiple exploits in a particular sequence. This is known as an exploit chain, or attack chain. For example, a modern browser exploit might chain a renderer vulnerability with a sandbox escape and a kernel privilege escalation — three separate flaws linked together to achieve full system compromise.

The Zero-Day Exploit Clock

The website “Zero Day Clock” presents a striking visualisation. It analyses publicly known vulnerabilities and estimates the time between public disclosure of a vulnerability and the first confirmed exploitation in the wild.

CVE gives a publicly disclosed vulnerability a common identifier, but it is only one part of the vulnerability-tracking ecosystem. CWE describes the underlying weakness, CVSS scores technical severity, EPSS estimates exploit likelihood, and CISA’s Known Exploited Vulnerabilities catalogue highlights flaws already exploited in the wild. Together, these systems help organisations decide what to fix first.

Strictly speaking, once a vulnerability is publicly disclosed, it is no longer a zero-day in the narrowest sense; exploitation after disclosure is often described as n-day or one-day exploitation. But from a defender’s point of view, the practical effect is similar: the available response window is collapsing.

The trend is alarming:

That compression — from months to days, and potentially from days to minutes — represents a fundamental shift in the threat landscape. Defenders used to have time to test, prioritise, patch, and monitor. Increasingly, they may have only hours, minutes, or seconds.

The AI Accelerant

The dramatic reduction in exploitation time is being accelerated by artificial intelligence, but AI is only one part of a wider shift. Automated scanning, exploit marketplaces, faster reverse engineering, criminal collaboration, and global infrastructure have all compressed the defender’s response window.

The same AI tools that security researchers use to find flaws in code can also be weaponised by adversaries. Their systems can continuously scour vulnerability databases, security mailing lists, vendor advisories, and even social media for news of a newly disclosed weakness. Once alerted, AI-assisted tools can help to:

Analyse the vulnerability — parsing patch diffs to understand exactly what was fixed and, therefore, what was broken.
Generate exploit code — assisting with proof-of-concept attacks targeting the flaw.
Adapt and mutate — creating variants that may evade simple signature-based detection.
Scale deployment — scanning the internet for unpatched systems and deploying attacks at machine speed.

This is no longer theoretical. Academic research has shown that AI agents can exploit some real-world vulnerabilities in controlled benchmark environments. The barrier to entry has lowered: what once required deep expertise in assembly language and months of reverse engineering can now, in some cases, be partially automated.

AI is also being used offensively for:

Spear-phishing at scale — generating personalised, convincing emails that bypass traditional spam filters.
Deepfake social engineering — cloning voices and video to impersonate executives in business email compromise (BEC) attacks.
Automated reconnaissance — mapping an organisation’s attack surface faster than any human team could.

What Can We Do?

We cannot eliminate risk, but we can significantly reduce our exposure.

For individuals:

Keep all devices updated with the latest security patches, and enable automatic updates wherever possible.
Retire technology that is outdated and no longer receiving security updates. End-of-life software is an open door.
Use multi-factor authentication or passkeys on every account that supports them.
Be sceptical of unexpected communications, even from known contacts. Verify through a separate channel.
Avoid reusing passwords. A single compromised password can become the first link in a much larger chain.

For organisations:

Maintain a rigorous patch management programme. The window between disclosure and exploitation is shrinking towards zero. That means patching can no longer be treated as a slow administrative process. Organisations should:
- Automate patch deployment where the risk profile allows it, especially for standard operating systems, browsers, endpoint software, and widely deployed services. Human approval remains important for high-impact systems, but approval workflows measured in days do not match a threat landscape measured in minutes.
- Prioritise vulnerabilities based on exposure, exploitability, business impact, and evidence of active exploitation. If twenty fixes are waiting, the most exposed and most exploitable systems should move first.
- Maintain an emergency patching route for critical vulnerabilities, with pre-agreed ownership, testing boundaries, rollback plans, and communication paths. The middle of an incident is the wrong time to decide who is allowed to approve a fix.
Prioritise internet-facing systems and known exploited vulnerabilities. These are the systems attackers can reach first and the weaknesses they are most likely to weaponise quickly.
Adopt a zero-trust architecture: assume breach and verify every access request.
Invest in defensive tools that can detect, prioritise, and respond at machine speed. Alerts alone are not enough; the goal should be rapid triage, containment, and recovery.
Conduct regular penetration testing and red-team exercises. These should test not only whether vulnerabilities exist, but whether the organisation can detect, escalate, and respond to them quickly.
Ensure robust, tested backup and recovery procedures. Ransomware is only effective if you have no viable alternative.

The uncomfortable truth is that vigilance alone is no longer sufficient. When exploitation timelines compress to minutes, human reaction time becomes the bottleneck. Organisations need automated detection, prioritisation, containment, and recovery capabilities that can operate at the same speed as the attacks they face.

We must also recognise that each of us can be one link in an exploit chain. A compromised personal device, a reused password, an unpatched home router — any of these can become the entry point that leads to a larger organisational breach.

Closing Thought

In three decades, hacking has evolved from teenagers on BBSs trading game cheats into a global criminal industry: commercialised, professionalised, backed by nation-states, and now accelerated by artificial intelligence.

The zero-day exploit clock is ticking faster than ever. For individuals, that means patching, retiring obsolete technology, using passkeys or multi-factor authentication, and treating unexpected communication with caution. For organisations, it means accepting that manual response is no longer enough.

The question is no longer whether an attack will come, but whether our defences can respond in the seconds we may have left.

Suggested references

Zero Day Clock: https://zerodayclock.com/
Cybersecurity Ventures, 2025 Official Cybercrime Report: https://cybersecurityventures.com/official-cybercrime-report-2025/
CVE-Bench: https://arxiv.org/html/2503.17332v3

"Complexity Ceiling" in Power Apps

Alan Bonnici — Wed, 20 May 2026 08:25:55 +0000

Power Apps is a powerhouse for rapid prototyping within the Microsoft ecosystem, but for complex development, the "seams" are becoming visible. This new analysis explores the "Complexity Ceiling" and the friction points faced by architects and developers.

Core Issues Identified:

Mental Model Mismatch: Chaining nested With blocks to simulate sequential logic.
Tooling Gaps: The need for real breakpoints and execution narratives over simple trace functions.
Governance Hurdles: The "abrupt cliff" of enterprise compliance for makers.

Microsoft has the opportunity to bridge the gap between business analysts and professional engineers by introducing controlled imperative constructs and better code organization.

https://www.alanbonnici.com/2026/05/the-complexity-ceiling-where-microsoft.html

Does your Linux server actually know when the power is failing?

Alan Bonnici — Thu, 19 Mar 2026 11:39:08 +0000

NUT (Network UPS Tools) is the go-to open-source solution for UPS monitoring on Linux — but it takes some know-how to set up properly, especially with hardware that isn't auto-detected out of the box.

This detailed guide covers everything from finding the right driver, dealing with stubborn read-only settings, configuring automatic shutdowns, and setting up email alerts so you know when things go wrong. There's even a section on how to safely test your shutdown logic before a real power cut hits.

Great resource for anyone running a homelab or Linux-based home server. 👇

https://www.alanbonnici.com/2026/03/setting-up-nut-ups-software-on-linux.html

How to configure Postfix to relay mail through Gmail (simple step-by-step guide)

Alan Bonnici — Tue, 17 Mar 2026 10:31:07 +0000

If you run Linux servers that need to send alerts, backup reports, or monitoring notifications, configuring SMTP can be a pain.

I wrote a short guide explaining how to install and configure Postfix with Gmail as an SMTP relay, including:

• Installing required packages

• Configuring TLS

• Setting up Gmail app passwords

• Securing credentials

• Testing email delivery

Good for small servers, homelabs, monitoring tools, and UPS notifications.

https://www.alanbonnici.com/2026/03/install-and-configure-postfix-using.html

Building a Production RAG Server with Ollama, Open WebUI and Chroma DB

Alan Bonnici — Wed, 11 Feb 2026 08:48:39 +0000

I've created a comprehensive guide on building a headless LLM server with RAG capabilities. The tutorial walks through the complete implementation including document ingestion, vector storage, and query optimization.

The setup is production-ready and can be completed in about 30 minutes.

There are optional code sections for those who would like to interact with the model programmatically.

MS PowerApps Wordle Engine HowTo

Alan Bonnici — Wed, 29 Jan 2025 14:05:24 +0000

In this HowTo, I dive into creating a Microsoft Power Apps Canvas Wordle app that checks a guessed word against an answer!

🔍 Features include:

✅ Duplicate detection

✅ Suggestions for improvement

✅ Tips for enhancing Power Apps to bridge the gap between no-code/low-code and procedural approaches!

https://youtu.be/-cz_lun6fo0?si=COXVB5vb2rXboyka