DEV Community: Mike

Passkeys Are the Death of Passwords — What CS Students and Self-Hosters Need to Know Now

Mike — Thu, 23 Jul 2026 02:00:01 +0000

Passkeys Are the Death of Passwords — What CS Students and Self-Hosters Need to Know Now

Passkeys have reached the annoying phase.

That is usually how you can tell a technology is real.

This week a Hacker News thread titled "Passkeys were invented by engineers with zero understanding of consumer brain" blew up. Two years earlier, firstyear's "Passkeys: A shattered dream" did even more damage to the happy marketing story. That post is worth reading because it is not anti-crypto nonsense. It is a technical complaint from someone who actually builds WebAuthn software.

I think both threads are right in the same way. The security model is better. The rollout has been messy.

I use Bitwarden passkeys daily. I also run an Authentik-based login layer in my homelab, with the same nginx auth pattern I have used elsewhere. So this is not a theory post for me. It is the next layer after the transport side I covered in TLS Certificates for Internal Services. Last month was the padlock on the wire. This is the proof of who is at the keyboard.

What a passkey actually is

A passkey is not magic. It is a public-private key pair created on your device.

The server stores the public key. Your phone, laptop, password manager, or security key keeps the private key. When you log in, the server sends a challenge. Your device signs it with the private key. The server verifies the signature with the public key. That is why this matters so much for teaching. It is Cambridge 9618 Chapter 6 authentication and Chapter 17 asymmetric cryptography turned into a login screen.

The old password model says, "send me the secret again." The passkey model says, "prove you hold the secret without sending it."

That difference removes three common password problems immediately.

First, there is no password for a fake site to phish out of you. Second, there is no password database to leak in the usual way because the server stores a public key, not a reusable shared secret. Third, the biometric step stays local. Touch ID, Face ID, Windows Hello, or a PIN unlocks the private key on your device. The biometric is not what the website receives.

If you teach A-Level Computer Science, this is one of the cleanest real-world examples you can put in front of a class. Students learn public and private keys, digital signatures, and authentication. Passkeys put all three in one system they already recognise.

Where passkeys live decides whether they feel brilliant or awful

This is the part vendors do not put on the poster.

A passkey can live in three places.

Platform keychains come first. Apple wants iCloud Keychain. Google wants Google Password Manager. Microsoft wants Windows Hello and its own ecosystem. These are convenient until your device mix stops matching the vendor's fantasy of your life.

Password managers are the second option. Bitwarden's Autofill Passkeys documentation is the reason I take passkeys seriously in everyday use. The passkey sits in the vault, the vault syncs across devices, and the browser or mobile app can autofill it like any other credential. That is the escape hatch from platform lock-in.

The third option is a hardware security key. This is still the cleanest model if you want a separate physical authenticator, but it has real limits. The complaints in GitHub's public passkey beta feedback thread and in firstyear's post keep coming back to the same point: resident key storage is finite, and older hardware keys can fill up fast.

If passkeys only work well when you stay obediently inside one vendor's ecosystem, that is not a solved authentication problem. That is a product strategy.

For homelab users, the best compromise is obvious. Put the passkeys in a password manager you control. I use Bitwarden. If you want the full self-hosted route, that points naturally towards Bitwarden/Vaultwarden-style setups rather than handing your whole login future to Apple, Google, or Microsoft.

The recovery question is the one normal people ask first

Every security enthusiast wants to talk about phishing resistance. Normal people ask a simpler question.

What happens when I lose my phone?

That is not a silly question. It is the whole trust test.

The answer depends on where the passkey lives. Platform sync works if you stay inside the platform family. If your iPhone dies but you still have a Mac or iPad on the same Apple account, the passkey is still there. Password-manager sync works the same way, except it is not tied to one vendor's hardware stack. Bitwarden, 1Password, and similar tools turn passkeys into another item in an encrypted vault rather than another reason to stay loyal to one phone brand.

There is also cross-device authentication, where a site shows a QR code and your phone completes the sign-in. It exists. It works. It is also slower and clumsier than typing a password when you are on a borrowed machine. I would not sell that flow as elegant.

Most services still keep a backup route as well. Recovery codes matter. Backup passwords still matter in plenty of deployments. Passkeys are replacing passwords, but the migration is not finished.

My own rule is simple. I keep passkeys in Bitwarden, I keep recovery codes, and I do not trust a single device with my whole identity. That is the difference between a nice demo and a setup I would actually live with.

The ugly part: passkeys still break in very ordinary ways

This is where the critics have a point.

firstyear's blog post documents Apple Keychain deleting passkeys on multiple occasions. The GitHub feedback discussions are full of the other class of problems: credentials saved on the client but not the server, duplicate entries, Android bugs, full resident-key slots, and devices that need resets before they behave properly.

That does not mean passkeys are a failure. It means the weakest part of the system is now the account and device plumbing around the cryptography.

This is the same lesson I made in my CISA GitHub leak write-up. Security failures are often boring. They are not spectacular cryptographic breaks. They are operational mistakes, storage mistakes, sync mistakes, and recovery mistakes.

So if you are moving to passkeys, do it with your eyes open.

Keep recovery codes.
Do not rely on one device.
Prefer a cross-platform vault if you use mixed hardware.
Expect some sites to implement the UX badly.

That last point matters. Plenty of websites support passkeys in a way that still makes you fight the browser, the phone, and the fallback screen before you are allowed in. The protocol is better than the product design.

What this looks like in a homelab

This is the part that matters to me most.

A lot of homelab services still do not support passkeys directly. Some do OIDC well. Some do it badly. Some have no business being exposed without an auth layer in front of them.

That is why I keep coming back to identity at the edge. Authentik's OAuth2 and OpenID Connect provider docs and Pocket ID's OIDC provider overview both point in the right direction: centralise the sign-in flow, then let applications trust the identity provider instead of rolling their own login logic.

For services that do not speak OIDC cleanly, the reverse proxy can still do useful work. Authentik's current nginx proxy integration docs use the outpost path /outpost.goauthentik.io/auth/nginx, which is the shape I would point people at now:

location / {
    auth_request /outpost.goauthentik.io/auth/nginx;
    error_page 401 = @goauthentik_proxy_signin;
    auth_request_set $auth_cookie $upstream_http_set_cookie;
    add_header Set-Cookie $auth_cookie;
    proxy_pass http://127.0.0.1:11434;
    proxy_set_header Host $host;
}

location /outpost.goauthentik.io {
    proxy_pass http://authentik.company:9000/outpost.goauthentik.io;
    proxy_set_header Host $host;
    proxy_set_header X-Original-URL $scheme://$http_host$request_uri;
    proxy_pass_request_body off;
    proxy_set_header Content-Length "";
}

That is not a full WebAuthn setup by itself. It is the shape of the gate. nginx hands the auth decision to Authentik's outpost, and the application behind it does not need to know anything about passkeys directly.

This is why I think the passkeys conversation makes more sense in homelabs than on random borrowed devices. I control the browser, the phone, the password manager, and the identity layer. That removes most of the chaos people hit in enterprise fleets or public computers.

It also completes the security picture from the TLS post. TLS proves you reached the right server. Passkeys prove the right user is logging in. And if you already liked the least-privilege argument I made in the Cloudflare self-managed OAuth post, passkeys feel like the same idea applied to human sign-in.

Why CS students need to care now

Because this is not optional background knowledge any more.

Students already learn authentication, biometrics, digital signatures, TLS, and asymmetric encryption as separate syllabus ideas. Passkeys are one of the first mainstream systems that make all of them visible in one place.

A good exam question almost writes itself.

Explain how a website using passkey-based authentication is more secure than one using only passwords.

The model answer is short.

For the passkey credential, the server stores a public key rather than a reusable shared secret. The client signs a challenge with a private key instead of sending a reusable secret. The signature proves possession of the key. The biometric or PIN unlock happens locally on the device, so the secret does not leave the client.

That is Chapter 6 and Chapter 17 with a real browser in front of it.

The other reason students should care is practical. Enterprise platforms are pushing hard towards passwordless sign-in, and Microsoft Entra is part of that direction even if the user experience is still catching up. By the time today's students hit serious workplace systems, passkeys will not be an optional curiosity. They will be normal. And if you want a parallel lesson in what happens when self-hosted authentication is treated casually, read my piece on self-hosted AI security.

My view after using them

Passkeys are not an upgrade you install. They are an architectural shift.

That is why the transition feels messy. Passwords were simple, portable, and terrible. Passkeys are stronger, less phishable, and tied to real devices, but they expose every weak assumption vendors made about sync, recovery, and platform control.

I am still on the side of passkeys.

Not because the current UX is perfect. It is not. I am on their side because the old model has run out of road. We have spent years teaching people to make longer passwords, store them properly, avoid phishing, rotate them after leaks, and stop reusing them. That advice still matters, but it is maintenance on a bad design.

Passkeys replace the bad design.

That is why I think they are the death of passwords, even if the funeral is taking longer than the vendors promised.

How to Determine Which Old Drives Are Still Good — SMART Data, badblocks, and Homelab Hardware Longevity

Mike — Mon, 20 Jul 2026 07:40:01 +0000

How to Determine Which Old Drives Are Still Good — SMART Data, badblocks, and Homelab Hardware Longevity

Someone on r/homelab posted a photo of a pile of old drives and asked one simple question: how do I determine which of these are still good? Every self-hoster ends up in the same position. Drives accumulate. Labels fade. One came out of an old desktop, one came from a USB dock, one has been in a drawer for three years, and now you have to decide what you can trust.

I do not make that decision from the drive's age or from one reassuring word in a terminal. I make it from SMART data, the right self-tests, and a very blunt rule: trends matter more than a single PASSED result.

That matters in my own setup. My homelab is not a theoretical diagram. I have a 465.8 GB Gigabyte NVMe boot drive in my Hermes host, a 931 GB P3 SATA SSD in my Proxmox box, and a 1.9 TB Solidigm NVMe handling local-lvm storage. Those are different drive types, different roles, and different failure patterns. The method is the same on all of them: pull the data, read the few numbers that matter, then decide whether the drive is fit for real work, background storage, or the recycling pile.

If you are building a small server around a Mini PC with upgrade room, like I described in my Chuwi Minibook X vs BMAX Pro 8 comparison, this is the step people skip. They happily add an old disk, then act surprised when the machine starts queueing I/O or losing data six weeks later.

SMART data is useful, but only if you read the right parts

The first trap is believing SMART gives you one simple answer. It does not. smartctl -a gives you a wall of attributes, and most of them do not matter in the same way on every vendor.

These are the fields I care about first on an HDD or SATA SSD:

Reallocated_Sector_Ct: sectors the drive has already given up on and remapped
Current_Pending_Sector: sectors the drive is worried about but has not remapped yet
Offline_Uncorrectable: sectors it could not read properly
UDMA_CRC_Error_Count: usually a cable problem, not a dying drive
Temperature_Celsius: context, not panic, but still worth watching
Power_On_Hours: useful background, not a verdict by itself

The mistake I see all the time is someone spotting one non-zero field and deciding the drive is dead. That is not how this works. A drive with stable numbers is a different story from a drive whose counts are growing every week.

This is why I think the overall SMART health line is nearly useless on its own. smartctl -H /dev/sda gives you PASSED or FAILED, and people treat that like an exam result. It is not. A drive can still say PASSED while collecting pending sectors and quietly moving towards the cliff. A drive can still report PASSED even when individual attributes are starting to deteriorate. By the time it flips to FAILED, you are already late.

My rule is simple: save the output, run it again later, and compare. If Reallocated_Sector_Ct sits at the same value for months, that is one conversation. If it grows from 0 to 3 in a week, that drive has started its decline.

smartctl -a /dev/sda
smartctl -H /dev/sda

There is another trap here: vendor interpretation. Seagate, WD, HGST and others do not all present raw values the same way. Raw_Read_Error_Rate is the classic example. On one vendor, a big raw number can be normal. On another, zero is what you expect from a healthy drive. If you copy the wrong interpretation from a random forum comment, you can talk yourself into binning a drive that is fine.

The USB dock problem catches almost everybody once

If you test old drives externally, this is the frustration you are likely to hit first. You run smartctl /dev/sdb and it tells you the device does not support SMART. Then you move the same drive to a direct SATA connection and suddenly the data appears.

The drive was not lying. The USB bridge was.

SMART is an ATA feature, and many cheap USB-to-SATA bridges do a poor job passing those commands through. In that case you need to tell smartctl what kind of translation layer it is dealing with.

smartctl -d sat -a /dev/sdb

Some bridges need device-specific flags such as -d usbjmicron or -d usbprolific, but -d sat is the one that fixes the problem most often. This is one of those details that wastes an hour the first time you hit it, because most tutorials quietly assume the drive is connected internally.

The tests that actually tell you something

If you only run -H, you are barely checking the drive. The useful test sequence is this:

smartctl -t short /dev/sda
smartctl -t long /dev/sda
smartctl -l selftest /dev/sda

The short test is the quick screen. It takes a couple of minutes and is worth doing on anything you have just pulled out of a drawer.

The long test is where the real evidence starts. That is the one that scans the surface properly and gives you a better chance of catching unreadable areas before you trust the disk with backups, media, or VM images. On large drives it takes hours, not minutes. On USB enclosures, it can also fail quietly if the enclosure sleeps or times out, which is why I always check -l selftest afterwards instead of assuming the command finished cleanly.

This is the same mindset behind my minimal homelab build: assume components fail eventually, and design around that reality. In that post I made the point that I want a server I can rebuild quickly if an SSD dies. SMART checks are the maintenance version of the same philosophy. You are not trying to predict the future perfectly. You are trying to avoid preventable surprises.

badblocks is useful, but it is not the answer for every drive

People recommend badblocks like it is a universal health check. It is not.

For an old HDD you are about to reuse, badblocks can be very helpful. For an SSD, the wrong mode is a bad idea. For NVMe, it is the wrong tool entirely.

The two modes most people see are:

badblocks -n -sv /dev/sda
badblocks -w -sv /dev/sda

-n is the non-destructive read-write mode. It preserves the data already on the HDD, but it still writes across the disk, takes ages on large drives, and is something I would run only on an unmounted disk I am deliberately testing.

-w is destructive. It writes patterns across the disk and checks them. It is thorough, but it also erases the drive. More importantly, it is the wrong thing to do on SSDs because you are burning write cycles for no good reason. It is also a poor fit for SMR drives, where the translation layer can make the process painfully slow and occasionally misleading.

My rule here is straightforward:

Old HDD with no data you care about: badblocks -w is fine if you have time
Old HDD you want to keep intact: badblocks -n
SATA SSD: skip destructive badblocks, use SMART and vendor tools instead
NVMe SSD: use NVMe health data, not badblocks

That last point matters more now than it did a few years ago. My Hermes and Proxmox boxes both rely on NVMe somewhere in the stack. For those drives I care about Percentage Used, Media_Errors, and Critical_Warning far more than I care about old HDD-style surface testing.

nvme list
nvme smart-log /dev/nvme0n1

One number that gets misread a lot is Percentage Used. If an NVMe reports 50%, that does not mean the drive is half-dead in the dramatic way people imagine. It means the drive estimates it has used half of its write endurance. In a homelab boot role, that can still mean years of service left.

The one SMART problem that deserves immediate attention

If I had to pick one field that changes my tone fastest, it is Current_Pending_Sector.

A pending sector is a sector the drive is struggling to read reliably but has not remapped yet. That means you are in the awkward middle ground where the drive knows there is trouble, but it has not fully resolved it. Sometimes a forced read or write will clear the issue. Sometimes it will confirm the damage.

A simple read pass is often the first thing I would try. Before you run it, double-check the device name. dd is read-only in this example because the output is /dev/null, but disk commands are still the kind of thing you do not want to aim at the wrong drive.

dd if=/dev/sda of=/dev/null bs=1M status=progress

That forces the drive to read the entire surface, which makes it a useful diagnostic first step. After that, run smartctl -a again and see whether the pending count dropped, stayed the same, or grew. If the count is still there, the next meaningful change usually comes only when the drive manages a successful write to the bad area and remaps it. If it stayed or increased, I stop pretending the drive is trustworthy.

This is also where backups stop being a nice idea and become the actual point. In my self-hosting toolkit post, I mentioned backing up databases to an external drive with a simple cron job. That matters here because old disks are not where you should discover your backup strategy is imaginary.

My keep, monitor, or retire framework

This is the decision table I would actually use.

Result	Verdict	What I do
SMART attributes normal, self-tests clean	Keep	Fine for normal use
Reallocated sectors present but stable	Monitor	Use for non-critical work, check again later
Reallocated sectors growing	Replace soon	Do not trust it long term
Pending sectors above zero	Rescue or retire	Re-test after a read/write pass
Offline uncorrectable above zero	Retire	The drive has recorded unreadable sectors
UDMA CRC errors rising	Check cable first	Often not the drive itself

Backblaze's annual drive statistics line up with this. Reallocated sectors and pending sectors are far better warning signs than raw age alone. A 30,000-hour drive is not automatically worse than a younger one. A stable old enterprise disk can be less worrying than a younger consumer drive whose counts have started moving.

That is also a useful lesson for students. Storage hardware sounds tidy in the syllabus and messy in real life. In class you learn what sectors, flash cells, fragmentation and endurance mean. In the lab, you learn when those ideas turn into real decisions about remapping, wear, and whether a warning is just background noise or the start of a failure curve.

That is why I keep tying homelab work back to the classroom. The theory only really sticks when you have to decide whether the drive in front of you is safe enough to keep using.

Final point: stop asking if the drive is good or bad

That is the wrong question.

The useful question is this: what is the trend, what role do I want this drive to play, and how much failure am I prepared to tolerate?

A clean SMART report plus a completed long self-test is enough for me to keep using a drive. Stable minor warnings mean monitor it and keep it away from anything important. Pending sectors, growing reallocations, or uncorrectable errors mean I stop negotiating with it.

That is the method. Pull the SMART data. Run the real tests. Watch the trend. Then decide whether the drive belongs in your Proxmox node, your backups shelf, or the bin.

SSH Honeypot Live — What Watching Bots Attack a Server Teaches CS Students About Network Security

Mike — Sun, 19 Jul 2026 02:00:01 +0000

SSH Honeypot Live — What Watching Bots Attack a Server Teaches CS Students About Network Security

Every internet-connected server lives in a shooting gallery.

Open https://honeypotlive.cc/ for a minute and the point lands immediately. The screen fills with source IPs, login attempts, usernames, guessed passwords, and commands being fired at a fake Linux box by machines that think they have found a real one. There is no drama needed here. This is just what the internet does to exposed SSH.

That is why I like the project so much as a teaching tool. Cowrie, the Python SSH and Telnet honeypot behind it, strips the theory out of network security and replaces it with evidence. Students do not have to imagine a brute force attack. They can watch one. They do not have to memorise that weak credentials are risky. They can see bots trying root, admin, pi, ubuntu, 123456, and password in real time.

I use SSH constantly, whether that is tunnelling into a box, checking a service, or fixing something late at night. I wrote about the practical side of that in my SSH tunnels guide. The security side is less comfortable. If you leave SSH exposed to the internet, bots start work long before you have finished your coffee.

I think that is one of the most useful lessons a CS student can learn early: the question is not whether a service will be probed. The question is what you put in front of it when that probing starts.

What the honeypot is actually showing you

The attraction of honeypotlive.cc is that it does not hide the messy bits. You see the boring repetition that real attack traffic is built from.

Most of it is not clever. It is industrial.

A bot connects, guesses a username, guesses a password, then moves on. Another one does the same thing from somewhere else. Then another. On a busy public SSH honeypot, that pattern keeps repeating all day. Community data around Cowrie deployments regularly shows root as one of the most targeted usernames, with admin, pi, ubuntu, test, and user close behind. The passwords are exactly as bad as you would expect: 123456, password, admin, 12345, 1234, root.

That is useful because students tend to imagine attackers as individuals picking targets one by one. Most early attack traffic is not personal. It is automated scanning plus credential spraying at scale.

Cowrie records all of that. Every login attempt. Every command. Every file transfer if the attacker gets that far. That gives you something far better than a textbook paragraph. It gives you data you can sort, count, graph, and argue about.

My own logs tell the same story from the defensive side

The live honeypot is the public demo. My own security logs are the quieter version of the same lesson.

On my own setup, the Hermes host at 192.168.1.117 is a private LAN machine, and public-facing services reach it through Cloudflare Tunnel rather than a directly exposed SSH port. My Proxmox node at 192.168.1.25 is also tucked away properly. That is deliberate. I use Cloudflare Tunnel and other layers in front of services because exposed admin ports are an invitation.

When I wrote about the CISA GitHub leak and homelab security, I showed the fail2ban output I actually use on Linux boxes that need protecting:

Status for the jail: sshd
|- Filter
|  |- Currently failed: 3
|  `- Total failed:     19
`- Actions
   |- Currently banned: 1
   `- Total banned:     4

That is the same story as the honeypot, just from the side that is trying to keep the gate shut. Failed attempts accumulate. IPs cross a threshold. Bans get applied. The traffic is not theoretical and it is not rare. I covered that defensive playbook in my CISA GitHub leak post. The honeypot is the visual proof of why that playbook exists.

This is the part students remember. "Defense in depth" sounds vague until they can match it to a sequence of events. A scan finds the host. SSH answers. Credentials get guessed. fail2ban notices. The firewall limits what else is reachable. An authentication layer stops the next mistake becoming a breach.

That chain is much easier to understand when each step leaves a visible trace.

How bots find a fresh SSH server so quickly

The speed shocks people the first time they see it.

Put a basic VPS online with SSH exposed and it does not stay undiscovered for long. Internet-wide scanners such as Shodan and Censys are built to catalogue exposed services, and tools like masscan can sweep huge parts of IPv4 at absurd speed. That is how fresh SSH services get noticed and indexed so quickly. The failed logins usually come from a different layer of cheap brute-force bots and credential-spray scripts that follow once exposed SSH is easy to find.

This is also where weak security advice starts to fall apart.

People sometimes talk about changing the SSH port as if it is either magic or useless. It is neither. Against a determined scan that checks every port, obscurity is thin. Against the enormous amount of lazy automation that only bangs on port 22, it still cuts noise. Students should understand both truths at the same time. Network security is full of that kind of answer: one measure is not enough, but one measure can still be worth having.

That is also why I keep pointing people toward stronger patterns such as tunnels and proper access layers. If you need a cleaner model for that, my Cloudflare self-managed OAuth write-up and my self-hosting toolkit post both make the same point in different ways: do not expose admin surfaces casually when a better route exists.

The hidden lesson: even the honeypot has failure modes

A honeypot is a security tool, but it is still software, and software still breaks in ordinary ways.

That is one reason I would use this topic in class. Honeypots also teach the ordinary systems failures that sit underneath security monitoring.

A Cowrie deployment in Docker, for example, can fail silently if the log directory is bind-mounted with the wrong ownership. The container runs, accepts SSH sessions, and looks healthy enough. Meanwhile the logs do not get written because the cowrie user inside the container cannot write to the host directory. That is exactly the kind of bug students miss at first because the service appears to be working.

Busy honeypots also create very normal infrastructure problems. JSON logs grow fast. A mid-size public deployment can collect thousands of login attempts per day, and every attempt is another event written to disk. SFTP capture adds a different problem again: now the system is storing whatever malware, tools, or junk an attacker uploads. One careless setup on a cheap VPS and your "monitoring box" becomes a disk-space lesson.

Containment matters too. Cowrie emulates a shell and filesystem well enough to keep attackers interested, which is the whole point. But that also means isolation is not optional. Run it in Docker or a separate VM. Do not place a honeypot beside production services and pretend that is good enough. I have already written about sensible homelab separation in my Proxmox homelab setup post. The same rule applies here: if a security tool can attract hostile traffic, it deserves its own blast radius.

Why this maps neatly onto the CS syllabus

This is the rare security topic that lands in both the logs and the exam specification.

Cambridge IGCSE Computer Science already asks students to understand brute force attacks, hacking, malware, data interception, and security measures. A live SSH honeypot gives them a concrete picture for several of those terms at once. The repeated login attempts are brute force. The guessed credentials show why weak passwords fail. The source IP churn hints at distributed infrastructure. The fail2ban response is a clear example of an automated security measure.

It also turns networking topics into something less sterile. Ports stop being numbers in a table and become doors that attract traffic. IP addressing stops being a diagram and becomes the reason a scanner can find a box on the public internet. Encryption stops being an abstract property and becomes the awkward reality that attackers are trying to abuse a legitimate secure protocol rather than break the maths underneath it.

That is why I would rather teach this than another tidy slide deck full of definitions. Students learn security terms faster when the system misbehaves in front of them.

The simplest experiment is still the best one

If you want the lesson without building a whole project, keep it small.

Spin up a cheap VPS. Lock it down properly first: use SSH keys, disable password authentication, and never test with real or reused credentials. Then install fail2ban.

sudo apt install fail2ban
sudo systemctl enable --now fail2ban

Then watch the logs:

grep "Failed password" /var/log/auth.log | tail -20
sudo fail2ban-client status sshd

If you want the fuller demonstration, run Cowrie in isolation. The current Cowrie README on the main branch still uses this quick-start container example:

docker run -p 2222:2222 cowrie/cowrie:latest

Port 2222 makes a cleaner demo and avoids throwing a default SSH service straight onto 22, but it also means you will not attract the same volume or pattern of lazy bot traffic that a public port-22 service gets.

Now compare what your own logs show against what honeypotlive.cc streams publicly. That is a better network security lesson than most classroom worksheets because it answers the question students usually care about first: "What is actually happening out there?"

And the answer is blunt. Bots are already trying. They do not care whether the box belongs to a student, a hobbyist, or a government agency. The traffic is the same. The difference is whether you left the door open, whether you can see the attempts, and whether your defences are awake when they arrive.

That is why I like SSH honeypots so much. They make network security stop sounding like advice and start looking like evidence.

The $20 Server — What Extreme Budget Computing Teaches CS Students

Mike — Sat, 18 Jul 2026 02:00:01 +0000

The $20 Server — What Extreme Budget Computing Teaches CS Students

The most educational server in the room is usually the worst one.

Give students unlimited cloud credits and they can launch machines all afternoon without learning much about limits. Give them a battered $20 office desktop with 4 GB of RAM and an old hard drive, and suddenly every operating system lesson becomes real. Containers fight for memory. Disk latency stops being a diagram and starts being the reason the machine feels stuck. "Virtual memory" is no longer a sentence to memorise for an exam. It is the pause they can feel in their hands.

I think we make systems teaching too comfortable. Comfortable hardware hides the lesson.

My own homelab is not running on scrap. The Hermes box here is an i7-1255U machine with 24 GB of RAM. My Proxmox node is an i5-8260U with 16 GB. Those are sensible little servers. They are reliable, quiet, and good at real work. But they are almost too polite as teaching tools. They have enough headroom to forgive bad habits. A $20 machine does not forgive anything, and that is exactly why it teaches so well.

A cheap box makes operating systems visible

A lot of CS students meet operating systems through neat definitions: process scheduling, paging, interrupts, secondary storage, background services. The problem is that on decent modern hardware, many of those ideas stay invisible.

On a constrained machine, they stop being invisible.

Open a few browser tabs, pull a container image, start a database, and then watch the machine try to keep up. That is process management with consequences. If a lesson asks a student to explain how memory pressure pushes pages out of RAM and into virtual memory, the answer is no longer abstract. They have already seen the machine slow down when it happens.

That is why I like this topic so much. A cheap server is not just a budget trick. It is a working model of the syllabus.

What a $20 server can actually do

People either romanticise these little boxes or dismiss them. Both are wrong.

A $20 used desktop can do useful work. It can run Linux properly. It can host a reverse proxy, a small Python app, a static site, a local Git service, a DNS blocker, or a couple of light Docker containers. It can teach SSH, permissions, package management, logs, services, backups, and the discipline of keeping a machine tidy.

What it cannot do is pretend to be a modern all-in-one homelab. This is not the box for 25 containers, heavy VMs, Plex transcoding, or local AI models. If that is your goal, skip the fantasy and buy stronger hardware. I wrote about that trade-off in Chuwi Minibook X vs BMAX Pro 8 — Which Budget Homelab Machine?. If you can stretch to a proper mini PC, life gets easier very quickly.

But easier is not always better for learning.

A small machine forces choices. Do you really need a database for this project, or will SQLite do? Do you need another container, or have you just become lazy because Docker Compose makes laziness easy? Do you actually understand what is running on the box, or have you stacked services until the machine became a mystery?

That is systems thinking, and expensive hardware often delays it.

RAM is where the lesson starts

If I had to pick one concept that cheap hardware teaches best, it would be memory pressure.

Students hear that RAM is fast and disks are slow. Fine. They hear that virtual memory uses secondary storage when RAM runs short. Also fine. Then they sit an exam and write the sentence back.

A 4 GB machine makes them understand it.

Run these before and during the experiment:

free -h
vmstat 1
dmesg | grep -i oom

Use free -h for a memory snapshot, vmstat 1 to watch activity live, and the dmesg command afterward if a process disappears.

That is a better lesson than three slides full of definitions.

When available memory shrinks, swap activity rises, and the system eventually cannot satisfy another memory request, the kernel may invoke the OOM killer. Then the student can see the operating system making trade-offs in real time. Suddenly the OOM killer is not a strange term from a textbook. It is the reason the database vanished. The machine is telling them, very bluntly, that every service has a cost.

This is also where bad habits get exposed. Students love overbuilding. They install a full stack because a tutorial told them to. On a roomy cloud VM, that habit survives for months. On a $20 box, it gets punished in an hour. Good. Better to learn restraint early.

I would rather teach a student to respect 4 GB properly than let them waste 32 GB carelessly.

Old disks teach I/O bottlenecks without any speech from me

RAM usually fails first, but the disk is where the machine becomes honest.

A cheap server often comes with the sort of storage that makes modern users impatient within minutes. That sounds like a drawback. In a classroom, it is gold.

Start an update, unpack a container image, and do a few file operations at the same time. On a modern NVMe drive, the box may shrug. On an old mechanical disk, the machine starts queueing work badly enough that everyone notices. The operating system is still functioning, but it is clearly waiting on storage.

That moment matters because it connects three ideas students often treat separately: throughput, latency, and contention. They stop thinking of storage as a silent cupboard where files live and start seeing it as a shared resource that can become the bottleneck for everything else.

It also teaches one of my favourite engineering rules: when a cheap upgrade fixes a structural bottleneck, do that first. On old office PCs, that usually means fitting even a modest SSD before doing anything clever. A tiny SSD will do more for the learning experience than another hour spent pretending a dying hard drive is "good enough."

Weak CPUs teach scheduling and background noise

CPU scheduling is another topic that sounds clean in notes and messy on real hardware.

On a budget desktop, background work stops being background in the magical sense. You can feel when the box is busy. A package install, a container restart, a compression job, and a browser tab all want time on the processor. The machine cannot hide the queue.

That is why I like giving students boring commands on weak machines. Open top. Start something heavier than the box likes. Watch what gets CPU time and what has to wait. Talk about foreground responsiveness, daemon work, and why "the computer is slow" is often really "the scheduler is making a sensible choice under pressure."

A strong machine can mask poor judgement. A weak one teaches prioritisation.

That lesson scales upward too. Once students understand why a small box struggles, they make better decisions later on bigger hardware. They stop assuming another service is free. They ask what it will cost in CPU time, memory, and storage pressure. That mindset matters far more than memorising a definition of round-robin scheduling and forgetting it a week later.

This is why I do not worship free cloud credits

Cloud labs have their place. I use them when they fit. But I do not think they are a good first teacher for systems thinking.

A cloud dashboard gives students the illusion that resources arrive by menu. Two vCPUs here. Eight gigabytes there. Another volume if you feel like it. The machine appears from nowhere and, just as importantly, the machine disappears from the student's mind. Hardware becomes abstract too early.

A cheap local box does the opposite. It makes every resource visible. It makes waste obvious. It teaches that there is no such thing as "just install one more thing" when the machine has no spare room.

That is a much healthier starting point.

If a student learns on constrained local hardware first, then moves into cloud platforms later, they usually make better choices. They size services more sensibly. They monitor memory sooner. They understand why idle background tasks matter. They respect logs and backups because they have already broken a real machine once or twice.

That is a better outcome than teaching someone to click "bigger instance" every time they hit trouble.

What I would tell students to buy in Thailand

The exact listings change all the time, so I would not pretend there is one magic model. But when I say "$20 server" in Thailand, I mean the sort of ex-office Dell OptiPlex, HP ProDesk, or Lenovo ThinkCentre box that turns up around ฿700-฿1,000 with 4 GB of RAM and a hard drive still inside it. That is not glamorous hardware. It is exactly the point.

Buy ex-office hardware, not somebody else's failed gaming experiment.

Old office desktops are boring, and boring is what you want. They are easier to service, usually cleaner inside, and more predictable than mystery machines built from leftovers. Ask the seller to boot into BIOS. Check the RAM amount properly. Check whether storage is a hard drive or an SSD. Listen for ugly fan noise. Look for missing screws, broken front ports, and proprietary power supplies that will annoy you later.

If the machine only has a hard drive, budget for a small SSD immediately. If it has 4 GB of RAM, treat that as a teaching box, not a forever server. If you can find 8 GB cheaply, take it. That is the point where Linux starts to feel less like punishment and more like a platform.

And if your budget can stretch beyond the bargain-bin tier, do it deliberately. A modest mini PC is still a far better long-term homelab host. My piece on minimal homelab design with a mini PC and Docker Compose is the route I would recommend once the student understands the basics. For students who want the journey from first Linux box to self-hosting habits, High School to Homelab — A Student's Linux Journey is the right mindset.

The classroom exercise I would actually run

If I were teaching this as a practical lesson, I would keep it brutally simple.

Install Debian. Enable SSH. Add Docker. Run one lightweight web service. Then add one more. Keep free -h, vmstat 1, and journalctl -f open in separate terminals before and after. Now ask the students what changed and why. If the box actually falls over, run dmesg | grep -i oom and make them explain that too.

That is enough.

Do not start with Kubernetes. Do not start with a dozen containers. Do not start with theory-heavy diagrams about "modern distributed systems." Start with one weak machine and one extra service too many. The operating system will do the explaining for you.

That is the real gift of budget hardware. It turns hidden computer science into visible computer science.

What the machine teaches

The point of the $20 server is not that everyone should chase the cheapest hardware on earth. The point is that constrained hardware makes students look at the system instead of waving at it.

On a box like this, they check memory before blaming the app. They notice disk wait before reinstalling Linux. They learn very quickly that background services are not free and that every extra process is spending somebody's RAM, CPU time, or storage I/O.

That is why I would put a grubby used desktop in front of a CS student before I handed over a big cloud budget. When the machine only has a little room to spare, the operating system stops being chapter three in a textbook and starts acting like a real machine with limits.

GitLost: How AI Agents Can Leak Your Code Through a Single GitHub Issue

Mike — Fri, 17 Jul 2026 02:00:00 +0000

GitLost: How AI Agents Can Leak Your Code Through a Single GitHub Issue

A public GitHub issue should not be able to pull text out of your private repositories.

With a dangerously over-permissioned GitHub Agentic Workflow, that is exactly what Noma Labs demonstrated.

Their July 2026 write-up showed a simple but nasty chain: an attacker opens an issue in a public repository, an AI agent reads it as part of an automated workflow, the agent reaches into other repositories it can access, and the results come back as a public comment. No stolen login. No insider access. No clever memory corruption. Just a workflow that trusted the wrong text.

I run Hermes agents against GitHub-adjacent workflows on my own kit, and this is the class of bug I worry about far more than model benchmark scores. Once an agent can read one place, write somewhere else, and decide its own next step in between, your problem is no longer "AI output quality." Your problem is access control.

My view is blunt: if an agent can read private code and post publicly in the same run, the design is already wrong.

What GitLost actually showed

Noma Labs published GitLost on 6 July 2026. The attack targeted GitHub's new agentic workflow pattern: Markdown-defined workflows tied to GitHub Actions, with an AI agent reading issues, calling tools, and replying automatically.

The dangerous part was not "AI" in the abstract. It was the permission mix.

According to Noma's write-up, the vulnerable pattern looked like this:

a public repository in the same GitHub organization as private repositories
an automated workflow triggered by issue activity such as issues.assigned
an agent that reads the issue title and body
permission for that agent to read other repositories in the organization
permission for that same agent to post back into the public issue

That combination is enough.

Noma's proof of concept showed the agent fetching README.md from both a public repo and a private repo, then posting the contents back into a public comment thread where anyone could read them. Their write-up says the leaked files came from repositories named poc and testlocal, and that the workflow was triggered after the public issue was assigned.

That matters because it turns a boring workflow event into a trust-boundary failure.

A lot of people still hear "prompt injection" and imagine someone typing "ignore previous instructions" into ChatGPT for fun. This is not that. This is a live automation path with real repository permissions attached.

The attack is simple because the system is over-trusted

The rough sequence looks like this:

An attacker opens an issue in a public repository.
The issue looks harmless enough to survive normal human scanning.
The workflow assigns or otherwise triggers the agent.
The agent reads the issue body as part of its input.
Hidden instructions in that issue push the agent to read other repositories and publish what it finds.

Noma says one small wording change, adding the word "Additionally", was enough in their testing to push the model around GitHub's intended guardrails and make it reframe the answer instead of refusing it.

That is the part people need to sit with.

This was not a case of the attacker breaking GitHub authentication. The attacker used the workflow exactly as exposed. The workflow itself bridged public input and private data too freely.

If you teach computer science, the classroom analogy is straightforward. This is like letting a student write a note on the whiteboard, having an assistant treat that note as an instruction sheet, then letting the same assistant walk into the locked filing cabinet and pin the contents back on the classroom door.

The filing cabinet is not the weak point there. The procedure is.

Why this matters even if you do not use GitHub's exact setup

Most people will read GitLost and ask, "Am I affected if I do not use GitHub Agentic Workflows?"

Maybe not by this exact proof of concept.

But the broader lesson applies to every agent stack, including self-hosted ones.

I wrote recently about why local AI matters and about securing self-hosted Ollama. Running your models locally does help with privacy. It stops your prompts and code from automatically leaving for a cloud provider.

It does not solve this problem.

GitLost is not mainly a model-hosting problem. It is an orchestration problem.

If your agent can:

read public issues or comments
access private repositories, secrets, or internal docs
open pull requests, comments, emails, or tickets automatically

then you have built a system where untrusted text can influence trusted actions.

That is the trust boundary that matters.

I see the same misunderstanding in a lot of AI tool conversations. People obsess over which model is smartest and ignore which tool has the broadest token, the loosest repository scope, or the most dangerous default action. That is backwards. In practice, the model is often the least interesting part. I made the same argument in my post on multi-agent debate and AI architecture: the intelligence people admire usually lives in the orchestration, and so do the failure modes.

The real mistake: one agent, too many powers

The cleanest way to think about GitLost is this: one actor in the system was allowed to do too many kinds of work.

It could inspect untrusted public text.
It could fetch private information.
It could speak publicly.

That is too much authority for one automated step.

If I gave a student account permission to read every exam paper, every answer sheet, and every staff note, I would not also let that same account publish messages to the whole class without review. Yet that is roughly what some agent workflows are doing.

And this is why GitLost is a better lesson than a lot of headline-friendly AI panic. It exposes a design error you can actually fix.

Five changes that reduce this risk fast

None of these make an agent workflow "safe." That word is too absolute for security work. They do reduce the attack surface.

GitHub Agentic Workflows are read-only by default and separate write operations through safe outputs, but those controls cannot compensate for an agent that has unnecessarily broad read access and is allowed to publish sensitive output.

1. Scope each agent to one repository whenever possible

Cross-repository access is convenient, but it is also how a public issue in one repo turns into a leak from another.

If a workflow only needs access to the repository where the issue was opened, give it only that. If a second private repository is genuinely required, make that an explicit separate step with its own token and its own review point.

This is the same least-privilege logic that applies to any secret or service account. If the agent does not need the key, do not hand it the key.

2. Split public-writing steps away from private-reading steps

This is the biggest architectural fix.

Do not let the same workflow run both of these operations automatically:

read internal or private material
publish directly to a public thread

If you need both, add a gate between them. Make the first step collect data into a private review channel. For sensitive workflows, require human approval before anything goes back out publicly.

Yes, this slows the workflow down.

Good.

Security controls that cost nothing usually protect nothing.

3. Treat issues, pull requests, and comments as untrusted input

Every issue body, PR description, code comment, changelog, and Markdown file the agent reads should be treated as hostile until proven otherwise.

That does not mean you stop using automation. It means you stop pretending user-controlled text is just data. In an agentic system, text is also control surface.

This is why I still like a simpler self-hosted LLM setup with Docker, Ollama, and Open WebUI for a lot of personal work. A local chat interface that helps me think is a very different risk profile from a workflow agent with repository permissions and public posting rights.

4. Use narrower tokens and installation scopes

Org-wide access is comfortable for setup and dangerous in production.

Use repository-specific installation permissions where you can. Separate read access from write access. Separate code access from issue-management access. Separate internal automation from external-facing bots.

The goal is not elegance. The goal is blast-radius reduction.

When something goes wrong, you want one repo affected, not twenty.

5. Log the agent's tool calls like you would log an admin session

If an agent fetches a private file because a public issue nudged it there, you want a trail.

Log which trigger fired.
Log what repositories were touched.
Log what tools were called.
Log what text was posted back out.

Without that, your first sign of trouble is often the leaked output itself.

I would also add one practical rule: run new workflows in dry-run mode against test repositories first. If a public comment can influence a private read in staging, it will do it in production too.

What GitLost teaches CS students

This is a strong teaching example because it bundles several core ideas into one incident.

Authentication is not authorization

The attacker did not need to break into the private repository directly. The system's own authorized agent did the reading on the attacker's behalf.

That is an authorization design failure.

Public input can change private outcomes

Students often think of user input as something that affects only the page or form it was typed into. Agent systems break that mental model. A sentence written in one place can trigger actions somewhere else entirely.

Security is a property of the whole system

The model, the workflow trigger, the GitHub token, the repository permissions, and the response path all matter together. Focusing on only one piece misses the point.

That is why I find incidents like this more useful than generic AI doom-posting. They are concrete. You can diagram the data flow. You can point to the trust boundary. You can explain exactly where the design went wrong.

My bottom line on agent workflows

I like agents. I use them. I think they are useful.

I also think a lot of people are wiring them into production systems far too casually.

GitLost is the warning shot. Not because AI is uniquely dangerous, but because it shows how quickly old access-control mistakes turn into leaks when you automate them.

If your agent reads public text, touches private code, and posts publicly without a human checkpoint, fix that before you worry about the next model upgrade.

That is the part worth taking seriously.

References

Linux Signals Explained — SIGTERM, SIGKILL and Process Control

Mike — Thu, 16 Jul 2026 02:15:01 +0000

Linux Signals Explained — SIGTERM, SIGKILL and Process Control

Pressing Ctrl+C looks simple until the day it stops working.

You hit it in a terminal. One Python script exits immediately. A Docker container hangs for ten seconds before disappearing. A runaway process ignores you completely until kill -9 ends the argument.

That difference is signals.

I teach Cambridge Computer Science, and this is the gap the syllabus never bridges cleanly. Students learn hardware interrupts, interrupt priority levels, and the interrupt dispatch table. Then they sit down at a Linux terminal and meet SIGINT, SIGTERM, and SIGKILL with no explanation of how those ideas connect. In practice, signals are the missing link. They are how the kernel talks to processes.

My view is blunt: signals should be taught right after interrupt handling, not left as Unix trivia for students to trip over later.

Ctrl+C is SIGINT, not magic

When you press Ctrl+C, the terminal sends SIGINT to the foreground process group. The default action is termination, but a program can catch it and decide what to do next.

That is why a shell script can react to it:

trap 'echo "Caught SIGINT"; exit' INT
while true; do
  sleep 1
  echo "still running"
done

Run that, press Ctrl+C, and the script does not just vanish. It receives a signal, runs the handler, prints the message, and exits cleanly.

Python does the same thing in its own way. When you press Ctrl+C during a Python program, Python translates SIGINT into KeyboardInterrupt. Students often think the traceback is the problem. It is actually the clue. The interpreter is telling you that the kernel interrupted the process and Python surfaced it in a readable form.

This matters because it shows the first rule of signals: a signal is not always an instant kill switch. Sometimes it is a request. The process gets a chance to respond.

SIGTERM is the polite knock

If you run kill 1234 with no extra option, Linux sends SIGTERM.

That is the normal shutdown signal. It tells the process to finish what it is doing and leave properly. A well-behaved process catches SIGTERM, flushes buffers, closes files, saves state, and exits with dignity.

This is the distinction I care about on my own Linux boxes. When I stop services on a Docker host, I do not want a process ripped out of memory halfway through a write. I want it to shut down cleanly.

Here is the simplest Python example:

import signal
import time
import sys

running = True

def shutdown(signum, frame):
    global running
    print(f"Received signal {signum}. Closing cleanly...")
    running = False

signal.signal(signal.SIGTERM, shutdown)
signal.signal(signal.SIGINT, shutdown)

while running:
    time.sleep(1)
    print("working...")

print("Buffers flushed. Connections closed.")
sys.exit(0)

It also explains what Docker is doing. docker stop does not begin with SIGKILL. It sends SIGTERM first, waits for the grace period, then escalates only if the process refuses to go.

That is why a bad container shutdown usually looks like a ten-second pause. Docker is giving PID 1 inside the container time to behave.

If you have read my post on building a container engine from scratch, this is the practical side of the same kernel mechanics. Containers are just processes with isolation wrapped around them. They still live and die by signals.

The Redis example is a good one here. In my Redis 8.8 homelab guide, the container is useful only if its shutdown is clean. Databases, caches, and queue workers need time to flush work to disk. If you know Docker sends SIGTERM first and SIGKILL later, stop_grace_period stops looking like a random YAML setting and starts looking like what it really is: extra time for the process to do the right thing.

SIGKILL is the last resort

Signal 9 cannot be caught, blocked, or ignored. The kernel does not negotiate. It removes the process from the scheduler, tears down its resources, and that is the end of it.

This is why kill -9 works when Ctrl+C does not.

You can prove the difference with a deliberately stubborn script:

import signal
import time

signal.signal(signal.SIGINT, signal.SIG_IGN)

while True:
    time.sleep(1)

Run it. Press Ctrl+C. Nothing happens, because the process is ignoring SIGINT.

Now find the PID:

pgrep -af python

Then send:

kill -9

Students remember this one because it feels dramatic, but the real lesson is not to reach for kill -9 as a habit. The real lesson is the opposite. Use it only when the polite options failed.

SIGKILL gives the process no chance to flush, save, or clean up. If you fire it at a database, log writer, or container doing disk work, you are choosing force over correctness.

It matters in container land too. Docker eventually escalates to SIGKILL for the same reason you do: sometimes a process is stuck and has to go. The Linux OOM killer may terminate a process using SIGKILL. A container killed with SIGKILL commonly exits with code 137. Running out of memory is one possible cause.

The signals students meet without realising it

Here are the ones worth knowing first. If you want the canonical reference for default actions and edge cases, the Linux signal(7) page at man7.org is the one worth bookmarking.

Signal	Number	Default action	What it usually means
SIGHUP	1	Terminate	Terminal closed, or daemon reload
SIGINT	2	Terminate	`Ctrl+C` from keyboard
SIGKILL	9	Terminate	Forced kill by kernel or admin
SIGSEGV	11	Core dump	Invalid memory access
SIGPIPE	13	Terminate	Wrote to a pipe with no reader
SIGTERM	15	Terminate	Normal shutdown request
SIGCONT	18	Continue	Resume a stopped process
SIGSTOP	19	Stop	Forced stop that cannot be ignored
SIGTSTP	20	Stop	`Ctrl+Z` from terminal

SIGPIPE: why `yes | head -5` stops

This one is all over Linux even if nobody names it.

Run:

yes | head -5

yes would happily print forever. head reads five lines and closes its end of the pipe. When yes tries to write again, the kernel sends SIGPIPE. Default action: terminate.

That is why the command ends cleanly instead of flooding your terminal forever.

If you have already worked through my post on Linux pipes and redirection, this is the next piece of the picture. Pipes are not just text plumbing. They are kernel objects with rules, and one of those rules is that a writer with no readers gets told to stop.

SIGHUP: why a remote job dies when SSH drops

SIGHUP began as the hangup signal for a terminal line going away. The name is old, but the behaviour still matters.

A student starts a long-running script over SSH, closes the laptop, comes back later, and the process is gone. That is usually not mysterious. The session closed, the shell's children got SIGHUP, and the default action terminated them.

The fix is not magic either. Use nohup, tmux, or screen so the process is not tied to that one terminal session.

SIGSTOP and SIGCONT: pausing a process without killing it

Start a job:

sleep 1000

Press Ctrl+Z and the shell sends SIGTSTP, which is the terminal stop signal. Then you can continue it with fg or send SIGCONT yourself.

You can also force the stop with:

kill -STOP 
kill -CONT

That maps nicely onto Cambridge process states. A stopped process is not finished. It is paused until the kernel allows it to continue.

Segmentation faults are signals too

A segmentation fault is usually SIGSEGV, signal 11. The process touched memory it was not allowed to touch, so the kernel terminated it. That is process protection doing exactly what it should do.

This is one of the clearest places where operating system theory becomes real. The OS is enforcing memory boundaries between processes. Without that rule, one broken program could scribble over everything else in RAM.

So when you see:

Segmentation fault (core dumped)

that is not just an error message. It is the kernel reporting a memory protection failure and sending the signal that ends the process.

The Cambridge connection: signals and interrupt handling

This is the part students should be taught explicitly.

In Chapter 16, Cambridge students learn the interrupt model: a device raises an interrupt, the CPU saves state, the interrupt dispatch table is consulted, a service routine runs, then the previous state is restored and execution continues.

Signal delivery follows the same pattern at process level.

An event occurs: keyboard interrupt, parent command, broken pipe, invalid memory access.
The kernel marks a signal as pending for the process or thread.
When control is returning to user mode, the kernel checks whether that signal is blocked, ignored, or caught.
If a handler exists, the kernel arranges for that handler to run.
After the handler finishes, sigreturn restores the saved state.
The process continues, exits, stops, or dumps core depending on the signal and disposition.

That is not a loose analogy. It maps neatly onto the same interrupt pattern students already know, just at process level rather than hardware level.

In You're the OS, I teach interrupt handling as a game because students understand it faster when they have to make the decisions themselves. Signals fit that same lesson perfectly. The kernel is still the traffic controller. The only difference is that now the conversation is between the kernel and a process rather than the CPU and a hardware device.

You can even inspect part of this on a live Linux system in /proc//status. Try grep -E "Sig(Pnd|Blk|Ign|Cgt)" /proc/1/status and you will see the signal bitmaps immediately. Fields such as SigPnd, SigBlk, SigIgn, and SigCgt show which signals are pending, blocked, ignored, or caught. That is not abstract theory. That is the kernel's process-control dashboard sitting in plain text.

What students should remember

If you remember only four things, remember these:

Ctrl+C sends SIGINT.
kill sends SIGTERM by default.
kill -9 sends SIGKILL, and the process gets no say in the matter.
A lot of weird Linux behaviour is not weird at all once you know which signal fired.

That includes broken SSH jobs, stuck containers, dead pipelines, and segfaults.

Signals are not obscure Unix trivia. They are core process control. They explain how the kernel interrupts, stops, resumes, warns, and kills running programs. For CS students, they are the cleanest bridge between textbook interrupt handling and the real behaviour of a Linux system.

The next time a process ignores Ctrl+C, ask a better question: which signal did it get, and what is it allowed to do with it?

Proxmox VE in Production — What Running It 24/7 for Over a Year Actually Teaches You

Mike — Wed, 15 Jul 2026 02:00:00 +0000

Proxmox VE in Production — What Running It 24/7 for Over a Year Actually Teaches You

The moment an outage interrupts something you actually use every day, your homelab has stopped behaving like a toy.

That is the cleanest definition of production I know.

I have been running Proxmox VE 24/7 for over a year on my BMAX Pro 8. On paper, it is still a homelab box. In practice, Plex, Nextcloud, and Home Assistant all depend on it staying healthy. That changes the standard immediately. You stop caring about how quickly you can spin up a VM and start caring about how calmly you can recover when something breaks.

Most Proxmox posts online are about installation, upgrade paths, or first impressions. That is useful, and I have already covered the build side in my Proxmox homelab setup on a mini PC guide. What those posts usually skip is the boring middle: the long stable stretches, the one migration that runs far too long, the backup routine you should have tightened earlier, and the network blip that changes how you think about remote access.

After 400+ days, my opinion is simple: Proxmox is excellent, but it rewards boring operators and punishes optimistic ones.

Production in a homelab still means consequences

People get strangely hung up on the word production.

If you are not billing customers, they assume it does not count. I think that misses the point. If a service is part of your normal day, if downtime immediately becomes your problem, and if recovery matters more than screenshots, you are already there.

A home setup does not need a sales team behind it to deserve production habits. It only needs real consequences.

When Plex buffers, someone notices. When Nextcloud disappears, the missing files matter. When Home Assistant falls over, the problem stops being an interesting virtualization issue and starts being a broken part of the day. Those are the same kinds of day-to-day services I wrote about in my self-hosting toolkit, and they are exactly why I no longer treat uptime as a hobby metric.

That is why I no longer judge infrastructure by how clever it looks. I judge it by how easy it is to understand when I am tired. Fancy is cheap. Recoverable is expensive.

Lesson 1: Storage migrations are where confidence goes to die

A fresh Proxmox install makes storage look neat and tidy. Local disks here. VM disks there. Buttons in the web interface. Progress bars. It all feels manageable.

Then you do a real migration on a live box.

One of the clearest lessons from this past year came from moving a VM disk between local datastores on the mini PC. On paper it was a routine reshuffle. In practice it ran for hours, and while it ran, the whole box crawled. Plex playback stuttered. Nextcloud got sluggish. The migration was saturating the machine's own disk IO, and every other service was standing in the same queue.

That is the part people underestimate on a single-node homelab. A storage move is not a copy that happens quietly in the background — it is an IO-bound operation that competes directly with everything your services need. If the guest is busy dirtying memory while it moves, the tail end drags even longer. And if your Proxmox system disk is something cheap and slow, that one weakness drags the entire host down with it. A storage migration is not housekeeping once the system is busy. It is an operational event, and it belongs in a maintenance window.

This is where new homelab thinking usually goes wrong. You treat storage moves like rearranging folders on a laptop. Proxmox treats them like large, stateful operations touching the thing your services are standing on.

The teacher version is simple: moving a VM disk sounds like changing seats in a classroom. It is not. It is more like moving the entire classroom while the lesson is still running.

What changed for me after that long move?

I stopped doing storage work casually.
I started assuming every simple move would take longer than I wanted.
I became much more suspicious of doing anything big during the hours when the services were actually in use.

That sounds obvious. It is obvious. The problem is that you only really believe it after one migration ruins your evening.

Lesson 2: Backups do not mature by themselves

For a while, backups in a homelab often exist as a good intention.

You know they matter. You mean to tighten them up. You tell yourself the current setup is fine for now.

It usually is not.

My backup strategy evolved from manual vzdump jobs to automated Proxmox Backup Server with retention policies. The real improvement was not the acronym. It was moving from "I should make a backup tonight" to scheduled, deduplicated backups that ran whether I was organised or not.

Manual backups feel responsible because they produce files. Manual anything has the same weakness: it depends on you being awake, consistent, and not distracted by ten other jobs. That works for a week. It does not work for a year.

What PBS changed was the shape of the problem. Instead of wondering whether I had remembered to capture a fresh state, I had a backup system with a schedule, retention, and an actual operational rhythm behind it. The policy is boring on purpose: a daily job at 2 AM, retention set to keep seven daily, four weekly, and six monthly snapshots, so I always have a recent restore point without the datastore growing forever. If you want the vendor view, the Proxmox Backup Server documentation is worth bookmarking because it makes the retention and datastore model much clearer than trial and error ever will.

One habit matters more than the schedule itself: a weekly verify job, so PBS actually checks the backups are readable rather than just present. I will be honest about the one I do not keep — I have never sat down and tested a full restore. I have never needed to, which is either luck or a quiet risk I keep meaning to close, because a backup you have never restored is a hope, not a backup. One rule I do follow without exception: the backups do not live on the same physical disk as the VMs. If that disk dies, I do not want it taking the backups with it.

Once the box started carrying important services, I stopped thinking about backups as insurance and started thinking about them as part of the service itself. If the restore path is weak, the service is weak. That is it.

This is also where Proxmox teaches the same lesson every good sysadmin eventually learns: boring automation beats heroic memory.

Lesson 3: The network is the part you forget until it bites

Every long-running setup has one moment that changes how you think about it.

For me it was a brief network blip that, for a few minutes, cut me off from the box entirely. Nothing was on fire. The services kept running locally. But the reverse proxy in front of everything, the tunnel I use to reach it from outside the house, and my remote management were all suddenly unreachable at once. On a single-node homelab, that is the quiet risk nobody warns you about: the network path to your box is a single point, and when it wobbles, you lose the ability to see or fix anything precisely when you most want to.

The network is not a side topic. It is the topic.

I teach computer science, and this is the bit textbooks often flatten too much. On paper, a network is a tidy diagram with arrows. In practice, it is the one dependency sitting underneath every other dependency. When it drops, the dashboard that would tell you what is wrong is on the far side of the thing that is wrong.

That blip made me much less casual about the phrase "just a brief network issue." Brief for whom? The link that recovers on its own in thirty seconds, or you, standing in a coffee shop unable to reach a single service until it does? It also made me plan for it: a way in that does not depend on the one path that just failed, and notes stored somewhere other than the machine I am trying to reach.

The practical rules I would write down on day one

This is the section I wish more Proxmox posts included.

Not philosophy. Operating rules.

If I were setting up the same box again with the same requirement that these services need to stay dependable, I would write down this checklist on day one:

A real maintenance window for any storage move, not "whenever I get time tonight."
A PBS backup job on a schedule with a written retention policy — mine keeps seven daily, four weekly, and six monthly.
Backups on a different physical disk from the VMs, so one dead drive cannot take both.
A restore-test habit — the one I still owe myself — because successful backups do not prove successful recovery.
Alerts that reach me before I stumble into a problem: disk usage crossing 80%, failed backup jobs, and offline services.
A second way to reach the box that does not depend on the one network path that just failed.
Clear notes stored outside the Proxmox host, so recovery steps do not disappear with the machine you are trying to recover.
A power plan: UPS and a graceful shutdown, or at least a decision made on purpose instead of by accident.

None of that is glamorous. That is exactly why it matters.

The romantic version of self-hosting is building the thing. The grown-up version is keeping it boring six months later.

What 400 days actually teaches you

Running Proxmox for a weekend tells you whether the install worked.

Running it 24/7 for over a year tells you whether your habits work.

Proxmox itself has been solid. The harder lessons came from operations: storage work takes longer than you think, backup maturity does not happen by accident, the network path to your box deserves more respect than the dashboard encourages, and small shortcuts become large problems once the box matters.

So yes, you can run Proxmox VE in production.

The better question is whether you are ready to run it like boring, dependable infrastructure. That is the standard that matters.

The CISA GitHub Leak: What It Should Teach Homelab Owners

Mike — Tue, 14 Jul 2026 02:00:01 +0000

The CISA GitHub Leak: Homelab Security Lessons for Self-Hosters

In May 2026, researchers discovered a public GitHub repository maintained by a CISA contractor containing credentials, deployment material, internal documentation, and cloud infrastructure details, as detailed in Krebs on Security's report on the leak. CISA removed the repository and began investigating the exposure. The repository reportedly contained plaintext passwords, AWS tokens, certificates, infrastructure code, deployment files and internal documentation. Some credentials were reportedly still valid when discovered.

That is not the same thing as some dramatic headline about CISA getting completely owned. It is a repository leak, and that distinction matters. It also happens to be the part self-hosters should care about most, because this is exactly the kind of mistake that turns an ordinary lab into an easy target.

I think this is the more useful kind of security story anyway. A lot of homelab failures do not begin with a clever exploit. They begin with a secret in the wrong place, a service exposed longer than intended, or a trust boundary that only existed in your head.

Keep secrets out of repositories

The most direct lesson from the CISA incident is not about firewalls or VLANs. It is about credential handling.

Never commit passwords, API keys, private keys, access tokens, .env files, database exports, or backup archives to a repository. A repository named "private" is not a security control, and deleting a secret from the latest commit does not remove it from Git history.

At minimum:

keep secrets in environment files excluded by .gitignore
use a password manager or secrets-management system
enable repository secret scanning and push protection
rotate a credential immediately if it has been committed
inspect Git history, not only the current working tree
give tokens only the permissions and lifetime they require

A leaked credential should be treated as compromised even if there is no evidence that somebody used it.

That last point is where people still talk themselves into delay. They tell themselves nothing bad happened, so there is nothing to rotate. I do not buy that. If a token was in a public repo, the safe assumption is that it is gone.

The same principle showed up in my write-up on GitHub token theft through a VS Code bug. The bug mattered, but the real damage came from what a leaked credential could unlock after it escaped.

Audit what is actually exposed

A surprising number of homelabs run on assumption. The owner thinks only 80 and 443 are open because that is what they meant to expose six months ago.

Check it from the outside.

nmap -Pn yourdomain.com
sudo nmap -Pn -p- your-public-ip

The first command checks common TCP ports. The second scans all TCP ports and takes longer. UDP requires a separate scan.

Run the external scan against your known public IP from a device outside the home network. Do not rely on scanning a proxied hostname, because that may test Cloudflare's edge rather than your router or server.

Also, scan only systems you own or have permission to test. That should be obvious, but it still needs saying.

Then check the host itself:

ss -tulpn

That gives you the local listening sockets. The question is not just what is running. The question is what is reachable from somewhere it should not be reachable from.

If you self-host AI services, this matters even more because too many people publish them first and think about access control later. A service with weak or missing auth becomes a very different problem the moment it stops being localhost-only.

One blunt rule helps here: if you no longer remember why a port is open, close it until you do remember.

Put authentication in front of selected services

Weak authentication still causes more real damage than most fancy attack chains.

If a service supports MFA, turn it on. If it does not, put something in front of it that does. Authelia, Authentik, and Cloudflare Access are all reasonable ways to stop a login page from sitting naked on the internet.

This only works when users cannot bypass the authentication proxy and connect directly to the backend service. Restrict the origin with firewall rules, private networking, or binding to localhost where appropriate.

MFA and proxy authentication reduce the risk of stolen login passwords, but they do not necessarily protect API tokens, session cookies, application keys, or credentials that bypass the interactive login.

That is why I like layered controls more than single-product confidence. A reverse proxy is useful. A reverse proxy plus localhost binding plus tight firewall rules is better.

The point is the same either way: do not confuse a clean login page with real isolation.

Segment the network properly

Flat networks are lazy, and they are generous to attackers.

If your laptop, phone, TV, random IoT plug, and Proxmox host all sit in one easy broadcast neighbourhood, one compromise can turn into a walk across your whole lab.

A simple layout is already better than the usual everything-on-one-LAN mess:

Main devices: 192.168.10.0/24
Homelab servers: 192.168.20.0/24
IoT devices: 192.168.30.0/24
Guest Wi-Fi: 192.168.40.0/24

Then make the policy boring and strict:

Guest reaches the internet, not your servers
IoT reaches only what it actually needs
Admin access to Proxmox, SSH, and dashboards comes from your trusted network only

Different IP ranges alone do not provide segmentation. The network equipment must place them in separate VLANs or interfaces and enforce firewall rules between them.

That is the part people skip. Changing the third octet is not segmentation. Enforcing traffic boundaries is segmentation.

A compact setup is easier to secure because you can still explain it to yourself six months later. My Proxmox homelab setup on a mini PC and the BMAX Pro 8 budget homelab comparison both come back to the same point: extra complexity creates blind spots fast.

Add brute-force protection and rate limits

Anything with a login page gets hammered eventually.

If you expose SSH, a reverse proxy, a dashboard, or any admin panel, assume bots are already trying it. Fail2ban still earns its place here, but only if it is actually watching the right logs.

sudo apt update
sudo apt install fail2ban
sudo systemctl enable --now fail2ban

A healthy jail might produce output similar to this:

Status for the jail: sshd
|- Filter
|  |- Currently failed: 3
|  `- Total failed:     19
`- Actions
   |- Currently banned: 1
   `- Total banned:     4

Fail2ban must be connected to the correct application or proxy logs. When traffic passes through Cloudflare or another proxy, configure trusted proxy handling carefully so the system reads the real client address without trusting spoofed headers.

Pair that with firewall rules that make sense for the host.

sudo ufw default deny incoming
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status numbered

If you are doing this over SSH, stage the allow rules first so you do not lock yourself out.

Be aware that Docker manages its own firewall rules. A container port published on 0.0.0.0 may not behave as expected under a simple UFW policy. Check published ports with docker ps and test reachability externally.

Reverse-proxy rate limits help too. They will not make a weak stack strong, but they do remove easy wins from bots that rely on unlimited retries.

Patch deliberately and on schedule

Most self-hosters do not ignore updates because they are careless. They ignore updates because the process is vague, annoying, and easy to postpone.

That is why cadence matters. Pick a rhythm and make it boring.

For Docker-based services, I still prefer two separate habits:

notice that an image changed
decide when to roll it out

That is why I like tools such as Diun or Dockcheck. They tell you something moved without automatically changing production. Auto-update tools have their place, but I would be careful with anything that touches your reverse proxy, auth layer, or database while you are asleep.

A safer update cycle looks like this:

docker compose pull
docker compose up -d

Run it when you can watch the result.

If you harden containers, test those changes one at a time:

read_only: true
security_opt:
  - no-new-privileges:true
cap_drop:
  - ALL

Test these settings one at a time. A read-only container may still have writable volumes, and dropping every capability may break software that requires a specific capability. Add back only what the application demonstrably needs.

Start where the risk drops fastest

Do not turn this into a heroic weekend project.

Start with the fix that removes the most risk in your own setup. Sometimes that is rotating an exposed token. Sometimes it is closing a port you forgot about. Sometimes it is putting one fragile service behind a proper auth layer and binding the backend to localhost.

The CISA contractor leak is a useful warning because it is ordinary. There was no magic to it. Just credentials in the wrong place, followed by the usual cleanup.

That is exactly why homelab owners should pay attention.

Homelab Burnout: Sustainable Self-Hosting When the Hobby Starts Feeling Like Work

Mike — Mon, 13 Jul 2026 02:00:01 +0000

Homelab Burnout: Sustainable Self-Hosting When the Hobby Starts Feeling Like Work

Browse r/homelab for long enough and you will regularly find people questioning whether their homelab has become more work than fun. People are not tired of technology itself. They are tired of turning a useful hobby into an unpaid support contract.

I teach computer science full-time, run a stack of self-hosted services, write here, and record YouTube videos. My homelab is useful, but only when it stays in its lane. The moment it starts demanding daily babysitting, it stops being a hobby and starts behaving like a badly managed side business.

That is my opinion on this, and I will argue it plainly: a homelab should solve problems you actually have. The second it exists mainly to generate maintenance work, the learning value starts to disappear. You are just doing chores in Docker.

The homelab burnout escalation trap is real

Most homelabs do not become messy in one big leap. They grow one perfectly reasonable decision at a time.

You add Plex because you want your media in one place. Then you add Uptime Kuma because monitoring feels responsible. Then a reverse proxy, then a dashboard, then a note-taking tool, then an automation tool, then a second database because one app prefers Postgres and another wants Redis. A few weeks later you have twenty moving parts and half of them are there because they looked useful on somebody else's YouTube channel.

That is how hobby scope creep works. Nobody wakes up and decides to build an unreliable mini data centre in the spare room. You just keep saying yes to one more service.

I already trimmed my own stack once. In Self-Hosting Toolkit: 10 Apps I Actually Use Every Day, the key phrase is "actually use." That was not branding. It was a survival rule. If a service does not save time, make money, teach me something worth keeping, or solve a regular problem in my week, it does not deserve a permanent slot.

A lot of burnout comes from running software with no job description.

Automation should remove toil, not create clever new failure modes

Automation is the part everyone talks about, but many people automate the wrong things. They build a complicated chain of scripts, webhooks, and notifications, then spend Saturday debugging the automation instead of the original problem.

Good homelab automation is boring. It cuts repeat work. It makes failure obvious. It does not need a diagram just to explain why your container restarted.

A simple Docker health check earns its place because it answers a basic question: is this service alive or not?

services:
  app:
    image: my-app:1.4.2
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:3000/health"]
      interval: 30s
      timeout: 10s
      retries: 3

This assumes the image includes curl and the application exposes a /health endpoint. Otherwise, use a command already available inside the container or add an appropriate health-check method.

That is useful. It replaces guesswork with one small signal. Pinning a known version also makes updates more deliberate and rollbacks easier.

The same applies to update routines. I do not want to remember which container needs a manual pull, which one changed its environment variables, and which one quietly stopped two weeks ago. If you are relying on memory for routine maintenance, you already have a fragile system.

This is also why I still like small, direct tooling. A maintenance script that checks container status, disk usage, and backup age is worth more than a beautiful dashboard I only open after something breaks. Monitoring should tell you what needs attention. It should not become another thing you maintain for its own sake. My Hermes Agent mini PC setup guide comes from exactly that mindset: use automation to remove repeated admin work, not to build a second system you have to babysit.

If you want the short version, build the minimum system that answers three questions:

Is the service up?
Is the data backed up?
Will I notice when something fails?

Everything beyond that needs to justify itself.

Boundaries matter more than enthusiasm

This is the part many homelab guides skip because it is less fun than listing tools.

You need boundaries.

Not every service should be self-hosted. Some are worth the effort because they protect privacy, centralise your workflow, or save ongoing subscription costs. Others are cheap in dollars and expensive in attention.

The best question I know is this: what happens if this service breaks for 48 hours?

If the honest answer is "nothing important," it probably should not sit in the centre of your setup. It might still be a good weekend experiment. It just should not become permanent infrastructure.

That is the difference between a lab and a dependency. Labs are where you test ideas. Dependencies are the things that quietly ruin your week when they go down.

For me, the keepers are the services that support real work: file storage, monitoring, automation, remote access, and a few tools I touch constantly. A smaller stack is not less serious. It is usually better engineered because you can still understand it on a tired Tuesday night. The same bias toward manageable infrastructure is why I built my Proxmox homelab setup on a mini PC around hardware I can understand, recover, and afford to leave running.

A lot of people also underestimate security overhead here. Every exposed dashboard, admin panel, and reverse proxy route increases the attack surface. Limiting permanent services is therefore not only a maintenance decision; it is also a security discipline. That is why I treat access control and certificates as infrastructure, not decoration. If you use Cloudflare and want an additional authentication layer in front of selected services, Cloudflare Self-Managed OAuth is worth understanding. It complements rather than replaces secure configuration, updates, certificates, and network controls.

Documentation is what saves you at 2 AM

The most underrated homelab tool is not Docker, Proxmox, or a reverse proxy. It is documentation.

I mean boring documentation. The kind that tells you which compose file matters, which ports are already in use, which volume paths hold real data, which containers depend on each other, and what to restore first after a bad update.

Without that, every outage becomes archaeology.

This is where my Obsidian vault earns its keep. I keep service notes, recovery steps, container locations, backup paths, and configuration decisions in one place. Not because it is elegant, but because I do not trust the tired version of myself to remember why something was set up a certain way six months ago.

A useful service note can be brutally simple:

Service: Uptime Kuma
Compose file: /opt/monitoring/docker-compose.yml
Data path: /opt/docker/uptime-kuma
Depends on: reverse proxy + local DNS
Backup: nightly tar to external drive
Restore order: DNS -> proxy -> Kuma

That note has saved more time than any fancy dashboard ever has.

The same habit applies to anyone learning to build real systems. Sustainable engineering is not just about writing code that works today. It is about leaving enough context behind that the next person, or future you, can support it without panic. Real systems fail. Good documentation turns failure from chaos into a checklist.

The economics are not just about money

Self-hosters love talking about savings, and sometimes the savings are real. A small mini PC, a couple of drives, and a handful of containers can absolutely replace several monthly subscriptions.

But cost is not just hardware plus electricity.

Time is part of the bill.

If a service saves you $4 a month but costs two hours of maintenance every other weekend, that is not frugal. That is expensive in a different currency. The maths gets worse if the service is something you barely use.

This is why I think the healthiest homelab mindset is selective self-hosting. Keep the things that give you one of these returns:

control over important data
a workflow improvement you genuinely notice
a solid learning payoff
a clear cost saving without constant babysitting

If it gives you none of those, it is probably clutter.

I also think there is a phase many homelabbers need to go through once: the purge. Not a dramatic teardown, just an honest audit. Open the compose files. List the services. Ask what each one is doing for you now, not what you hoped it would do when you installed it.

Delete the tourists: services that arrived as experiments, contributed nothing, and never left.

A quick homelab sustainability audit

For each permanent service, ask:

Did I use it during the last 30 days?
Does it hold data that matters?
Is it backed up and tested?
Do I know how to restore it?
Would replacing it with a hosted service reduce meaningful work?
Is its learning value still greater than its maintenance cost?

A service that fails most of those questions probably belongs in the lab, not in permanent production.

My rule now

I still enjoy self-hosting. I am not interested in a purity test where everything must live on somebody else's cloud. But I am also done pretending that more containers automatically mean more skill.

My rule now is simple: every service must earn the right to keep running.

If it saves time, protects data I care about, supports teaching or content work, or teaches me something I will reuse, it stays. If it mostly produces updates, alerts, and extra surface area, it goes.

That shift changed the feel of my homelab completely. It is calmer. Easier to recover. Easier to explain. Easier to leave alone for a few days without wondering what is quietly breaking.

That is what sustainable self-hosting looks like. Not the biggest stack. The one you can still live with.

Six arguing AI agents: what multi-agent debate teaches CS students about AI architecture

Mike — Sun, 12 Jul 2026 14:58:14 +0000

Six arguing AI agents: what multi-agent debate teaches CS students about AI architecture

Most students meet AI through prompts. Type a question, get a paragraph back, move on.

That framing is useful for five minutes and then it gets in the way.

The more interesting lesson is what happens when you stop treating a model like a single all-knowing assistant and start treating it like one noisy component in a bigger system. Give several model instances the same problem. Ask each one for an answer. Make them criticise each other. Force another round. Then have a judge combine the best points.

Suddenly the discussion stops being about prompt wording and starts being about architecture.

That is why multi-agent debate matters. Not because six chat windows arguing with each other are cute, and not because debate turns an LLM into a reliable thinker. It matters because the pattern exposes what these systems are actually good at, where they fail, and what good computer science students should notice before they get hypnotised by demo videos.

My take is blunt: prompt engineering is the least interesting part of this story. System design is the lesson.

What multi-agent debate actually is

The 2023 paper "Improving Factuality and Reasoning in Language Models through Multiagent Debate" describes a simple setup: multiple language model instances propose answers, debate them over several rounds, and then converge on a final answer. The authors report improvements on mathematical and strategic reasoning tasks, along with reductions in fallacious and hallucinatory responses compared with single-agent baselines.

The idea is not mystical. You are building a feedback loop.

A basic debate pipeline usually looks like this:

Agent A, B, and C each answer the same prompt independently.
Each agent reads the others' answers and points out weak logic, missing steps, or contradictions.
Each agent revises its own answer after seeing criticism.
A final judge, or one more synthesis pass, picks the strongest parts.

That workflow borrows from something humans already do in programming: code review. One developer writes the first pass. Another person spots the broken assumption. Someone else notices the edge case. The final version is better, not because one brain became smarter, but because the process exposed mistakes before they shipped.

If you have read my piece on searching old AI conversations, the same pattern shows up there too. The value is often in the review loop, not the first output.

What debate reveals about LLM architecture

This is the part students should care about.

A large language model is still an autoregressive system that predicts the next token from context. The GPT-3 paper describes it as an autoregressive language model, and the 2022 paper "Chain-of-Thought Prompting Elicits Reasoning in Large Language Models" showed that giving models room to produce intermediate reasoning steps can improve results on arithmetic, commonsense, and symbolic tasks. Multi-agent debate pushes that same idea one step further: instead of one long reasoning trace, you create several traces and let them attack each other.

That tells you something important.

A lot of what people call "reasoning" in practice is not a hidden mode that switches on inside the model. It is often the result of structure wrapped around the model: extra steps, extra context, criticism, retries, and selection. In other words, the intelligence you see is partly in the orchestration.

I think students need to hear that early, because the internet keeps selling the opposite story. One flashy demo appears, people act as if the model has crossed some mysterious line, and the system design vanishes from the conversation.

But look at what debate is doing under the hood:

redundancy, because several agents attempt the task
error detection, because agents inspect each other's reasoning
iterative refinement, because answers are revised instead of accepted once
aggregation, because a judge or synthesis pass chooses among alternatives

Those are architecture decisions. They are not prompt tricks.

That is also why multi-agent systems look a lot like classic CS ideas wearing new clothes. Distributed systems use redundancy because one node can fail. Compilers use multiple passes because one pass is not always enough. Software teams use tests and review because the first version is rarely the final one. Debate is the same instinct applied to a probabilistic text engine.

The useful lesson is not "AI thinks better in groups"

Be careful with the headline version of this idea.

It is tempting to say that a group of agents "thinks" better than one agent. I would phrase it more carefully. A structured group process can produce better outputs than a single pass because it creates more chances to catch obvious mistakes.

That difference matters.

If three weak answers all share the same bad assumption, debate does not magically rescue you. It can even make the wrong answer sound more convincing, because now it arrives with consensus theatre attached. Agreement is not proof. A polished synthesis is not proof either.

That is one reason I still prefer grounded workflows over pure discussion workflows. The 2022 paper "ReAct: Synergizing Reasoning and Acting in Language Models" is useful here because it combines reasoning traces with actions that fetch external information. For factual tasks, a debating panel with no access to evidence is still trapped inside its own guesses.

So yes, debate can help. No, it does not turn a language model into a truth machine.

Why CS students should care

If I were teaching this to a class, I would not present multi-agent debate as an AI party trick. I would use it to teach systems thinking.

Students usually start with the surface layer: the prompt. That is understandable. It is the part they can see.

But the deeper question is this: what kind of system are you building around an unreliable component?

That question is pure computer science.

1. It teaches decomposition

A decent multi-agent workflow splits one job into smaller responsibilities: proposer, critic, judge, verifier. That is exactly how good software systems are designed. Different parts do different jobs. Nobody asks the same component to generate, verify, rank, and explain everything perfectly in one pass.

2. It teaches evaluation

Students often stop at "the AI answered." Debate forces the next question: answered according to what standard?

Was the logic complete? Did it use evidence? Did two agents agree for the same reason, or did one just copy the other? What would count as a failure case?

Those are evaluation questions, and they matter more than the prompt itself. I made a similar point in my guide on how to set up a test pipeline that actually catches bugs, because good systems need clear checks, not vibes.

3. It teaches observability

One reason single-shot AI tools are slippery is that you only see the final sentence. Debate systems expose more of the path. You can inspect the initial answers, the objections, the revisions, and the final merge. That makes the system easier to debug.

It is the same reason tracebacks matter in programming. If you only saw "program failed," you would learn very little. If you can inspect the chain of events, you can actually reason about the bug. That is why I keep pushing students toward explicit debugging habits instead of one-shot guesswork.

4. It teaches trade-offs

Debate sounds clever until you pay for it.

Three rounds with four agents is already twelve model calls before the judge speaks. Latency goes up. Cost goes up. Failure modes multiply. Prompt leakage between agents becomes a design issue. Logging becomes a design issue. Rate limits become a design issue.

That is good news for a CS classroom, because now the conversation is finally real. Students are not just asking, "Can AI do it?" They are asking, "What did this architecture cost, and was the gain worth it?"

That is a much better question.

Three debate frameworks students can try

You do not need a research lab to experiment with this. You just need to keep the tasks small and the scoring clear.

The two-agent critic loop

This is the simplest version.

Agent 1 writes an answer.
Agent 2 critiques it line by line.
Agent 1 revises once.

Use it for short explanations, code comments, or a paragraph answering a theory question. Students learn quickly that the second pass is often less about brilliance and more about catching omissions.

The three-agent committee

Give the same question to three isolated agents. Do not let them see each other at first. Then ask a judge to compare them against a rubric.

This works well for structured tasks: design choices, algorithm explanations, short essays, and worked examples. The point is not to vote blindly. The point is to inspect disagreement. When two answers agree and one dissents, students should ask why.

The evidence-first debate

This is the one I would push hardest.

Before any agent can argue, each one has to quote or retrieve evidence from an approved source set: lecture notes, a paper extract, a dataset, or a fixed documentation page. Only then can the critique round begin.

That rule stops the exercise from turning into a confidence contest. It also teaches a habit students badly need in the AI era: separate fluent output from justified output.

If students want to run these experiments locally, they do not need an expensive machine. Smaller local setups are enough for classroom-scale tests, especially if the tasks are short and the models are modest. I covered that trade-off in my guide on running AI models on old hardware. For experimentation, architecture matters more than brute force.

Where the hype gets ahead of reality

This is where I part company with the more breathless AI commentary.

When people show a multi-agent demo, they often imply that the magic comes from "more agents." I do not think that is the right takeaway. Past a certain point, adding more agents can just add more noise, more cost, and more nicely formatted nonsense.

The better question is whether each added step has a job.

Does one agent bring independent evidence?
Does one agent act as a strict verifier?
Does one round of critique catch a failure mode you have actually observed?

If the answer is no, you are probably building theatre.

I see the same issue in some local AI tooling conversations too. People pile on frameworks because the stack looks impressive, not because each layer solves a real problem. That is why I keep coming back to practical architecture choices in posts like Learning Docker by Building a Container Engine from Scratch. More moving parts only help when each part earns its place.

What I would want students to remember

If this topic lands anywhere useful in a CS classroom, it should land here.

A language model is not the whole system.

The prompt is not the whole system.

The benchmark screenshot is definitely not the whole system.

What matters is how you handle uncertainty, error checking, evidence, revision, and cost around a component that is fluent but unreliable. Multi-agent debate is worth studying because it makes those design choices visible.

That is the lesson I would argue for: AI architecture is a systems problem first, and a wording problem second.

A simple classroom version is to give three agents the same algorithm explanation, ask each one to identify the weakest assumption in the others' answers, then have students compare the final synthesis against a rubric. That keeps the lesson concrete. Students can see whether the extra critique round exposed a real weakness or just produced more polished noise.

Students who understand that will build better tools, ask better questions, and fall for fewer demos.

That seems more useful than teaching them how to write one more clever prompt.

TLS Certificates for Internal Services: HTTPS Inside Your Homelab

Mike — Sat, 11 Jul 2026 04:35:52 +0000

TLS Certificates for Internal Services: HTTPS Inside Your Homelab

Most homelab TLS setups stop at the front door.

You put Caddy, Nginx, or Traefik in front, get a valid certificate for app.example.com, and call it done. From the browser to the reverse proxy, everything is encrypted. From the reverse proxy to the actual container, it is often plain HTTP on port 80 or 8080.

That is not automatically wrong. On a single mini PC with a couple of containers, internal TLS is rarely the first problem I would fix. Weak passwords, overexposed dashboards, and lazy network rules usually matter more.

But once a homelab grows beyond "one box, three containers," plain HTTP starts spreading everywhere. A reverse proxy talks to one VM over HTTP. A helper container calls another API with no encryption. A dashboard sends session cookies across the LAN because "it's only internal." That is the point where internal TLS stops being cosmetic and starts being decent housekeeping.

My view is simple: if a service is worth exposing through a nice hostname, it is probably worth deciding how you want to trust it internally too.

Where plain HTTP sneaks back in

The usual pattern looks like this:

Browser to reverse proxy: HTTPS
Reverse proxy to upstream app: HTTP
App to database: whatever the default was
Admin tool to API endpoint: plain HTTP because it lives on the same subnet

That can be acceptable in a tiny setup where everything runs on one host and nothing else can reach the traffic path. I still would not call it a long-term design. The moment you add another VM, another Docker host, or a tunnel back into the network, your "internal only" assumption gets weaker.

This came up for me while thinking about the same problem I discussed in my self-hosted AI security guide. Once you start putting tools behind reverse proxies and opening remote admin paths, the old habit of "just use HTTP inside" sticks around long after the setup stopped being small.

Internal TLS does not make a homelab secure. It does do three practical things:

It reduces casual packet snooping on segments you do not fully trust.
It lets you test the same HTTPS assumptions internally that you expect externally.
It stops you from building workflows that silently depend on insecure defaults.

That third one matters more than people admit. If your automation only works when certificate checks are disabled, you have already taught your stack a bad habit.

The three approaches that actually make sense

I would ignore the endless certificate-tool rabbit hole and stick to three realistic options.

1. Local CA for quick wins

If your goal is "I want valid-looking HTTPS inside the lab by tonight," a local CA is the low-friction option.

Caddy has a local_certs option that tells it to issue certificates internally instead of using a public ACME CA. That makes it useful for development environments and small private networks where you control the clients. The catch is the same as every private CA setup: the clients have to trust your root certificate first.

This is the best fit when:

you mostly browse services from your own machines
you control the phones, laptops, or tablets that need trust installed
you want HTTPS warnings gone without involving public DNS

This is the worst fit when:

guests or unmanaged devices need access
you do not want to distribute a root CA to every client
you expect native apps to behave nicely with an untrusted chain

In other words, local CA is tidy for a personal lab, messy for a shared one.

2. Let's Encrypt with DNS-01 for services under your real domain

If your internal services already use names under a public domain you control, DNS-01 is usually the cleanest answer.

Let's Encrypt's DNS-01 challenge docs explain that your ACME client places a TXT record under _acme-challenge for the domain. Those docs also note two details that matter here: DNS-01 can issue wildcard certificates, and it only makes sense when your DNS provider has an API you can automate.

That means you can issue certificates for names under your domain without exposing the service itself on the public internet, as long as you can update DNS records for the validation step.

This is the setup I would reach for if you already use a real domain for your homelab and want browser-trusted certificates without teaching every client to trust a private root.

It works especially well with reverse proxies that already understand ACME. Traefik's ACME docs explicitly support dnsChallenge, and the same docs note that ACME v2 wildcard certificates require a DNS-01 challenge.

The downside is operational, not conceptual. You now have DNS API credentials in the mix. That is manageable, but it deserves the same care I talked about in the Cloudflare self-managed OAuth post: narrowly scoped credentials beat one giant token that can edit everything.

3. Private ACME with step-ca when the lab stops being small

If you have multiple machines, multiple services, and want certificates to renew automatically without public DNS hacks, this is where step-ca starts making sense.

The useful part is not just "run your own CA." The step-ca ACME basics docs show that step-ca supports ACME, so standard ACME clients can request certificates from your own certificate authority instead of from a public provider. That means you can keep the familiar issuance flow while staying fully inside your own trust model.

This is the option I would choose when:

you have several hosts, not just several containers
internal service names are not part of public DNS
you want one internal CA instead of one-off self-signed certs everywhere
you are prepared to manage trust distribution properly

This is also the point where you need to be honest with yourself. A private CA is not hard because of certificate issuance. It is hard because every client needs the root certificate installed correctly, and you need a plan for renewal, revocation, and what happens when you rebuild devices.

If that sounds annoying, that is because PKI is annoying. The tooling got better. The responsibility did not.

My rule of thumb

Here is the version I would give to anyone building a real homelab instead of a PKI hobby project.

One box, mostly personal devices, low stakes: local CA is fine.
Public domain already in use, want normal browser trust: Let's Encrypt with DNS-01.
Several hosts, several internal names, long-term setup: private ACME with step-ca.

What I would not do is collect five half-solutions at once.

I have seen a lot of homelabs end up with one self-signed cert for Grafana, a reverse proxy wildcard from Let's Encrypt, one app running plain HTTP forever, and a note to "fix later" on the service that matters most. That is not a design. That is archaeology.

What to test before you call it done

After you flip a service to HTTPS, test the boring parts.

These examples are worth keeping around:


bash
curl -vk https://service.example.internal/
openssl s_client -connect service.example.internal:443 -servername service.example.internal

Beyond Ollama: 3 Homelab AI Tools Worth Trying Next

Mike — Fri, 10 Jul 2026 06:05:29 +0000

Beyond Ollama: 3 Homelab AI Tools Worth Trying Next

Ollama is the tool I recommend when someone wants their first local model running tonight. It is simple, fast to understand, and good enough for a lot of homelab jobs. Pull a model, hit an API, wire up Open WebUI, done. I already covered that full path in my Docker + Ollama + Open WebUI guide.

But Ollama also nudges you toward one shape of local AI: text models behind a neat API. That is fine until you want more control over image workflows, a cleaner OpenAI-compatible service layer, or one local runtime that can cover chat, speech, embeddings, and image generation from the same box.

That is where the next wave gets interesting.

The three projects I would watch right now are ComfyUI, LocalAI, and Lemonade. They are not direct replacements for Ollama in the sense of "uninstall Ollama and forget it exists." They solve different problems:

ComfyUI is for visual AI workflows where you want to control every step
LocalAI is for homelabbers who want a broader local API surface with OpenAI-compatible apps
Lemonade is for people who want to experiment with a broader local AI runtime with chat, image, speech, and standard APIs

My view is simple: Ollama is still the easiest first stop, but it is no longer the whole conversation. If your homelab AI stack stops at Ollama, you are missing some genuinely useful tools.

Why look past Ollama at all?

Ollama does one thing very well: it makes local LLM inference approachable. That matters. A lot of local AI users never move past the "I want a model on my machine" stage, and they do not need to.

Still, three limits show up pretty quickly.

First, Ollama is mostly about model serving, not workflow design. If you want to build a repeatable image pipeline with prompt branches, upscalers, masks, and post-processing, Ollama is the wrong tool.

Second, plenty of apps expect an OpenAI-style API and want more than text generation. LocalAI leans hard into that compatibility layer, and that can save time when you are wiring tools together.

Third, local AI is getting more multimodal. Text alone is no longer the interesting part. Speech to text, text to speech, embeddings, image generation, and agent tooling are all creeping into the same workflows. Lemonade is appealing because it is trying to package that broader stack into one local service.

That does not mean these projects are automatically better. It means they cover ground Ollama does not.

1. ComfyUI: the visual workflow engine

If your local AI curiosity has moved from "run a model" to "build a pipeline," ComfyUI is the one to try next.

ComfyUI's documentation describes it as an AI creation engine for visual professionals who want control over every model, every parameter, and every output. The docs also push two ideas that matter for homelab users: you can run ComfyUI on your own hardware through Comfy Desktop, and the whole product is built around workflows you can load, remix, and iterate on.

That workflow-first approach is the real difference.

Instead of typing one prompt into a chat box and hoping for the best, you build a graph. One node loads a checkpoint. Another encodes the prompt. Another sets sampler settings. Another upscales. Another handles masks or ControlNet-style conditioning. It feels more like wiring a media pipeline than chatting with a bot.

That is why ComfyUI matters beyond image hobbyists. It teaches a better mental model for local AI. You stop thinking in one-shot prompts and start thinking in reusable systems.

What ComfyUI is good at

For a homelab, I think ComfyUI makes sense in four cases:

You want stable, repeatable image generation workflows instead of random prompting.
You are generating assets for a blog, course, or small business and you need consistency.
You want to understand how visual AI pipelines actually fit together.
You want to save and reuse workflows instead of rebuilding the same prompt stack every time.

The community workflow library is a big part of the appeal. You can start from a template, inspect how it works, then trim it down to fit your own hardware.

Where ComfyUI gets awkward

The downside is also obvious: ComfyUI asks more from you.

If Ollama feels like docker run plus a chat window, ComfyUI feels like learning a node editor. That is not a flaw. It is the point. But it means the learning curve is real.

It is also the least forgiving option here if your hardware is modest and your expectations are not. Text inference on CPU-only hardware can still be useful, as I found in my old hardware local AI write-up. Image workflows are different. You can run them locally, but speed and model choice matter a lot more.

So my practical advice is this: do not start ComfyUI by chasing the flashiest workflow on social media. Start with a basic community workflow, a smaller model, and one job you actually have. Blog art. Thumbnail variations. Product mockups. Something concrete.

My homelab take on ComfyUI

ComfyUI is the tool in this list that most rewards a dedicated box or VM. If your homelab has a machine with a decent GPU, this is where it earns its keep. If your homelab is just a mini PC with integrated graphics and limited RAM, ComfyUI can still be educational, but I would not make it your first "serious production" local AI service.

That is also why I do not see ComfyUI as an Ollama replacement. I see it as the moment local AI stops being only about chat.

2. LocalAI: the compatibility play

LocalAI is interesting for a completely different reason.

Its homepage makes the pitch very clearly: it is a free OpenAI and Anthropic alternative, built as a small composable AI stack. The LocalAI quickstart also states that it is a drop-in replacement for the OpenAI API, supports multiple model families, and can run language models, images, audio, and more locally. Docker is the recommended installation method, and the quick start is straightforward:

docker run -p 8080:8080 --name local-ai -ti localai/localai:latest

That single detail tells you who LocalAI is for.

It is for the homelabber who already has a pile of tools that expect a familiar API shape and does not want to keep adapting everything around one runtime. If you have scripts, agent tools, or self-hosted apps that speak OpenAI-style endpoints, LocalAI can be easier to slot into place than a more opinionated stack.

What LocalAI is good at

The best part of LocalAI is not that it tries to do everything. It is that it tries to expose local models through interfaces a lot of existing software already understands.

That gives it three obvious uses in a homelab:

replacing cloud API calls in tools that already support OpenAI-compatible backends
centralising local inference behind one service instead of a stack of ad hoc wrappers
experimenting with broader local AI features without locking yourself into a single app UI

I also like the way the project frames itself as composable. That usually matters more in a homelab than polished marketing. You want pieces you can fit into your own stack, not a black box that wants to own the whole machine.

The part people gloss over

OpenAI-compatible does not mean identical behaviour across every app.

This is where a lot of local AI blog posts get sloppy. A compatible API gets you much closer to plug-and-play, but model capabilities, latency, backend support, and edge-case behaviour still vary. The docs promise the API shape. They do not promise every upstream app will behave exactly as if it were talking to OpenAI.

That is not a criticism of LocalAI. It is just the honest version.

If you are the kind of homelabber who likes clean interfaces and clear boundaries, LocalAI is a strong fit. If you want a dead-simple chat setup in ten minutes, Ollama is still easier. If you want a broad local API target for apps and experiments, LocalAI starts to look better.

My homelab take on LocalAI

This is the one I would try first if I were moving from "local toy" to "local platform."

Why? Because a homelab gets messy fast. One service wants text generation. Another wants embeddings. Another expects an OpenAI-style endpoint. Another needs audio later. A compatibility layer matters more than people think.

I would still keep the deployment conservative. Bind it locally first. Put a reverse proxy and auth in front of it if you need remote access. The same local AI security rules I wrote about for Ollama still apply here: do not confuse "self-hosted" with "safe by default."

3. Lemonade: the all-in-one runtime I would keep an eye on

Lemonade is the newest-feeling project in this list, and it is the one I find easiest to imagine on a modern homelab.

Its site pitches Lemonade as local AI for text, images, and speech. The installation flow says it equips your machine with an AI runtime, GUI, CLI, and API endpoints. The Lemonade GitHub project pushes the same idea harder: Lemonade Server exposes standard OpenAI, Anthropic, and Ollama APIs, while the broader platform covers chat, coding, speech, image generation, transcription, and embeddings.

For Docker users, the site shows this quick start:

docker run -d \
  --name lemonade-server \
  -p 13305:13305 \
  -v lemonade-cache:/root/.cache/huggingface \
  -v lemonade-llama:/opt/lemonade/llama \
  ghcr.io/lemonade-sdk/lemonade-server:latest

That is a serious homelab signal. It tells me the project understands how people like us test things: one container, mapped port, persistent volumes, then decide whether it deserves a permanent home.

What Lemonade is good at

What I like here is the ambition.

Lemonade is not saying "here is one local model runner." It is saying "here is a local-first AI runtime that can serve apps and agents, speak familiar APIs, and cover multiple modalities." The README even exposes CLI flows like lemonade run, lemonade pull, and lemonade list, plus a built-in model manager.

That makes Lemonade appealing for a homelab in three situations:

you want one local runtime for more than chat
you want a local service that can talk to existing apps over standard APIs
you want multimodal experiments without stitching five unrelated projects together yourself

There is also a quiet but important detail in the README: Lemonade documents support across CPU, GPU, and in some cases NPU-focused configurations depending on backend and platform. I would not promise all of that on every random mini PC. But the direction is interesting. It feels closer to where local AI is heading.

Why I would still be cautious

The more all-in-one a project becomes, the more you need to be realistic about scope.

Broad capability claims are easy to love in a README and harder to operate in a homelab. Multimodal stacks usually mean more storage, more dependency churn, more model management, and more opportunities to end up debugging the runtime instead of using it.

So I would treat Lemonade as a platform experiment before I treated it as core infrastructure.

That is not an insult. Plenty of the best homelab tools start as platform experiments.

My homelab take on Lemonade

If I wanted one box in the rack to become "the local AI services machine," Lemonade is the project here that best matches that idea.

Not because it is automatically the best at every individual task. It probably is not. But because it is trying to unify text, image, speech, and app-facing APIs into one local runtime. That is a more useful direction than yet another single-purpose model wrapper.

Which one should you choose?

Short version.

Pick ComfyUI if:

you care about image generation or visual workflows more than chat
you want to build reusable pipelines, not just send prompts
you have the patience to learn a node-based tool properly

Pick LocalAI if:

you want a local OpenAI-compatible service for apps and scripts
you are building a broader self-hosted AI platform, not just a chat box
you prefer composable infrastructure over an all-in-one user experience

Pick Lemonade if:

you want one local runtime that reaches beyond text
you care about chat, speech, embeddings, and image generation in the same stack
you like the idea of standard APIs plus local-first multimodal tooling

Stick with Ollama if:

your real need is still just local text inference
you want the smallest setup burden
you value simplicity more than feature breadth right now

That last one matters. There is no prize for making your homelab more complicated than it needs to be.

Practical setup advice before you install any of them

This is the part that matters more than choosing the "right" brand.

1. Separate experiments from core services

Do not dump every AI runtime onto the same VM that already runs your password manager, DNS, and media stack. Give local AI tools their own box, LXC, or VM where possible. They pull big models, eat disk, and change quickly.

If you are planning that separation now, my homelab network design guide is the post I would read before carving out another AI host.

2. Start with one use case, not three

The fastest way to waste a weekend is to install ComfyUI, LocalAI, and Lemonade in one sitting and learn none of them properly. Pick one actual job first.

If the job is image workflows, use ComfyUI.
If the job is API compatibility, use LocalAI.
If the job is "I want a broader local AI service layer," try Lemonade.

3. Budget storage before you budget hype

Model storage gets out of hand fast. Between checkpoints, embeddings, speech models, caches, and image assets, local AI can eat SSD space long before it maxes out CPU.

4. Keep security boring

Local AI projects are still software stacks with ports, APIs, and update cycles. Keep them off the open internet unless you have a very good reason. Bind locally first. Add auth deliberately. Reuse the same caution you would use for any self-hosted dashboard or API. My Ollama security guide exists for a reason.

My actual opinion

I do not think the next step after Ollama is "the one true replacement." I think the better question is what kind of homelab AI user you are becoming.

If you are becoming a workflow builder, ComfyUI makes sense.
If you are becoming an infrastructure person, LocalAI makes sense.
If you want a local-first service layer that feels closer to a real AI platform, Lemonade is the one I would watch.

That is why this space is more interesting now than it was six months ago. Local AI is finally splitting into real categories instead of pretending every problem is solved by one model runner and a chat window.

Ollama got a lot of people through the front door. Good. It needed to happen.

But the room is bigger now.

If you want the philosophy behind local-first AI, read Right to Local Intelligence. If you want the easy starting point, use Ollama. If you want the next layer in a homelab that is starting to do serious work, these three are where I would look next.

DEV Community: Mike

Passkeys Are the Death of Passwords — What CS Students and Self-Hosters Need to Know Now

Passkeys Are the Death of Passwords — What CS Students and Self-Hosters Need to Know Now

What a passkey actually is

Where passkeys live decides whether they feel brilliant or awful

The recovery question is the one normal people ask first

The ugly part: passkeys still break in very ordinary ways

What this looks like in a homelab

Why CS students need to care now

My view after using them

How to Determine Which Old Drives Are Still Good — SMART Data, badblocks, and Homelab Hardware Longevity

How to Determine Which Old Drives Are Still Good — SMART Data, badblocks, and Homelab Hardware Longevity

SMART data is useful, but only if you read the right parts

The USB dock problem catches almost everybody once

The tests that actually tell you something

badblocks is useful, but it is not the answer for every drive

The one SMART problem that deserves immediate attention

My keep, monitor, or retire framework

Final point: stop asking if the drive is good or bad

SSH Honeypot Live — What Watching Bots Attack a Server Teaches CS Students About Network Security

SSH Honeypot Live — What Watching Bots Attack a Server Teaches CS Students About Network Security

What the honeypot is actually showing you

My own logs tell the same story from the defensive side

How bots find a fresh SSH server so quickly

The hidden lesson: even the honeypot has failure modes

Why this maps neatly onto the CS syllabus

The simplest experiment is still the best one

The $20 Server — What Extreme Budget Computing Teaches CS Students

The $20 Server — What Extreme Budget Computing Teaches CS Students

A cheap box makes operating systems visible

What a $20 server can actually do

RAM is where the lesson starts

Old disks teach I/O bottlenecks without any speech from me

Weak CPUs teach scheduling and background noise

This is why I do not worship free cloud credits

What I would tell students to buy in Thailand

The classroom exercise I would actually run

What the machine teaches

GitLost: How AI Agents Can Leak Your Code Through a Single GitHub Issue

GitLost: How AI Agents Can Leak Your Code Through a Single GitHub Issue

What GitLost actually showed

The attack is simple because the system is over-trusted

Why this matters even if you do not use GitHub's exact setup

The real mistake: one agent, too many powers

Five changes that reduce this risk fast

1. Scope each agent to one repository whenever possible

2. Split public-writing steps away from private-reading steps

3. Treat issues, pull requests, and comments as untrusted input

4. Use narrower tokens and installation scopes

5. Log the agent's tool calls like you would log an admin session

What GitLost teaches CS students

Authentication is not authorization

Public input can change private outcomes

Security is a property of the whole system

My bottom line on agent workflows

References

Linux Signals Explained — SIGTERM, SIGKILL and Process Control

Linux Signals Explained — SIGTERM, SIGKILL and Process Control

Ctrl+C is SIGINT, not magic

SIGTERM is the polite knock

SIGKILL is the last resort

The signals students meet without realising it

SIGPIPE: why yes | head -5 stops

SIGHUP: why a remote job dies when SSH drops

SIGSTOP and SIGCONT: pausing a process without killing it

Segmentation faults are signals too

The Cambridge connection: signals and interrupt handling

What students should remember

Proxmox VE in Production — What Running It 24/7 for Over a Year Actually Teaches You

Proxmox VE in Production — What Running It 24/7 for Over a Year Actually Teaches You

Production in a homelab still means consequences

Lesson 1: Storage migrations are where confidence goes to die

Lesson 2: Backups do not mature by themselves

Lesson 3: The network is the part you forget until it bites

The practical rules I would write down on day one

What 400 days actually teaches you

The CISA GitHub Leak: What It Should Teach Homelab Owners

The CISA GitHub Leak: Homelab Security Lessons for Self-Hosters

Keep secrets out of repositories

Audit what is actually exposed

SIGPIPE: why `yes | head -5` stops