DEV Community: Big Mazzy

Fix Only Console Users Are Allowed to Run the X Server

Big Mazzy — Sun, 25 Oct 2026 13:30:50 +0000

TL;DR: If the failing xrdp session uses the wrapper launcher, change the [Xorg] executable parameter in /etc/xrdp/sesman.ini to /usr/lib/xorg/Xorg, then restart xrdp-sesman.

The problem

An xrdp connection may fail with this message:

/usr/lib/xorg/Xorg.wrap: Only console users are allowed to run the X server

This denial can appear when a remote desktop session tries to start X through Xorg.wrap. Before changing anything, confirm that the failing session actually uses that wrapper launcher. An SSH test can reproduce the denial, but by itself it does not prove which executable your remote desktop backend invokes.

The steps below are for Ubuntu 22.04 and a legacy xrdp backend configured to use the wrapper. If your session does not use that launcher, stop here; this fix does not apply.

Why it happens

Xorg.wrap is a wrapper that restricts which users can start the X server. A remote desktop session may run under a user who is not considered a console user, so invoking the wrapper can produce the denial.

The xrdp session manager’s [Xorg] section specifies the executable used to start the graphical session. In the example configuration, that section has two param lines. The first points to /usr/bin/Xorg, which can be the wrapper launcher. Replacing only that executable parameter with the direct Xorg executable avoids the wrapper check for this backend. Preserve the remaining parameters in your actual configuration.

Step-by-step fix

Inspect the session manager configuration and find the [Xorg] section. Confirm the failing session’s backend is configured to invoke the wrapper before proceeding.

   sudo less /etc/xrdp/sesman.ini

Example configuration:

   [Xorg]
   param=/usr/bin/Xorg
   param=/usr/lib/xorg/Xorg

Verify the executable in your own configuration; do not assume it matches the example.

Inspect the launcher to see whether it points to Xorg.wrap.

   grep Xorg.wrap /usr/bin/Xorg

Test as the affected user over SSH, without sudo. This example reproduces the denial:

   /usr/bin/Xorg -version

An SSH denial alone does not prove that the failing remote desktop backend uses the wrapper. If the session configuration does not use this launcher, stop rather than applying the change.

Confirm that xorgxrdp is installed.

   dpkg -s xorgxrdp

The expected sample status is:

   Status: install ok installed

Edit the session manager configuration.

   sudoedit /etc/xrdp/sesman.ini

In the [Xorg] section, replace only the executable parameter that points to the wrapper with:

   param=/usr/lib/xorg/Xorg

Keep all other parameters in place. The example’s remaining line is param=/usr/lib/xorg/Xorg; avoid adding or removing parameters based only on the example, since your actual configuration may differ.

Restart the session manager to apply the configuration change.

   sudo systemctl restart xrdp-sesman

This can interrupt remote sessions, so reconnect after the restart completes.

How to verify

Reconnect to the remote desktop and select Xorg as the session type. Confirm that the graphical desktop opens and that you can use it normally. Check the actual remote desktop after login; a successful SSH test is not a substitute for verifying the graphical session.

Any IP addresses, process IDs, or session numbers shown in examples are placeholders. Use the connection and session details that apply to your server.

Common mistakes

Applying this change because an SSH test showed the denial, without first confirming that the failing xrdp backend uses the wrapper.
Replacing or deleting more than the executable parameter in [Xorg]. Preserve all other parameters in the section.
Forgetting to restart xrdp-sesman after editing the configuration.
Treating the package status as proof that the graphical session works; reconnect and verify the desktop.
Restarting without accounting for the possibility of interrupting remote sessions.

If you want a clean Ubuntu box to practice on, PowerVPS is a reasonable choice: https://powervps.net/?from=32

Watch more fixes on Server Lab: https://www.youtube.com/channel/UCov7rfWAnS4c2wF5E9V1Yag

How to Rotate Logs on a Linux Server Without Losing Data

Big Mazzy — Sun, 25 Oct 2026 10:00:21 +0000

A disk alert fired at 2 a.m. because one application log had grown to 38 GB—and the service kept writing to it after the file was renamed. Renaming a log is not the same as making a process start a new one. Here’s how to set up log rotation, check that your application reopens its logs, and avoid the common traps that can cost you data.

What log rotation does (and doesn’t do)

Log rotation typically renames a full log, creates a fresh file at the original path, and eventually compresses or removes older files. On most Linux distributions, logrotate handles this work.

But rotation alone doesn’t tell an application to stop writing to the old file. A process usually keeps an open file descriptor, which points to the original file even after it’s renamed. The result: the new log stays empty while the renamed file keeps growing.

The safe approach is to rotate the file and tell the application to reopen its logs—or use copytruncate as a fallback when the application cannot do that.

Check what’s already installed

Before adding a configuration, check whether logrotate is present and how it’s scheduled:

logrotate --version
systemctl list-timers | grep -i logrotate

On Debian or Ubuntu, install it with:

sudo apt update
sudo apt install logrotate

On Fedora or RHEL-compatible systems:

sudo dnf install logrotate

Many systems already include a daily timer or cron job. Avoid adding a second schedule without checking first; two rotation runs can make troubleshooting confusing.

Configure a log that the application can reopen

Suppose an application writes to /var/log/myapp/app.log, runs as myapp, and has a systemd unit called myapp.service. Create a dedicated config file:

sudo nano /etc/logrotate.d/myapp

Add:

/var/log/myapp/app.log {
    daily
    rotate 14
    missingok
    notifempty
    compress
    delaycompress
    dateext
    create 0640 myapp adm
    su myapp adm

    postrotate
        /usr/bin/systemctl kill --signal=HUP --kill-whom=main myapp.service
    endscript
}

Adjust the path, user, group, and unit name for your setup. The myapp process must handle SIGHUP by closing and reopening its log file. Check its documentation before relying on that behavior. If it doesn’t support reopening logs, use the fallback below.

Here’s what the directives do:

daily checks the log once a day; rotation happens only if it isn’t empty.
rotate 14 keeps 14 rotated files. With daily rotation, that’s roughly two weeks, but actual retention depends on when the file was last rotated.
compress gzips older logs; delaycompress leaves the newest rotated file uncompressed until the next rotation.
dateext gives rotated files a date suffix, which makes them easier to identify.
create 0640 myapp adm creates a replacement file with the specified permissions and ownership.
su myapp adm runs file operations as that user and group. This is useful for logs in directories owned by an application account. Check that the logrotate version and permissions on your system support this arrangement.
postrotate runs the command that asks the service to reopen its log.

A practical tip: test the signal handling before trusting the schedule. Check that the application logs to the newly created app.log after a manual rotation. A successful systemctl command only proves the signal was sent; it doesn’t prove the application acted on it.

Test without forcing a rotation

First, check the config for errors without changing files:

sudo logrotate --debug /etc/logrotate.d/myapp

Debug mode doesn’t rotate logs or update the state file. To test the real rotation, back up a non-production log or use a test service, then run:

sudo logrotate --verbose --force /etc/logrotate.d/myapp

--force rotates regardless of the schedule, so don’t use it casually on production logs. It can also make the next scheduled run behave differently because logrotate tracks when files were last rotated.

After a test, inspect the files and ownership:

sudo ls -l /var/log/myapp/
sudo tail -n 20 /var/log/myapp/app.log

Generate a line of application activity and confirm it appears in the new app.log, not just in a rotated file. Then check the service:

sudo systemctl status myapp.service

When the application cannot reopen its log

If the application has no supported reopen mechanism, copytruncate may be the only practical option:

/var/log/myapp/app.log {
    daily
    rotate 14
    missingok
    notifempty
    compress
    copytruncate
    su myapp adm
}

With copytruncate, logrotate copies the current file and then truncates the original in place. The application continues writing through its existing file descriptor.

There’s a trade-off: a small amount of log data can be lost if the application writes between the copy and the truncate. For high-volume or audit-critical logs, prefer a supported reopen signal or another logging setup that streams to syslog or journald. Don’t treat copytruncate as perfectly lossless.

Set retention based on disk, not habit

Fourteen daily files can be reasonable for a small service, but log volume matters more than the number of files. If an app writes 500 MB a day, two weeks of uncompressed logs can use several gigabytes. Compression helps, but the savings depend on the log contents.

Check actual usage before choosing a retention period:

sudo du -sh /var/log/myapp
sudo du -h /var/log/myapp | sort -h | tail

Also check whether your application already performs its own rotation. Two independent rotation mechanisms acting on the same file can cause missing logs or unexpected filenames.

Another tip from practice: monitor free disk space separately from log rotation. Rotation can fail because of a full filesystem—the very situation it’s meant to help with. Alert before the disk is nearly full, and consider whether logs should live on a separate filesystem on busy servers.

If you’re choosing a server

For a standard Linux VM that runs an application and its logs, PowerVPS is one option to evaluate. For GPU-heavy workloads, Vast.ai offers GPU rental; compare availability, storage, and pricing for the specific workload before committing. Neither changes the rotation basics: size retention for the disk you have, and make sure the application can reopen its logs.

The Server Rental Guide is also a useful resource when comparing hosting options.

Summary

A reliable rotation setup needs more than a schedule: configure retention and permissions, then make sure the application reopens its log after rotation. Test with --debug first, and use --force only when you’re prepared for an actual rotation. When reopen signals aren’t supported, copytruncate is a workable fallback—but it has a small data-loss window. Finally, monitor disk space; logrotate is useful, not a substitute for an alert.

From Queue to Cleanup: A One-Command Vast.ai Batch Pipeline

Big Mazzy — Fri, 23 Oct 2026 08:00:02 +0000

TL;DR: We use one local command to search for a GPU, start an instance, wait for SSH, upload a queue, run its processing script, copy results home, verify the transfer, and destroy the instance. A detached watchdog sets a spending deadline in case the controller crashes. The script below is a starting point: supply the model image, an offer query, and a queue-specific process.sh. Keep a separate backup of valuable results before destroying the instance.

Why we put the steps in one command

We are a small content team that rents GPUs on Vast.ai for AI video work. Earlier in this series, we covered individual model runs and the setup they needed. The recurring problem was everything around inference: finding a suitable machine, pulling weights, moving inputs and outputs, and remembering to clean up.

Setup can dominate a short run. In one LatentSync self-test, the queued batch took 15.9 minutes, including 11 minutes of setup. Running more jobs in one session made that setup easier to justify. It also made cleanup more consequential: a stopped instance still billed for storage.

Our pipeline has two rules. Results must pass a local count and size check before normal cleanup. Separately, a watchdog must be able to destroy the instance even if the controller disappears. Those rules can conflict at the deadline, so the deadline is an emergency cap, not a promise that unfinished work will be saved.

The contract for a batch

The local queue is a directory containing inputs and an executable process.sh. The script in this article copies that directory to /workspace/queue and runs its process.sh on the rented machine. That script must put every deliverable under /workspace/results. It is where model-specific commands belong.

Prepare an image with the runtime and repository your workload needs. We built a public Docker image on GHCR using GitHub Actions, then used an on-start script to download only the selected weights. For example, DL_GROUPS=h3 selected the H3 weights, and the Vast on-start command was bash /workspace/bootstrap/onstart.sh. The image build took about 11 minutes in our setup.

The pipeline deliberately treats process.sh as a black box. An image-to-video queue and a lip-sync queue need different arguments, but they need the same rental, transfer, verification, and cleanup sequence.

Step	What the controller checks
Search	Offer is rentable, reliable, has enough disk, and has no inbound traffic charge
Create	Instance starts with the selected image, disk, label, and on-start command
Connect	SSH accepts a command before any upload begins
Process	`process.sh` exits successfully and writes to `/workspace/results`
Copy	Results arrive in the local output directory
Verify	Local file count and total bytes match the remote directory
Destroy	The instance is destroyed and the current instance list confirms it

Choose the offer before starting the clock

The CLI search we used is vastai search offers '<query>' -o 'dph+' --raw. Filtering matters more than selecting the first low hourly price. We look for reliability of at least 0.97, sufficient disk, and inet_down_cost of zero. We also check cuda_max_good or driver_version against the image. One A100 we found in Sweden had driver 535, which did not fit the software we needed.

Provision disk generously when pulling large weights. We use at least 150 GB for those sessions; 40 GB is suitable only for smaller ones. On an earlier RTX PRO 6000 Blackwell test, inbound traffic for 245 GB of model weights added $0.64. The GPU portion was $1.47, but the total was $2.18. That is why the query in the example excludes inbound traffic charges.

Read the offer returned by search before supplying the hourly rate to the controller. Its cost deadline is calculated from your stated rate, so a wrong rate makes the deadline wrong too. It is a guardrail, not accounting software: it cannot predict every charge. If comparing this approach with another vendor’s API, check that vendor’s current pricing page.

A controller and a detached watchdog

Save the following as batch.py. It uses the verified Vast CLI commands for offer search, creation, transfer, and destruction. The CLI should already be configured with your key. Set VAST_API_KEY in the environment for the separate v1 instance-list check; the script never prints it.

The watchdog is another invocation of the same file, started as a detached process. It waits until the deadline, tries destruction up to three times, then checks the v1 instance list. On a normal run, the controller destroys the instance after verification. A later watchdog check will find nothing to do.

import argparse
import json
import os
import shlex
import subprocess
import sys
import time
import urllib.request
from pathlib import Path
from urllib.parse import urlsplit


API = "https://console.vast.ai/api/v1/instances/"


def vast(*args, check=True, input_text=None):
    return subprocess.run(
        ["vastai", *map(str, args)],
        text=True, input=input_text, capture_output=True, check=check
    )


def listed(instance_id):
    key = os.environ["VAST_API_KEY"]
    request = urllib.request.Request(
        API, headers={"Authorization": f"Bearer {key}"}
    )
    with urllib.request.urlopen(request, timeout=20) as response:
        payload = json.load(response)
    items = payload.get("instances", []) if isinstance(payload, dict) else payload
    return any(str(item.get("id")) == str(instance_id) for item in items)


def destroy(instance_id):
    for attempt in range(3):
        if not listed(instance_id):
            return True
        vast("destroy", "instance", instance_id,
             check=False, input_text="y\n")
        time.sleep(5)
    return not listed(instance_id)


def ssh_prefix(instance_id):
    value = vast("ssh-url", instance_id).stdout.strip()
    if value.startswith("ssh://"):
        url = urlsplit(value)
        return ["ssh", "-p", str(url.port),
                f"{url.username}@{url.hostname}"]
    parts = shlex.split(value)
    if parts and parts[0] == "ssh":
        return parts
    raise RuntimeError(f"Unrecognized ssh-url output: {value}")


def remote(prefix, command, check=True):
    return subprocess.run(
        prefix + [command], text=True,
        capture_output=True, check=check
    )


def manifest(directory):
    files = [p for p in directory.rglob("*") if p.is_file()]
    return len(files), sum(p.stat().st_size for p in files)


def watchdog(instance_id, deadline):
    while time.time() < deadline:
        time.sleep(min(20, max(1, deadline - time.time())))
    if not destroy(instance_id):
        raise RuntimeError("Destroy was not confirmed")


def main(args):
    queue = Path(args.queue).resolve()
    output = Path(args.output).resolve()
    if not (queue / "process.sh").is_file():
        raise RuntimeError("Queue needs a process.sh")
    if output.exists() and any(output.iterdir()):
        raise RuntimeError("Output directory must be empty")
    if args.cap_usd <= 0 or args.hourly_rate <= 0:
        raise RuntimeError("Cap and hourly rate must be positive")

    offers = json.loads(vast(
        "search", "offers", args.query, "-o", "dph+", "--raw"
    ).stdout)
    offer = next(
        item for item in offers
        if item["rentable"]
        and item["reliability"] >= 0.97
        and item["disk_space"] >= args.disk
        and item["inet_down_cost"] == 0
    )
    created = json.loads(vast(
        "create", "instance", offer["id"],
        "--image", args.image, "--disk", args.disk,
        "--label", args.label, "--onstart", args.onstart,
        "--env", f"-e DL_GROUPS={args.dl_groups}",
        "--ssh", "--direct", "--raw"
    ).stdout)
    instance_id = created["new_contract"]
    deadline = time.time() + 3600 * args.cap_usd / args.hourly_rate

    subprocess.Popen(
        [sys.executable, str(Path(__file__).resolve()),
         "--watchdog", str(instance_id), str(deadline)],
        start_new_session=True,
        stdin=subprocess.DEVNULL, stdout=subprocess.DEVNULL,
        stderr=subprocess.DEVNULL
    )
    print(f"Instance {instance_id}; deadline set")
    prefix = None
    while time.time() < deadline:
        try:
            prefix = ssh_prefix(instance_id)
            if remote(prefix, "true", check=False).returncode == 0:
                break
        except (RuntimeError, subprocess.CalledProcessError):
            pass
        time.sleep(10)
    else:
        raise RuntimeError("SSH did not become ready before deadline")

    remote(prefix, "mkdir -p /workspace/queue /workspace/results")
    vast("copy", str(queue) + "/", f"{instance_id}:/workspace/queue/")
    remote(prefix, "bash /workspace/queue/process.sh")
    output.mkdir(parents=True, exist_ok=True)
    vast("copy", f"{instance_id}:/workspace/results/", str(output) + "/")

    count = remote(
        prefix,
        "python3 -c 'import pathlib,json; p=pathlib.Path("
        "\"/workspace/results\"); f=[x for x in p.rglob(\"*\") "
        "if x.is_file()]; print(json.dumps([len(f), "
        "sum(x.stat().st_size for x in f)]))'"
    )
    expected = tuple(json.loads(count.stdout))
    actual = manifest(output)
    if expected[0] == 0 or actual != expected:
        raise RuntimeError(f"Transfer mismatch: remote={expected}, local={actual}")
    print(f"Verified {actual[0]} files, {actual[1]} bytes")
    if not destroy(instance_id):
        raise RuntimeError("Results copied, but destroy was not confirmed")
    print("Instance destroyed and absence confirmed")


if __name__ == "__main__":
    parser = argparse.ArgumentParser()
    parser.add_argument("--watchdog", nargs=2)
    parser.add_argument("--queue")
    parser.add_argument("--output")
    parser.add_argument("--query")
    parser.add_argument("--image")
    parser.add_argument("--label", default="video-batch")
    parser.add_argument("--onstart", default="onstart.sh")
    parser.add_argument("--dl-groups", default="h3")
    parser.add_argument("--disk", type=int, default=150)
    parser.add_argument("--cap-usd", type=float)
    parser.add_argument("--hourly-rate", type=float)
    args = parser.parse_args()
    if args.watchdog:
        watchdog(int(args.watchdog[0]), float(args.watchdog[1]))
    else:
        main(args)

The script expects JSON output with an offer list and a creation response containing new_contract. Check those shapes against your installed CLI before relying on an unattended run. It also assumes the rented image has python3 for the remote count and size check. The watchdog inherits VAST_API_KEY, so keep that variable available until cleanup is confirmed.

A call can then be reduced to one command. Replace the image and offer query with values you inspected, and set --hourly-rate to the rate you used when calculating the cap:

python3 batch.py \
  --queue ./queue \
  --output ./results \
  --query 'reliability >= 0.97 num_gpus = 1 disk_space >= 150 inet_down_cost = 0' \
  --image YOUR_IMAGE \
  --label h3-batch \
  --onstart onstart.sh \
  --dl-groups h3 \
  --disk 150 \
  --cap-usd 1 \
  --hourly-rate YOUR_OFFER_HOURLY_RATE

For a first run, use a small queue and watch the output. The command starts the rental immediately; the cap begins at creation, so weight downloads and SSH startup consume part of it. Increase the cap only after allowing for setup and the work you have actually queued.

Verify, back up, then destroy

The local comparison catches incomplete transfers by checking file count and total bytes. It does not prove that two same-sized files have identical contents. For valuable output, keep a separate backup before destruction. In our workflow, we copied results to a cloud remote with rclone and checked its file count and size; one backup check matched 5,342 files and 1.379 GiB against local storage.

We learned the cleanup rule the expensive way. A stopped instance with a 90 GB disk kept billing about $0.0167 per hour, or about $0.40 per day. We left one stopped for days, the balance went negative, the stopped instance could not be restarted, and we never downloaded the trained result. Now “finished” means results are copied, backed up, and the instance is destroyed. We verify absence through the v1 instances endpoint because an older v0 listing returned an error that looked like “0 instances.”

The code sends y to vastai destroy instance, matching the CLI prompt we encountered. Newer CLI versions also offer -y. If processing fails, the controller raises an error and leaves recovery time until the watchdog deadline; inspect the instance promptly. Once that deadline arrives, the watchdog prioritizes the cap.

What it cost us

On a Vast A100 SXM4 80GB in Czechia, MiniMax H3 image-to-video took about 86 seconds per five-second clip. The GPU rate was $1.19 per hour, about $1.25 with disk, and the run cost about $0.03 per clip. A session producing 82 hero clips cost $3.59 over about three hours, including setup; 72 GB of weights downloaded in roughly eight minutes.

Our earlier RTX PRO 6000 Blackwell test ran for 56 minutes and cost $2.18: $1.47 for GPU time and $0.64 for inbound traffic on 245 GB of weights. A LatentSync 1.6 run on a Vast RTX 3090 at $0.27 per hour with free inbound took 238 seconds for one clip, about $0.09. These are our runs, not promises about future offers.

Conclusion

A batch script does not make rented GPU work effortless. It gives each session a repeatable beginning and a definite end. We still inspect the offer, match its driver to the image, size the disk for the weights, and keep model commands inside the queue’s processing script. The payoff is that setup serves a batch, outputs are checked before cleanup, and a controller failure has a deadline.

If you want to try the same approach, this is our Vast.ai referral link. Start with a small queue, confirm that your CLI and image match the skeleton, and make destruction part of the successful path.

Caching Strategies for Dynamic Websites on a Budget VPS

Big Mazzy — Thu, 22 Oct 2026 09:00:43 +0000

A page that normally loaded in 180 ms took nearly four seconds during a small traffic spike—not because the VPS ran out of CPU, but because every request repeated the same database work. A short cache at the web server would have absorbed most of that load. This walkthrough covers a practical caching stack for a budget VPS, including what to cache, how to avoid leaking personalized pages, and how to check that the cache is actually working.

Start with the cheapest wins

Caching is not one switch. It’s a few layers, each useful for different kinds of content:

Browser caching reduces repeat downloads of static files.
Reverse-proxy caching lets Nginx serve eligible responses without calling your application.
Application or data caching avoids repeating expensive database queries or computations.

Start with the layer that matches the bottleneck. If a page is slow because it runs a costly query, caching the CSS won’t fix it. If the page is fast but users repeatedly download a large bundle, browser caching is an easy win.

Before changing anything, get a baseline. Check a representative page a few times and note its response time, status code, and cache-related headers. Then check your application and database metrics if you have them. A cache that serves the wrong content is worse than a slow page.

Cache static assets in the browser

Give versioned assets—such as app.a1b2c3.js—a long browser lifetime. The filename changes when the content changes, so returning visitors can reuse the old download safely until they receive the new filename.

For Nginx, a typical rule looks like this:

location ~* \.(css|js|png|jpg|jpeg|gif|svg|webp|woff2)$ {
    expires 30d;
    add_header Cache-Control "public, max-age=2592000, immutable";
}

Use immutable only if your build process changes the asset URL when its contents change. If you overwrite app.js in place, a 30-day cache can leave users running stale code. For HTML, use a shorter lifetime or revalidate; pages often need to reflect changes sooner than static assets.

Add a small Nginx microcache

For public pages that are expensive to render but can tolerate a few seconds of staleness, Nginx’s proxy cache is a good fit. A 5–15 second cache can collapse a burst of identical requests into one application request without making a page feel stale to most visitors.

The example below assumes Nginx proxies requests to an application listening on 127.0.0.1:3000. Put the map and proxy_cache_path directives in the http block, not inside a server block:

http {
    proxy_cache_path /var/cache/nginx
        levels=1:2
        keys_zone=site_cache:10m
        max_size=256m
        inactive=10m
        use_temp_path=off;

    map $http_cookie $skip_cookie_cache {
        default 1;
        ""      0;
    }

    map $http_authorization $skip_auth_cache {
        default 1;
        ""      0;
    }

    server {
        listen 80;
        server_name example.com;

        location / {
            proxy_pass http://127.0.0.1:3000;
            proxy_set_header Host $host;
            proxy_set_header X-Real-IP $remote_addr;
            proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
            proxy_set_header X-Forwarded-Proto $scheme;

            proxy_cache site_cache;
            proxy_cache_key "$scheme$request_method$host$request_uri";
            proxy_cache_methods GET HEAD;
            proxy_cache_valid 200 10s;
            proxy_cache_valid 404 1s;
            proxy_cache_lock on;

            proxy_cache_bypass $skip_cookie_cache $skip_auth_cache;
            proxy_no_cache     $skip_cookie_cache $skip_auth_cache;

            add_header X-Cache $upstream_cache_status always;
        }
    }
}

This conservative setup bypasses caching whenever a request has a cookie or an Authorization header. That’s safer than trying to guess which cookies imply a logged-in user. If your site sets an anonymous analytics cookie on every visit, this rule may bypass more requests than necessary; refine it only after confirming which cookies affect page content.

Check the configuration and reload Nginx:

sudo nginx -t
sudo systemctl reload nginx

Then request a public page twice:

curl -sSI https://example.com/ | grep -iE 'HTTP/|x-cache|cache-control'

The first request will often show MISS; a later request may show HIT. A response can still be uncacheable if the application sends headers such as Cache-Control: private or no-store. That’s usually intentional—don’t override those headers until you understand why they’re there.

proxy_cache_lock on is a useful, non-obvious safeguard: when a popular item expires, it helps prevent many simultaneous requests from all regenerating it at once. Also, keys_zone=site_cache:10m is Nginx’s in-memory index, not the maximum disk cache size; max_size=256m sets the disk limit.

Keep personalized and changing content out

A page containing a user’s name, cart, account balance, or private dashboard should not be served from a shared cache. Bypassing on cookies and authorization headers is a safe starting point, but the application should also send appropriate Cache-Control headers for private responses.

Be equally deliberate with query parameters. Nginx’s default cache key includes the request URI, including its query string. That’s often right for searches or filters, but it can create many nearly identical cache entries for tracking parameters such as utm_source. If you normalize URLs, confirm first that the removed parameters never change the response.

For content that changes immediately—inventory, account settings, or breaking news—use no cache or a very short lifetime. For articles and public landing pages, a few seconds may be enough to smooth traffic spikes while keeping updates reasonably fresh.

Use application caching for expensive work

A reverse proxy can cache whole responses; it can’t efficiently cache every expensive operation inside your application. If one database query dominates page generation, cache that result in the application using a key that includes every input that affects the answer.

For example, a product list might be cached by category and page number, with a short expiration. Avoid a single key such as products if different users, locales, currencies, or filters receive different results.

A Redis server can be useful for shared application data, but it adds another service to monitor and size. On a small VPS, start with a bounded cache and watch memory use. An unbounded in-process cache can quietly consume the RAM your database or application needs.

Fit the cache to the VPS

Disk cache is useful when RAM is limited, but it is not free: the cache competes with logs, uploads, and database files for storage. Start small—perhaps a 128–256 MB Nginx cache—and inspect disk usage before increasing it. For a single-site VPS, a short microcache and browser caching are often better starting points than adding multiple services.

When choosing a VPS, compare memory, disk type, bandwidth, and support as well as the monthly price. PowerVPS is one option worth comparing for a small web deployment. For a site that adds AI-generated features, Vast.ai can be useful for GPU rental, but a GPU usually won’t improve ordinary page caching; keep that workload separate unless you have a specific reason to colocate it. The Server Rental Guide is also a useful resource when comparing hosting and rental options.

Make changes safely

A practical rollout looks like this:

Record baseline response times and identify one slow, public page.
Add browser caching for versioned static assets.
Enable a short reverse-proxy cache for eligible GET and HEAD responses.
Confirm repeated requests show cache hits and personalized requests bypass the cache.
Watch application load, disk usage, and error rates during real traffic.
Adjust the cache lifetime only after checking freshness requirements.

One more tip from practice: test as both an anonymous visitor and a logged-in user. A page that looks correct in your own browser may be getting a cache hit from a previous anonymous request—or the reverse. Also test after a content update. A cache that works perfectly until an editor publishes an urgent correction still needs an invalidation plan; for a small site, a short TTL may be the simplest plan.

Summary

A budget VPS can handle more traffic without a complicated cache stack. Cache versioned static assets in browsers, use a short Nginx microcache for safe public pages, and cache expensive application data where it makes sense. Keep personalized responses out of shared caches, verify hits and bypasses with real requests, and size the cache against your actual disk and memory budget. Start small, measure the result, and expand only where the numbers point.

Before You Destroy a Rented GPU, Prove Your Results Are Safe

Big Mazzy — Tue, 20 Oct 2026 08:00:02 +0000

TL;DR We rent GPUs on Vast.ai for AI video work. Our rule now is simple: copy every result off the instance, make a second copy in cloud storage, compare file counts and byte totals, then verify content before destroying the instance. A stopped instance still bills for storage. We learned that after leaving one stopped for days; the balance went negative, we could not restart it, and we never downloaded the trained result.

Why “the job finished” is not the finish line

A render finishing on a rented GPU means the files exist on that machine. It does not mean we have them.

This distinction matters when a session produces many clips. Our MiniMax H3 session produced 82 hero clips. The GPU work was useful only if those clips made it back to our workspace. The same applies to model weights, training outputs, logs, and any intermediate files we might need to diagnose a failed run.

We once stopped an instance while a trained result was still on its disk. Storage continued billing at about $0.0167 per hour, or about $0.40 per day for a 90 GB disk. We left it stopped for days. The balance went negative, a stopped instance could not be started with that balance, and the result was never downloaded.

That mistake changed our shutdown sequence. “Stop” is not our archival step. We copy, verify, back up, and then destroy.

Give every session its own output directory

Before starting a batch, we choose one directory on the rented machine for final results. On the receiving machine, we give that session its own local directory and its own cloud backup prefix.

A dedicated prefix makes verification much easier. If an old file is already in the cloud destination, a file count might look reassuring while hiding a missing new file. It also lets us tell exactly which session a backup belongs to.

For a new Vast.ai account, this is our referral link. Check current offer details before renting: disk capacity and inbound traffic charges can change the cost of a session. We have seen inbound traffic billed separately when downloading large model weights.

Keep the output directory focused on files you intend to retain. Temporary caches and downloaded weights belong elsewhere unless you explicitly want to archive them. For large weights, we provision disk generously; our H3 setup downloaded 72 GB of weights.

Copy results back with rsync

We use an SSH connection to pull results to a local machine. vastai ssh-url ID provides the connection details for an instance. Put the host, port, and user from that connection into environment variables, then run the transfer from the receiving machine:

export INSTANCE_ID="your-instance-id"
export SSH_USER="root"
export SSH_HOST="your-instance-host"
export SSH_PORT="your-instance-ssh-port"
export LOCAL_DIR="$PWD/results/session-a"

vastai ssh-url "$INSTANCE_ID"

mkdir -p "$LOCAL_DIR"

rsync -a --partial \
  -e "ssh -p ${SSH_PORT}" \
  "${SSH_USER}@${SSH_HOST}:/workspace/outputs/" \
  "$LOCAL_DIR/"

The trailing slash on /workspace/outputs/ means “copy the contents of this directory.” The local directory is a session-specific destination. --partial helps when a connection drops during a large transfer, and rerunning rsync is a practical way to resume the copy.

The same general rule applies if you use vastai copy SRC DST: confirm the destination contains the expected files before treating the transfer as complete. A successful command is useful evidence, but we still check the files themselves.

Do not destroy the instance just because rsync exited successfully. The local copy is only the first checkpoint.

Make a SHA-256 manifest

A manifest records the expected digest of every file in the local results directory. We keep it outside that directory so it does not accidentally become part of the result set.

The script below walks regular files, rejects symbolic links, and writes paths relative to the session directory. It also prints the file count and total bytes. It rejects filenames containing newlines because they make a plain text checksum manifest harder to inspect safely.

# save as make_manifest.py
import hashlib
import pathlib
import sys

root = pathlib.Path(sys.argv[1]).resolve()
manifest = pathlib.Path(sys.argv[2]).resolve()

if not root.is_dir():
    raise SystemExit(f"Missing results directory: {root}")

files = sorted(path for path in root.rglob("*") if path.is_file())

if any(path.is_symlink() for path in root.rglob("*")):
    raise SystemExit("Results contain a symbolic link; inspect it first")

total_bytes = 0

with manifest.open("w", encoding="utf-8") as output:
    for path in files:
        relative = path.relative_to(root).as_posix()
        if "\n" in relative or "\r" in relative:
            raise SystemExit(f"Unsupported filename: {relative!r}")

        digest = hashlib.sha256()
        with path.open("rb") as source:
            for chunk in iter(lambda: source.read(1024 * 1024), b""):
                digest.update(chunk)

        output.write(f"{digest.hexdigest()}  {relative}\n")
        total_bytes += path.stat().st_size

print(f"files={len(files)} bytes={total_bytes}")

For example, run it with python3 make_manifest.py "$LOCAL_DIR" "$PWD/SHA256SUMS.session-a". Save the manifest alongside your local backup records. If a file changes later, its digest changes too.

A manifest made after the local transfer cannot prove what existed on the rented machine before the transfer. It gives us a clear record of what we received and a basis for checking later copies. If the source application reports an expected output count, compare that count with the local result as well.

Back up to cloud storage and compare the copies

We use rclone to copy the local results to a cloud remote, then upload the manifest. The remote path below should be a fresh prefix for this session:

export REMOTE_DIR="your-remote:gpu-results/session-a"
export MANIFEST="$PWD/SHA256SUMS.session-a"

python3 make_manifest.py "$LOCAL_DIR" "$MANIFEST"

rclone copy "$LOCAL_DIR" "$REMOTE_DIR"
rclone copyto "$MANIFEST" "$REMOTE_DIR/SHA256SUMS"

Our verification has three parts:

Compare the names and sizes of the local result files with a recursive listing of the cloud prefix.
Compare file contents, downloading remote data for the check where needed.
Download the cloud copy of the manifest and compare it with the local manifest.

Here is a compact pre-destroy check to run after the upload. It assumes LOCAL_DIR, REMOTE_DIR, and MANIFEST are set as above.

#!/usr/bin/env bash
set -euo pipefail

: "${LOCAL_DIR:?Set LOCAL_DIR}"
: "${REMOTE_DIR:?Set REMOTE_DIR}"
: "${MANIFEST:?Set MANIFEST}"

listing="$(mktemp)"
remote_manifest="$(mktemp)"
trap 'rm -f "$listing" "$remote_manifest"' EXIT

rclone lsjson "$REMOTE_DIR" --recursive --files-only > "$listing"

python3 - "$LOCAL_DIR" "$listing" <<'PY'
import json
import pathlib
import sys

root = pathlib.Path(sys.argv[1])
listing = json.loads(pathlib.Path(sys.argv[2]).read_text())

local = {
    path.relative_to(root).as_posix(): path.stat().st_size
    for path in root.rglob("*")
    if path.is_file()
}

remote = {
    item["Path"]: item["Size"]
    for item in listing
    if item["Path"] != "SHA256SUMS"
}

if local != remote:
    missing = sorted(local.keys() - remote.keys())
    extra = sorted(remote.keys() - local.keys())
    changed = sorted(
        name for name in local.keys() & remote.keys()
        if local[name] != remote[name]
    )
    raise SystemExit(
        f"Backup mismatch: missing={missing}, extra={extra}, "
        f"different_size={changed}"
    )

print(f"Matched files={len(local)} bytes={sum(local.values())}")
PY

rclone check --download "$LOCAL_DIR" "$REMOTE_DIR" --exclude SHA256SUMS
rclone copyto "$REMOTE_DIR/SHA256SUMS" "$remote_manifest"
cmp "$MANIFEST" "$remote_manifest"

printf '%s\n' "Local results and cloud backup verified."

The name and size comparison catches missing, extra, and truncated files quickly. rclone check --download adds a content comparison; it may transfer substantial data, so allow time for it. The downloaded manifest confirms that the cloud prefix also contains the exact checksum record we generated locally.

In one backup verification, we matched 5,342 files and 1.379 GiB between local storage and the cloud remote. Those were the checks we wanted to see before removing the rented disk.

Our pre-destroy checklist, in the order we run it:

Step	Evidence we want
`rsync` from the instance	Exit code 0 and the expected files locally
SHA-256 manifest	File count and byte total recorded
`rclone copy` to a fresh prefix	Listing matches names and sizes
`rclone check --download`	Contents match
Manifest compared in the cloud	`cmp` reports no difference

The last check happens after destroy

Once the local copy, cloud copy, and manifest checks pass, we destroy the instance. The CLI prompts for confirmation; this form supplies the answer for a scripted shutdown:

echo y | vastai destroy instance "$INSTANCE_ID"

We then verify that the instance is gone. An old v0 instances listing was deprecated and once returned an error that looked like “0 instances,” so we do not treat that response as proof. We check the v1 instances endpoint at https://console.vast.ai/api/v1/instances/ with an Authorization: Bearer <key> header, keeping the key in an environment variable or a protected file rather than printing or hard-coding it.

We also use a cost cap and a detached deadline watchdog during unattended sessions. The watchdog retries destroy if the controller crashes. It is a guardrail for the rental meter, not a replacement for copying results promptly.

What it cost us

Our H3 batch on a Vast A100 SXM4 80GB in Czechia produced 82 hero clips during a roughly three-hour session for $3.59, including setup. A separate LatentSync 1.6 queued batch self-test took 15.9 minutes and cost $0.07; setup accounted for about 11 minutes. The painful cost was the stopped 90 GB disk: about $0.0167 per hour, or $0.40 per day, followed by a trained result we never recovered. These are our runs, not price guarantees.

Batching work makes GPU rental economical for us because setup takes a meaningful share of a short session. Safe shutdown is part of that economics. We treat the local copy, the cloud copy, and their verification as deliverables of the job. Only then is the instance finished.

Fix colord Color Profile Authentication Popup in xRDP Ubuntu 24.04

Big Mazzy — Mon, 19 Oct 2026 13:33:18 +0000

TL;DR: Create a colord-users group for the RDP user and add a polkit rule that allows only the specified colord actions in active sessions.

The problem

On Ubuntu 24.04 over xRDP, opening a color setting can show an authentication popup: “Authentication is required to create a color managed device.” It interrupts what should be a routine desktop task and asks for administrator authentication.

This guide applies a scoped fix. It grants the RDP user permission for two color-management actions when their session is active. It does not grant sudo access or broadly remove authentication requirements.

Why it happens

The session matters. Polkit treats a remote session as non-local, and the default colord policy asks for administrator authentication. There is no stock rule covering remote sessions for these colord actions, so the prompt appears.

The rule below checks three conditions: the requested action is one of two colord actions, the user belongs to a dedicated group, and the session is active. If all three conditions match, the action is allowed. Unrelated actions still follow their existing policies.

The session ID, user name, and numeric IDs shown in the examples are placeholders. Use the session ID and RDP user that apply to your system; the sample output illustrates the expected format.

Step-by-step fix

Check the current session ID:

   echo $XDG_SESSION_ID

Example output:

Inspect the session’s remote status and type. Replace 4 with the session ID from the previous step if needed:

   loginctl show-session 4 -p Remote -p Type

Example output:

   Remote=yes
   Type=x11

Remote=yes confirms that this session is remote. The output’s session ID and session properties are examples; your values may differ.

Create the dedicated group:

   sudo groupadd colord-users

Add only the RDP user to that group. Replace rdpuser with the account used to log in over RDP:

   sudo usermod -aG colord-users rdpuser

Check the account’s group membership:

   id rdpuser

Example output:

   uid=1002(rdpuser) gid=1002(rdpuser) groups=1002(rdpuser),1004(colord-users)

The numeric IDs and account details in this sample are placeholders. Confirm that colord-users appears in the groups list.

Create /etc/polkit-1/rules.d/50-colord-xrdp.rules with exactly this content:

   polkit.addRule(function(action, subject) {
   var ok = ["org.freedesktop.color-manager.create-device", "org.freedesktop.color-manager.create-profile"];
   if (ok.indexOf(action.id) != -1 && subject.isInGroup("colord-users") && subject.active)
   return polkit.Result.YES;
   });

This rule allows the two listed colord actions for members of colord-users only while their session is active.

Log out of the desktop session and back in over RDP. The new group membership takes effect in a fresh login session, so reconnect before checking the permission.

How to verify

After logging in again, check the create-device action:

A=org.freedesktop.color-manager.create-device
pkcheck -a $A -p $$; echo $?

Example output:

A result of 0 means colord is allowed for this action in the current session.

Then check an unrelated systemd action:

B=org.freedesktop.systemd1.manage-units
pkcheck -a $B -p $$; echo $?

Example output:

A result of 1 here means administrator authentication is still required for that unrelated action. The PID and result values shown are examples; the PID is supplied by $$ in your shell. Together, these checks show that the rule grants the intended colord permission without allowing systemd unit management.

Common mistakes

Do not add every desktop user to colord-users; the rule is intended for the RDP account that needs the color settings. Check the account name carefully when adding it to the group.

Do not skip logging out and back in after changing group membership. An existing session may not reflect the new membership, which can make a correct rule appear ineffective.

Keep the rule’s action list, group check, and active-session check intact. Removing a condition broadens who can use the exception or when it applies. If the create-device check does not return 0, confirm the session is active, the RDP account belongs to colord-users, and the file contents match the rule above.

If you want a clean Ubuntu box to practice on, PowerVPS is a reasonable choice: https://powervps.net/?from=32

Watch more fixes on Server Lab: https://www.youtube.com/channel/UCov7rfWAnS4c2wF5E9V1Yag

Running Stable Diffusion on a Rented GPU Server

Big Mazzy — Mon, 19 Oct 2026 09:00:25 +0000

A rented GPU that looked cheap on the hourly listing cost me more than expected: I left it running overnight after the first image finished. The model took minutes to load, but the idle machine kept billing for hours. This walkthrough covers choosing a GPU, setting up Stable Diffusion on Ubuntu, connecting securely, and avoiding that kind of surprise.

Choose a server that fits the model

For a first run, an NVIDIA GPU with 8 GB of VRAM is a reasonable target for Stable Diffusion 1.5. SDXL is more comfortable with 12 GB or more, though actual requirements depend on resolution, batch size, and optimizations. Check the GPU model and VRAM before renting; a listing’s “GPU memory” and its system RAM are different things.

Disk space matters too. I’d start with at least 50 GB of persistent storage for the operating system, Python dependencies, model checkpoints, and generated images. More is useful if you plan to keep multiple models.

Vast.ai is a good option I’ve used and tested for on-demand GPU rentals, with a range of hardware and prices. PowerVPS is another provider I’ve used and tested for VPS workloads; check its current offerings if you need a longer-lived server or supporting services, and confirm GPU availability before choosing a plan. In either case, compare the total cost, including storage and any charges that continue when the GPU is idle. The Server Rental Guide is a useful resource when comparing rental options and terms.

Start with a clean Ubuntu server

Choose an Ubuntu 22.04 image with NVIDIA drivers already installed if the provider offers one. That avoids spending your first session debugging driver setup. After connecting over SSH, confirm the GPU is visible:

nvidia-smi

You should see the GPU model, driver version, and memory. If the command is missing or reports no devices, check the provider’s GPU image and driver instructions before installing anything else. Driver installation varies by image and provider.

Install the basic tools:

sudo apt update
sudo apt install -y git python3 python3-venv python3-pip

For this example, use AUTOMATIC1111’s Stable Diffusion WebUI. Its first startup installs Python dependencies, so allow time for that, and keep the server’s disk persistent if you want to reuse the installation.

git clone https://github.com/AUTOMATIC1111/stable-diffusion-webui.git
cd stable-diffusion-webui

You’ll need a model checkpoint that you’re allowed to use. Download it through the model host’s normal process, then copy it to the server or download it from the server if the host provides a direct link. Some model hosts require accepting a license or signing in. Put the .safetensors file in the WebUI’s model directory:

mkdir -p models/Stable-diffusion
# Copy your checkpoint into models/Stable-diffusion/

For example, from your local computer, copy a file with scp:

scp ./my-model.safetensors USER@HOST:stable-diffusion-webui/models/Stable-diffusion/

Replace USER and HOST with the SSH login details supplied by your provider. This command assumes the clone is in the remote user’s home directory.

Launch it without exposing the UI publicly

Start the WebUI so it listens on the server’s network interfaces:

./webui.sh --listen --port 7860

Keep this process running in your SSH session for the first test. The UI does not require you to open port 7860 to the internet. Instead, keep the provider firewall or security group closed to inbound traffic on that port and create a tunnel from your local machine:

ssh -L 7860:127.0.0.1:7860 USER@HOST

Then open http://127.0.0.1:7860 in your local browser. The tunnel forwards your local port to the WebUI on the server. If port 7860 is already in use locally, choose another local port, for example:

ssh -L 7861:127.0.0.1:7860 USER@HOST

In that case, open http://127.0.0.1:7861. Avoid exposing an unauthenticated image-generation UI directly to the public internet.

Generate a first image and check utilization

Select the checkpoint in the UI, enter a prompt, and start with a modest configuration: 512×512 resolution, one image, and around 20 steps. Once that works, try larger resolutions or other settings.

Watch GPU usage in a second SSH session:

nvidia-smi

A GPU that appears underutilized may be waiting on model loading, disk access, or a setting that sends work to the CPU. Don’t assume a higher step count is always better; it increases generation time, and the visual difference depends on the prompt and sampler.

Two rental habits that save trouble

Keep models on persistent storage. Many rental platforms distinguish between stopping a machine and deleting it. A stopped instance may retain its disk; deleting it may remove the model files too. Check the provider’s rules before you end a session, and copy anything important off the server.

Treat the hourly rate as only part of the cost. A GPU with more VRAM may finish a job faster, but the cheapest listing is not automatically the cheapest run. For a short task, compare estimated runtime as well as hourly price. Also check whether persistent disks are billed while the GPU is stopped. The details vary by provider and rental type.

Shut down when you’re done

Stop the WebUI with Ctrl+C in its SSH session. Then stop or release the rented instance using the provider’s controls. Don’t assume closing SSH stops the server. On some platforms, the machine keeps running—and billing—until you explicitly stop or terminate it.

If you expect to return later, choose the option that preserves the disk and confirm its storage charges. If you no longer need the files, back up the outputs and remove the instance according to the provider’s instructions.

Summary

A rented GPU makes Stable Diffusion accessible without buying hardware, but the setup is only half the job: model storage, network access, and shutdown behavior all affect the experience and the bill. Start with a preconfigured Ubuntu GPU image, verify it with nvidia-smi, use a model you’re licensed to run, and access the WebUI through an SSH tunnel. Most importantly, understand what happens to the server and its disk when you stop renting it.

Fix No Space Left on Device on Ubuntu 24.04 Server

Big Mazzy — Sun, 18 Oct 2026 13:31:14 +0000

TL;DR: Find the known oversized leftover file on the full lab volume, remove it, then retry the write and check that free space has returned.

The problem

A write can fail with “No space left on device” even when you’re working on a separate lab volume rather than the server’s root disk. In this example, the lab volume is 64 MB. Filling it with a large file leaves no room for a normal write.

The failed write is a symptom, not a diagnosis. First confirm which filesystem is full, then identify what is using its space. The commands below follow that path and remove only the known lab leftover.

Why it happens

Filesystems track both data capacity and inodes. Bytes measure the space used by file contents; inodes track how many files and directories can be created. Either limit can prevent a new file from being written.

Here, the volume is at 100% usage with zero bytes available. The large file accounts for 57 MB. The inode limit is fine: this is a bytes-full problem, not an inode shortage. The lost+found permission message is separate; it means the current user cannot read that directory, and does not prevent du from reporting the large file.

Step-by-step fix

Reproduce the failure by writing zeroes to the lab file. The command stops when the volume runs out of space:

dd if=/dev/zero of=/srv/lab/export.csv bs=1M

Example error:

dd: error writing '/srv/lab/export.csv': No space left on device

Try a normal write so you can see whether the filesystem also rejects a small file:

echo test | tee /srv/lab/new.txt

Example error:

tee: /srv/lab/new.txt: No space left on device

Check the capacity of the lab mount:

df -h /srv/lab

Example output:

Filesystem Size Used Avail Use% Mounted on
/dev/loop0 59M 59M 0 100% /srv/lab

Find the large entry under the lab directory:

du -sh /srv/lab/*

Example output:

57M /srv/lab/export.csv
du: cannot read directory '/srv/lab/lost+found': Permission denied

Remove the known lab leftover:

rm /srv/lab/export.csv

The target is specific: /srv/lab/export.csv. Check that you have identified the intended file before removing anything on a real server.

How to verify

Retry the same small write, then check the mount’s capacity again:

echo test | tee /srv/lab/new.txt

df -h /srv/lab

The write should print test. The filesystem should show available space again:

test
/dev/loop0 59M 2.0M 57M 4% /srv/lab

The IP, PID, and session numbers used in examples are placeholders where relevant; this walkthrough’s sample output does not include them. On a real server, confirm the mount and the file before removing anything, since deleting the wrong file can disrupt a service or remove needed data.

Common mistakes

Treating every “No space left on device” error as an inode problem. Check filesystem usage first; in this case, bytes are full and inodes are fine.
Assuming the root disk is full. The example concerns /srv/lab, a separate volume, so inspect the filesystem that contains the affected path.
Deleting a large file just because it appears in a usage report. Confirm it is a known leftover before removing it.
Misreading the lost+found permission message as the cause. It only reports that du could not read that directory.
Expecting the original large file to remain after removal. The verification checks a new write and confirms that space is available again.

If you want a clean Ubuntu box to practice on, PowerVPS is a reasonable choice: https://powervps.net/?from=32

Watch more fixes on Server Lab: https://www.youtube.com/channel/UCov7rfWAnS4c2wF5E9V1Yag

Fix Cannot Connect to Docker Daemon on Ubuntu 24.04

Big Mazzy — Sat, 17 Oct 2026 13:30:45 +0000

TL;DR: On a test VM with no containers you need, check whether Docker is inactive and start it with sudo systemctl start docker.

The problem

You run a Docker command on Ubuntu 24.04 and get a connection error instead of a container list:

Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?

This means the Docker client cannot reach the daemon through its default Unix socket. The command itself may be fine; the daemon may simply be stopped.

The steps below reproduce that situation by stopping Docker, check its state, start it again, and verify that it responds. Use this procedure only on a test virtual machine with no containers you need. It uses a local Docker Engine with its default socket.

Why it happens

Docker has a client and a daemon. The docker command is the client; the daemon runs as a system service and handles requests such as listing containers. If the daemon is not running, the client cannot get a response through /var/run/docker.sock.

On this test VM, the socket and Docker services are stopped deliberately to reproduce the failure. The sudo docker ps command then demonstrates what the error looks like. Checking the Docker service state helps confirm the cause before you start it again.

Step-by-step fix

Use a safe test environment. Do not stop Docker on a real server or a machine running containers you depend on. These commands reproduce the issue by stopping socket activation first and then stopping Docker. After that, list containers with sudo to see the connection error. IP addresses, process IDs (PIDs), and session numbers in examples are placeholders where relevant.

   sudo systemctl stop docker.socket

   sudo systemctl stop docker

   sudo docker ps

Check whether Docker is active. Query the service state:

   sudo systemctl is-active docker

For this reproduction, the expected sample output is:

   inactive

inactive confirms that the Docker service is stopped. If your situation differs, use the state shown by the command to guide your diagnosis rather than assuming the daemon is down.

Start the Docker service. Once you have confirmed that the service is inactive, start it with system control:

   sudo systemctl start docker

Starting Docker should make the daemon available again through its default Unix socket.

Repeat the container listing. Use the same listing command to check whether the client can now reach the daemon:

   sudo docker ps

How to verify

A successful response includes the container table header:

CONTAINER ID   IMAGE   COMMAND   CREATED   STATUS   PORTS   NAMES

The header proves the daemon responded to the request. An empty list is fine: it means Docker answered, and there are no running containers to show. The important change is that the command now returns the table instead of the connection error.

If you still see a connection error, check the service state again with the command above and make sure the start step completed. Keep this reproduction limited to a test VM; stopping Docker on a server can interrupt workloads.

Common mistakes

Skipping the safety check: stopping the socket and Docker services is only for a test VM with no containers you need.
Starting the wrong thing first: the reproduction stops docker.socket before docker. Follow that order when recreating the issue.
Mistaking an empty list for failure: the table header means the daemon responded, even when no containers appear.
Treating the client as the daemon: docker ps requests information; the Docker service must be running to answer it.

If you want a clean Ubuntu box to practice on, PowerVPS is a reasonable choice: https://powervps.net/?from=32

Watch more fixes on Server Lab: https://www.youtube.com/channel/UCov7rfWAnS4c2wF5E9V1Yag

Fix externally-managed-environment on Ubuntu 24.04 with venv

Big Mazzy — Fri, 16 Oct 2026 13:30:46 +0000

TL;DR: Install python3-venv, create a virtual environment for your app, and use that environment’s pip to install FastAPI.

The problem

On Ubuntu 24.04, a pip install into the system Python can fail with this message:

error: externally-managed-environment

For example, the command below tries to install FastAPI into the Python environment managed by Ubuntu:

python3 -m pip install fastapi

That error can be confusing if you are used to running pip directly on other systems. It does not mean Python is broken, and it does not mean FastAPI is unavailable. Ubuntu is stopping pip from changing packages in its system-managed Python environment.

The practical fix is to create a separate virtual environment for your application. Install the application dependency there, keeping it apart from packages installed and maintained by Ubuntu.

Why it happens

Ubuntu uses apt to manage its system Python packages. That Python installation is part of the operating system, and other system tools may rely on its packages and versions. If pip changes those packages independently, it can create conflicts or leave system software in an unexpected state.

The externally managed environment error is a guardrail: it prevents pip from writing into that protected system environment. For an application, use a virtual environment instead. A venv gives the app its own place for Python dependencies, separate from system packages.

On a clean Ubuntu image, you may need to install the venv support package first. Without it, creating the environment can fail with an ensurepip error. Installing python3-venv with apt provides the support needed for the next step.

Step-by-step fix

First, reproduce the issue by attempting the system-level install. This command is included to show what triggers the error; it is not the command to use for installing the app dependency.

   python3 -m pip install fastapi

Ubuntu may respond with error: externally-managed-environment. That is the expected protection for the managed Python environment.

Install the venv support package and create a virtual environment for the app. The command uses apt for Ubuntu’s system package and then creates the environment at ~/app/.venv.

   sudo apt install -y python3-venv && python3 -m venv ~/app/.venv

If the app directory does not exist, venv creation may report an error. In this example, ~/app/.venv is the target path; keep the same path in the install and verification steps so they use the environment you created.

Install FastAPI using the pip executable inside the virtual environment.

   ~/app/.venv/bin/pip install fastapi

This uses the environment’s pip directly. Activation is optional: a full executable path makes clear which pip is being run and avoids relying on shell activation.

How to verify

Ask the virtual environment’s pip to show FastAPI’s package information:

~/app/.venv/bin/pip show fastapi

The output includes the package name:

Name: fastapi

That name confirms FastAPI is installed in the environment queried by this command. The important detail is the executable path: ~/app/.venv/bin/pip points to the app’s virtual environment, rather than Ubuntu’s system pip. If you use this approach for another application dependency, use the same environment’s pip path for both installation and inspection.

Common mistakes

Installing into system Python again: Running python3 -m pip install fastapi repeats the operation that Ubuntu blocks. Use the venv’s pip path for the application package.
Skipping python3-venv: On a clean image, venv support may be missing, and environment creation can stop with an ensurepip error. Install the Ubuntu package before creating the venv.
Using a different pip by accident: A bare pip command may refer to a different Python environment. The full path in these steps makes the target explicit.
Checking the wrong environment: Run pip show through the same ~/app/.venv/bin/pip path used for installation. Otherwise, the result may describe a different Python environment.
Assuming activation is required: You can use the virtual environment’s executable paths directly, so these steps do not depend on activation.
Treating sample details as literal server values: Any IP addresses, process IDs (PIDs), or session numbers in examples are placeholders where relevant; they are not values to copy as part of this Python fix.

If you want a clean Ubuntu box to practice on, PowerVPS is a reasonable choice: https://powervps.net/?from=32

Watch more fixes on Server Lab: https://www.youtube.com/channel/UCov7rfWAnS4c2wF5E9V1Yag

How to Automate Server Provisioning with Bash Scripts

Big Mazzy — Fri, 16 Oct 2026 09:00:23 +0000

A new VM came up with the wrong firewall defaults, and a “quick” manual fix turned into 40 minutes of checking package state and user permissions. The next time, I wrote down the steps as a script—and made it safe to run again. In this guide, we’ll build a small, repeatable Bash bootstrap for Debian and Ubuntu servers, then cover how to run and verify it without locking yourself out.

Decide what the script should own

A provisioning script is most useful when it handles a short list of predictable tasks: install baseline packages, create an administrative user, install that user’s public SSH key, and set basic firewall rules.

Keep application deployment, secrets, and provider-specific networking out of a first-pass script. Those concerns vary by project and environment. This example also does not change SSH daemon settings or disable password authentication; make those changes only after confirming key-based access works.

The script below expects a fresh Debian or Ubuntu server with systemd and apt. Run it from the provider’s console or over an existing SSH session as root. Don’t test a new access setup by closing your only working session.

Write the provisioning script

Save this as provision.sh:

#!/usr/bin/env bash
set -Eeuo pipefail

log() {
  printf '[%s] %s\n' "$(date -Is)" "$*"
}

fail() {
  printf 'ERROR: %s\n' "$*" >&2
  exit 1
}

[[ $EUID -eq 0 ]] || fail "Run this script as root."
[[ -n "${ADMIN_USER:-}" ]] || fail "Set ADMIN_USER."
[[ -n "${PUBLIC_KEY_FILE:-}" ]] || fail "Set PUBLIC_KEY_FILE."
[[ "$ADMIN_USER" =~ ^[a-z_][a-z0-9_-]*[$]?$ ]] || fail "Invalid ADMIN_USER."
[[ -r "$PUBLIC_KEY_FILE" ]] || fail "Cannot read PUBLIC_KEY_FILE."

if [[ -r /etc/os-release ]]; then
  # shellcheck disable=SC1091
  source /etc/os-release
else
  fail "Cannot identify operating system."
fi

case "${ID:-}" in
  ubuntu|debian) ;;
  *) fail "This script supports Debian and Ubuntu only." ;;
esac

log "Updating package metadata and installing baseline tools."
export DEBIAN_FRONTEND=noninteractive
apt-get update
apt-get install -y --no-install-recommends \
  ca-certificates \
  curl \
  fail2ban \
  git \
  ufw

if ! id "$ADMIN_USER" >/dev/null 2>&1; then
  useradd --create-home --shell /bin/bash "$ADMIN_USER"
fi

if getent group sudo >/dev/null; then
  usermod -aG sudo "$ADMIN_USER"
elif getent group wheel >/dev/null; then
  usermod -aG wheel "$ADMIN_USER"
else
  fail "Could not find a sudo or wheel group."
fi

home_dir="$(getent passwd "$ADMIN_USER" | cut -d: -f6)"
[[ -n "$home_dir" && -d "$home_dir" ]] || fail "Could not find the user's home directory."

install -d -m 700 -o "$ADMIN_USER" -g "$ADMIN_USER" "$home_dir/.ssh"
install -m 600 -o "$ADMIN_USER" -g "$ADMIN_USER" \
  "$PUBLIC_KEY_FILE" "$home_dir/.ssh/authorized_keys"

systemctl enable --now fail2ban

log "Configuring UFW. Allowing SSH before enabling the firewall."
ufw default deny incoming
ufw default allow outgoing
ufw allow OpenSSH
ufw --force enable

log "Provisioning complete for $ADMIN_USER."

Make it executable:

chmod 750 provision.sh

The script uses set -Eeuo pipefail so command failures and unset variables don’t pass silently. It also checks its operating system and inputs before making changes. The package install and user creation can be repeated; UFW rules may report that they already exist, but the final policy remains the same.

Run it with a public key

On the server, place the administrator’s public key in a readable file, such as /root/admin.pub. Do not copy a private key to the server. Then run:

sudo env \
  ADMIN_USER=deploy \
  PUBLIC_KEY_FILE=/root/admin.pub \
  ./provision.sh

If you’re already root, you can omit sudo env and set the variables directly:

ADMIN_USER=deploy PUBLIC_KEY_FILE=/root/admin.pub ./provision.sh

Check that the user exists, the services are active, and UFW allows SSH:

id deploy
systemctl is-active fail2ban
ufw status verbose

Open a second terminal and test the new account before ending your original session:

ssh -i ~/.ssh/id_ed25519 deploy@your-server

Replace the key path and server name with your own values. The account will need a public key whose matching private key is available on your client.

Be careful with firewall and SSH changes

ufw allow OpenSSH uses the firewall application profile named OpenSSH. Check that the profile exists with ufw app list; if your SSH daemon listens on a nonstandard port, allow that port before enabling UFW. Cloud firewalls and network ACLs are separate from UFW, so check both layers if SSH is unreachable.

A practical tip from experience: don’t combine “create a new admin user,” “change the SSH port,” and “disable password login” into one untested run. That makes it harder to identify the cause of a lockout. First confirm key-based login in a second session; then change SSH settings in a separate, reversible step.

Another useful habit is to keep the script in version control and test it on a throwaway VM before using it on a long-lived server. Shell scripts are executable infrastructure: a typo in a firewall rule can have a real outage attached to it.

Make it repeatable, not magical

Bash is a good fit for a small bootstrap, but it isn’t a full configuration-management system. This example doesn’t manage drift—for example, it won’t remove packages you no longer want or detect every manual change. Keep the script focused, review changes before re-running it on production, and log each run somewhere appropriate for your environment.

For more complex setups, consider adding a dry-run mode, writing a run log to a protected file, and testing with ShellCheck:

shellcheck provision.sh

If you need frequent changes across many servers, tools such as Ansible can provide clearer state management. A Bash script can still handle the initial bootstrap that installs the prerequisites.

For compute, I’ve used/tested PowerVPS for general-purpose VPS workloads and Vast.ai when I need to rent GPU capacity. Availability, pricing, and setup options depend on the workload and region, so compare them against your requirements. The Server Rental Guide is also a useful starting point when comparing rental options.

Summary

A useful provisioning script is small, explicit, and safe to repeat. This one installs a few baseline packages, creates a sudo-capable user with a public SSH key, enables Fail2ban, and configures UFW without rewriting SSH settings.

Test it on a disposable VM, verify access in a second session, and keep firewall and SSH changes deliberate. That modest discipline turns a pile of one-off commands into a dependable starting point for the next server.

What AI Video Actually Cost Us on Rented GPUs

Big Mazzy — Fri, 16 Oct 2026 08:00:02 +0000

TL;DR We rent GPUs on Vast.ai for AI video work. In one MiniMax H3 image-to-video run, generation cost about $0.03 per five-second clip; the full session cost $3.59 for 82 clips. The H3 API price in our notes was $0.08 per video second, or $0.40 for a five-second clip. Rental won clearly for that batch, but the $0.03 figure excludes setup and idle time. We use the full-session number for budgeting, count failed generations, and choose APIs when avoiding infrastructure work matters more than the possible savings.

Start with the unit you need

“Cost per clip” can mean at least two things: the cost while a model is generating, or the total bill divided by usable clips. Both are useful, but they answer different questions.

For a rented GPU, our basic calculation is:

generation cost per clip = clip runtime in seconds × hourly instance rate / 3,600

session cost per usable clip = total session bill / usable clips

session cost per usable video minute =
    total session bill / (usable video seconds / 60)

The rate should include disk when disk is billed separately. The session bill also captures time spent starting the instance, downloading weights, fixing problems, and waiting between jobs. If a generation fails or produces a clip we cannot use, it still consumes GPU time and belongs in the denominator calculation.

For an API billed by output duration, the simple calculation is price per video second × clip duration. Check the provider’s current pricing terms before applying it: model, resolution, audio, and billing rules can affect the comparison.

Our measured rental costs

Here are the runs behind our estimates. The per-clip numbers describe different accounting boundaries, so the session column matters.

Workload	Rented GPU	Clip and runtime	Recorded cost	Session context
MiniMax H3 image-to-video	A100 SXM4 80GB in Czechia, $1.19/h GPU; about $1.25/h with disk	Five-second clip, about 86 seconds to generate	About $0.03 in generation time	82 hero clips; $3.59 total, about three hours including setup
MiniMax H3 text-to-video with audio, earlier test	RTX PRO 6000 Blackwell 96GB, $1.625/h	Five-second clip, about 51 seconds to generate	About $0.02 in GPU time	56-minute session; $2.18 total, including $0.64 inbound traffic
LatentSync 1.6 lip-sync	RTX 3090 in UAE, $0.27/h, free inbound	Eight-second talk clip, 238 seconds to process	About $0.09 recorded for the clip	A separate queued batch self-test took 15.9 minutes and cost $0.07; setup took 11 minutes

The H3 A100 arithmetic is straightforward: 86 × $1.25 / 3,600 is about $0.03. At that rate, twelve five-second clips make one minute of output, so generation alone is about $0.36 per output minute. Our actual 82-clip session delivered 410 seconds of video. Dividing $3.59 by those 410 seconds gives about $0.53 per output minute, or about $0.044 per clip. That session figure is the more useful one for a budget.

The lip-sync row deserves care. At $0.27/h, 238 seconds of GPU time calculates to about $0.02, while our recorded clip cost was about $0.09. Those figures do not describe the same accounting boundary; the available notes do not fully reconcile them. We would use the recorded cost for a rough budget and measure a new batch before treating it as a repeatable unit price. At $0.09 per eight-second clip, the rough output-minute figure is $0.675.

This small Python calculation shows the boundaries explicitly:

h3_rate_with_disk = 1.25
h3_runtime_seconds = 86
h3_clip_seconds = 5
h3_session_cost = 3.59
h3_session_clips = 82

api_rate_per_video_second = 0.08

generation_cost = h3_runtime_seconds * h3_rate_with_disk / 3600
session_cost_per_clip = h3_session_cost / h3_session_clips
session_output_minutes = h3_session_clips * h3_clip_seconds / 60
api_cost_per_clip = api_rate_per_video_second * h3_clip_seconds

print(f"H3 rental, generation only: ${generation_cost:.3f}/clip")
print(f"H3 rental, full session: ${session_cost_per_clip:.3f}/clip")
print(f"H3 rental, full session: ${h3_session_cost / session_output_minutes:.2f}/output minute")
print(f"H3 API at stated rate: ${api_cost_per_clip:.2f}/clip")

Compare APIs fairly

Our H3 API reference price is $0.08 per second of video. That makes a five-second clip $0.40 and a minute of output $4.80 at that rate. For 82 five-second clips, the duration-based calculation is $32.80. Against our $3.59 rental session, that is roughly nine times the cost. Comparing only the $0.03 generation figure makes rental look about thirteen times cheaper, but hides the rest of the session.

We do not have measured, directly comparable API prices for every model people might consider:

API option	Price to use in a current comparison	What to check
MiniMax H3	$0.08 per video second in our reference	Confirm current rate and settings
Veo	Check current pricing	Model, output duration, resolution, and billing terms
Kling	Check current pricing	Model, output duration, resolution, and billing terms
Seedance	Check current pricing	Model, output duration, resolution, and billing terms

This table is a budgeting prompt, not a claim that the outputs are interchangeable. Compare results that meet the same creative need. A cheaper clip that must be regenerated several times may cost more per accepted clip; an expensive clip that works immediately may save time. We would also compare the actual workflow offered by each API, rather than assume every model supports the same inputs and controls.

Where the break-even point moves

A useful planning formula is:

rental total = hourly rate × (setup + generation + idle time) / 60
             + transfer charges + ongoing storage charges

API total = accepted and rejected billable output × API rate

In that rental formula, time is in minutes. For a quick estimate, divide expected rental total by the number of usable clips, then compare it with the API cost of obtaining those usable clips. The difficult inputs are usually how many attempts you will reject and how long the rented instance will sit idle.

Setup makes batch size important. Our H3 run downloaded 72 GB of weights in about eight minutes, and the full 82-clip session took about three hours including setup. For LatentSync, setup took roughly 9–11 minutes; in the queued self-test, that was most of a 15.9-minute run. When we queue work before renting, more output shares that startup time.

Idle time moves the answer the other way. A GPU meter can run while we inspect clips, revise prompts, or wait for the next input. Failed generations also increase cost per accepted clip. The H3 session produced 82 hero clips, but that count should not be treated as a guarantee that another batch will have the same acceptance rate. Our own time belongs in the decision too: preparing images, handling weights and drivers, watching jobs, copying results, and cleaning up are work even when the provider does not put them on the invoice.

APIs are often the better choice for a few clips, an uncertain experiment, or work that must start without environment setup. They also remove much of the operational burden when the team cannot queue a batch or does not want to manage an instance. Rental becomes more attractive when we have many ready inputs, a known workflow, and someone available to check the run and collect its outputs.

Avoid charges that spoil the calculation

The earlier Blackwell test taught us to inspect inbound traffic pricing. Its $2.18 total included $0.64 for 245 GB of model weights; GPU time accounted for $1.47. We now check inet_down_cost when looking at offers and prefer zero inbound cost for weight-heavy work. We also filter for reliability of at least 0.97, check that the driver and CUDA version fit the image, and provision enough disk. For large weights, our working threshold is at least 150 GB; 40 GB is suitable only for smaller downloads.

The CLI can help inspect offers and verify what is still running:

vastai search offers 'reliability >= 0.97 inet_down_cost = 0 disk_space >= 150' -o 'dph+' --raw
vastai show instances --raw

We made a more painful mistake with a stopped instance. Its 90 GB disk continued billing at about $0.0167/h, roughly $0.40 a day. We left it stopped for days, our balance went negative, and we could not start it to retrieve a trained result that we had never downloaded. “Stopped” was not the end of the bill.

Our rule now is to copy results back, back them up, and verify the backup before destroying the instance. In one verification, local and remote counts and sizes matched: 5,342 files and 1.379 GiB. For long runs, we use a cost cap and a detached deadline watchdog that retries destruction if the controller crashes. We then verify that the instance list is empty. The CLI destruction command prompts for confirmation; this is the form we use in automation after results are safe:

echo y | vastai destroy instance "$INSTANCE_ID"
vastai show instances --raw

What we would budget

What it cost us

H3 image-to-video on an A100: about $0.03 of generation time per five-second clip; $3.59 for 82 clips over about three hours, or about $0.044 per clip across the session.

Earlier H3 text-to-video test on Blackwell: $2.18 for 56 minutes, including $1.47 of GPU time and $0.64 of inbound traffic.

LatentSync 1.6 on an RTX 3090: about $0.09 recorded for an eight-second talk clip. A separate 15.9-minute queued self-test cost $0.07.

An overlooked stopped 90 GB disk: about $0.0167/h, or about $0.40/day.

H3 API reference: $0.08 per video second, equivalent to $0.40 for a five-second clip at that rate.

For our 82-clip H3 batch, rental saved enough to justify managing the instance. We would still price a small or unfamiliar job both ways, using current API pages and an estimate for setup, rejected output, idle time, and our own work. If you want to run that comparison on Vast.ai, this is our referral link.