DEV Community: Khadirullah Mohammad

Accidentally Pushed a Password? How to Scrub Secrets from Git History

Khadirullah Mohammad — Tue, 01 Sep 2026 07:36:47 +0000

It is the moment every developer dreads. You just pushed a massive feature to your public GitHub repository, and as you double-check your commits, your stomach drops. Staring right back at you is a hardcoded AWS Access Key.

Your first instinct might be to quickly delete the line, run git commit -m "removed api key", and push again.

Do not do this.

Git is a version control system. Its purpose is to preserve the complete history of your repository. If you just delete the key in a new commit, automated bots scraping GitHub will still find the exposed key sitting perfectly intact in your commit history.

In my previous guide, I showed you how to completely erase deleted files from Git history. Today, we are going to use the same tool, git filter-repo, to perform surgical Global Text Replacement.

We will rewrite reachable Git objects containing the leaked text and, when necessary, scrub commit messages (and annotated tag messages) using Git filter-repo's message replacement option.

⚠️ First Response: Invalidate and Backup Immediately

Before you do anything with Git, revoke the leaked credential.
If it is an AWS key, deactivate/delete the exposed key in AWS IAM immediately. (Note: GitHub's Secret Scanning often detects exposed AWS keys automatically. This is helpful, but you still must manually revoke the key!) Automated scanners can discover exposed credentials quickly, so treat any leaked secret as compromised and rotate it first.

Second, create a full backup of your repository. git filter-repo is a highly destructive command. If you make a typo in your replacements file, it will rewrite every commit in your repository. Copy your .git folder to a safe location before proceeding.

When NOT to rewrite history

Don't rewrite history if:

The repository is heavily shared among many developers, and the secret was already rotated.
Regulatory requirements strictly require preserving the original history.
The coordination effort to wipe history across all team members would be more disruptive than simply revoking the credential.

If these apply, just revoke the key and move on. Otherwise, proceed with the cleanup.

Is it too late? (The Wayback Machine Anxiety)

If you realize you leaked a secret three days ago, your first thought might be panic, followed by defeat: "Bots or internet archives probably already scraped it. What's the point of rewriting history now?"

Do not give up. This is a core cybersecurity concept called Defense in Depth.
Just because a burglar could have already smashed a window doesn't mean you should leave the front door wide open forever. Most automated secret scanners prioritize newly pushed commits and the current repository state, although historical commits may still be indexed or scanned.

By scrubbing your history now, you reduce future exposure from the repository's visible history and prevent new clones from receiving the leaked value through normal history traversal.

Why Interactive Rebase Is Usually the Wrong Tool

Many developers try to fix this by using an interactive rebase (git rebase -i). They rewind history back to the specific commit where they leaked the secret, manually delete it, and replay the history.

While this works for a typo in your most recent commit, it is a terrible idea for a leaked secret because:

Human Error: If you accidentally pasted that API key in multiple files across different commits, you will likely miss one.
Tedious: Rebasing hundreds of commits is slow and prone to painful merge conflicts.

We need a solution that is automated, thorough, and far less error-prone.

The Right Way: Global Text Replacement

git-filter-repo is the Git project's recommended replacement for the older git filter-branch command. Instead of dealing with files directly, it manipulates the internal .git database.

1. Create a Replacements File

Instead of writing a complex terminal command, filter-repo lets you use a simple text file to map out your redactions.

Create a temporary file outside of your repository (e.g., ~/replacements.txt) and define your rules using the exact string, followed by ==>, and the replacement text.

# replacements.txt
AKIAIOSFODNN7EXAMPLE==>REDACTED_AWS_KEY
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY==>REDACTED_AWS_SECRET
MySuperSecretPassword123!==>REDACTED

(Note: If you leave the right side of the arrow blank, the tool will just delete the string entirely. For more advanced cases, git filter-repo also supports regular-expression replacements, although exact string matching is usually the safest option for secrets.)

2. Verify and Execute the Scrub

Before you run the command, locate commits that introduced or removed the secret using the -S flag, which searches the exact text added or removed in past commits:

git log -p -S "AKIAIOSFODNN7EXAMPLE"

(This returns commits where the number of occurrences of the specified string changed, along with their diffs.)

Now, run the replacement command. Make sure you have no unsaved changes in your working directory!

git filter-repo --replace-text ~/replacements.txt --force

In seconds, filter-repo will process reachable objects, rewrite blobs containing the matching text, and repack the repository.

If you run your verification command again:

git log -p -S "AKIAIOSFODNN7EXAMPLE"

It should return no matching commits in the rewritten history. For additional confidence, also scan the repository contents and objects directly (e.g., git grep "AKIAIOSFODNN7EXAMPLE" $(git rev-list --all)).

Bonus: Scrubbing Commit Messages

What if you accidentally wrote the secret inside a commit message itself?
(e.g., git commit -m "Testing AWS connection with key AKIAIOSFODNN7EXAMPLE")

To verify if your secret is hiding in a commit message, you can search all commit metadata directly using:

git log --all --grep="AKIAIOSFODNN7EXAMPLE"

The --replace-text command rewrites matching text found in Git blobs (file contents stored throughout history). It does not rewrite commit messages. To scrub your actual commit metadata, you can use the --replace-message flag with the exact same text file! It scans commit messages (and annotated tag messages) and rewrites matching text.

git filter-repo --replace-message ~/replacements.txt --force

💡 Tip:
Pro-Tip: If you need to scrub both file contents and commit messages, you can specify both options in a single invocation to rewrite history only once: git filter-repo --replace-text ~/replacements.txt --replace-message ~/replacements.txt --force

The Butterfly Effect (Hashes)

When you run these commands, you will notice that your commit IDs (hashes) have all completely changed. Why?

Git uses cryptographic hashing for integrity. A commit's hash is generated from its exact contents, metadata, and the hash of its parent.

graph TD
    classDef old fill:#fecaca,stroke:#dc2626,stroke-width:2px,color:#000
    classDef new fill:#bbf7d0,stroke:#16a34a,stroke-width:2px,color:#000
    classDef neutral fill:#e2e8f0,stroke:#64748b,stroke-width:2px,color:#000

    subgraph Before: Leaked Secret
        A[Commit 1 <br> Hash: a1b2c3d]:::neutral --> B[Commit 2 <br> Contains 'AKIA...' <br> Hash: e4f5g6h]:::old
        B --> C[Commit 3 <br> Hash: i7j8k9l]:::old
    end

    subgraph After: git filter-repo
        D[Commit 1 <br> Hash: a1b2c3d]:::neutral --> E[Commit 2 <br> Contains 'REDACTED' <br> NEW Hash: z9y8x7w]:::new
        E --> F[Commit 3 <br> NEW Hash: v6u5t4s]:::new
    end

ℹ️ Info:
Because we changed a single word in Commit 2, it became a brand-new commit with a different hash. Because Commit 2 changed, Commit 3 (its child) also had to get a new hash. This chain reaction rewrites every commit from the affected point forward to the present day.

Advanced Tips and Common Pitfalls

1. The Exact Match Trap

git filter-repo replaces exact strings. If your replacements.txt says MyPassword123, but you actually typed MyPassword1234 in one file, it will completely ignore it! Keep your replacement strings as short and specific as possible (matching is byte-for-byte and case-sensitive; target the exact secret, not the surrounding code).

2. Deep Search: `-S` vs `-G`

When verifying that your secret is gone, we used git log -S "secret". This searches for commits where the number of occurrences of the string changed. For an even deeper verification, you can use -G (e.g., git log -p -G "secret"). This runs a Regular Expression search across the actual diff lines, catching edge cases where -S might fail.

3. The "No Modified Files" Illusion

After running the scrub, you might run git status and be confused when it says working tree clean. Why aren't your files marked as "Modified"?
Because filter-repo alters your history backward in time, your physical files are instantly synced to match the newly rewritten history. Since your hard drive perfectly matches the new commit, Git sees zero unsaved changes!

4. Git Submodules Are Separate

If the leaked secret exists inside a submodule repository (like a Hugo theme), git filter-repo running in the parent repository will not clean it. You will have to run the cleanup process inside that specific submodule's repository separately.

5. Where do the old commits go? (Garbage Collection)

If you are worried that the old, dirty files containing your leaked secrets are still lingering on your hard drive, you do not need to worry about this immediately. Unreachable objects remain in your local repository until Git's garbage collection eventually prunes them after the appropriate expiration period (or sooner if you explicitly expire reflogs and run git gc).

(Advanced tip: If you want to force immediate cleanup, you can manually run git reflog expire --expire=now --all followed by git gc --prune=now. Note that --aggressive is optional and usually unnecessary for simply removing unreachable objects. **Warning:* After expiring reflogs and running garbage collection, the old objects are no longer recoverable from that local clone. Copies may still exist in other clones, forks, backups, or hosting-provider storage. Only do this after you're certain the rewritten history is correct.)*

The Final Step: Force Pushing

Because your entire history has been rewritten, GitHub will not recognize your new commits. In fact, git filter-repo may remove the origin remote after rewriting history as a safety precaution, especially when operating on a freshly cloned repository. This prevents accidental force-pushing before you verify the rewritten history.

1. Reconnect the Remote

First, reconnect your repository to GitHub:

git remote add origin https://github.com/yourusername/your-repo.git

2. Verification (Diff)

Before you do something as destructive as a force push, you can verify that your project files contain only the intended differences by comparing your local history to what is currently live on GitHub. (Remember that rewritten commit hashes do not necessarily mean your project files changed. You're verifying the contents of your working tree, not whether commit IDs match. Replace main with your repository's default branch if necessary.)

git fetch origin
git diff origin/main HEAD --name-status

💡 Tip:
Bonus Terminal Tip: If your diff is massive and you want to save it to a file to review it carefully, you might try git fetch origin && git diff origin/main HEAD > diff.log. Don't panic if you still see the git fetch progress print to your terminal! git fetch outputs to stderr (Standard Error), while > only redirects stdout (Standard Output). Your pure diff data is safely in the log file!

3. The Force Push (All Branches and Tags!)

If you only force push your main branch, the compromised commits might still be hiding on GitHub inside your dev branch or your old release tags!

To completely eradicate the leaked secret from every corner of the remote repository, you must overwrite all branches and all tags:

git push origin --force --all
git push origin --force --tags

(Review your local refs before pushing. If your repository contains unrelated tags or branches that should not be replaced, push only the affected refs instead.)
(Note: If you created signed tags, rewriting history changes the tagged commits, so existing cryptographic signatures on annotated tags are no longer valid.)
(Alternative: git push origin --mirror mirrors every local reference to the remote. This is convenient for complete repository rewrites, but be aware that it also deletes remote refs that no longer exist locally.)

4. Notify Collaborators (Crucial!)

Anyone who previously cloned the repository should re-clone it or carefully reset their local branches to the rewritten history. They should also delete or recreate any local branches that still reference the old history after migrating to the rewritten repository. Merging old branches back into the cleaned repository can accidentally reintroduce the removed commits!

5. Rotate Dependent Secrets

If applications, CI pipelines, or deployments used the exposed credential, update those systems with the new credential after rotation. Cleaning Git history does not update already deployed environments!

💡 Tip:
Working on a team? If you are worried about accidentally overwriting a colleague's recent push, you can use --force-with-lease instead of --force. This acts as a safety valve: Git will refuse to overwrite the remote branch if someone else has pushed new commits to it since you last fetched!

⚠️ Warning:
Expect a slow push! A normal git push only uploads a few kilobytes of recent changes. Because git filter-repo just assigned brand-new hashes to every commit in your history, GitHub doesn't recognize any of them. Your computer will have to compress and re-upload your entire repository history from scratch. Large repositories with extensive histories may take several minutes to rewrite and upload.

⚠️ Warning:
Check GitHub Alerts: If GitHub Secret Scanning generated an alert for the leaked credential, rewriting Git history does not automatically close the alert. After rotating the credential and cleaning the repository, review the alert in GitHub to confirm the secret has been addressed.

After the Push: GitHub's Server Cache

Git hosting providers may retain cached views, references, or backups of rewritten objects. Open pull requests and issues that reference old commit hashes can also keep the data temporarily accessible.

Remember that rewriting your repository does not remove the secret from existing forks or clones. If the credential was sensitive, treat it as compromised even after cleaning the history.

For sensitive leaks, contact GitHub Support to request a garbage collection on their servers and purge any cached views of the old commits.

Some Git hosting providers retain hidden references (such as pull request refs or backup refs) that are not updated by normal pushes. If a secret is especially sensitive, contact your hosting provider to ensure any server-side references are cleaned up.

Prevention: Stop It From Happening Again

The best cleanup is the one you never have to do:

.gitignore: Block .env, credentials files, and build artifacts before they are ever staged.
gitleaks: A pre-commit hook that scans your staged changes for hardcoded secrets, API keys, and tokens before they ever enter a commit.
detect-secrets: An alternative secret scanner by Yelp that maintains a baseline of known secrets in your repo.
GitHub Push Protection: If your organization uses GitHub Secret Scanning with Push Protection, GitHub may automatically block pushes containing supported secret types before they reach the remote.

For a deeper dive on .gitignore, pre-commit hooks, and Git LFS for large files, see my companion guide: Removing Objects From Git History.

Your leaked secrets have now been removed from Git history, and your repository is safe once again. Just remember: Always revoke the credential first, clean up the history second.

Found this helpful? Follow me for more real-world Linux troubleshooting and DevOps content.

Originally published on khadirullah.com

The Ultimate Guide to Automating SVG Rasterization: Inkscape vs ImageMagick vs Headless Chrome

Khadirullah Mohammad — Mon, 31 Aug 2026 11:48:03 +0000

If you are automating SVG-to-PNG conversions on a Linux server and notice your colors look washed out or your drop shadows are missing, this post is for you.

Engine	Browser CSS	SVG Filters	Emoji	Speed	Best Use
Headless Chrome	✅	✅	✅	Medium	Browser-accurate rendering
Inkscape	⚠️ SVG 1.1	✅ (SVG 1.1)	❌	Fast	Standards-compliant SVG
ImageMagick (`librsvg`)	⚠️ SVG 1.1	✅ (SVG 1.1)	❌	Very Fast	Batch rasterization

When building out the DesignOps pipeline for this blog (as detailed in my guide on Fixing Broken Hugo Open Graph Previews), my goal was simple: use a Python script to automatically read my vector featured.svg images, inject my branding, and convert them to social-fallback.webp thumbnails for LinkedIn and Twitter.

The pipeline worked flawlessly, until I started using modern SVG features like feGaussianBlur, rgba alpha blending, and complex neon gradients. Suddenly, the automated images didn't look right. The colors were muddy, and some CSS filters disappeared entirely.

This led me down a rabbit hole of testing three different command-line rendering engines. Here is what I found.

Phase 1: The Browser-Oriented SVG Test

(Want to follow along? Download the old.svg and new.svg files used in this test to compare the source code yourself.)

To test the engines, I started with a browser-oriented SVG that used modern CSS concepts (like rgba()) and newer shorthand filters (like feDropShadow). It featured dark transparent cards layered over a neon linear gradient, with a heavy feGaussianBlur applied to a background circle.

Test environment: Debian 13 (Trixie) · Inkscape 1.4 · ImageMagick 7.1.1-43 (librsvg 2.60) · Google Chrome 148. Cross-validated on Fedora 44 (Inkscape 1.4.4, ImageMagick 7.1.2-13, librsvg 2.62, Chrome 149) with identical results.

Results may differ slightly with newer rendering engines, but the underlying compatibility differences described here remain relevant.

Here is how the three major Linux rasterization tools handled this browser-oriented SVG. All screenshots below were generated from the same source SVG without any manual edits.

1. ImageMagick (librsvg)

On many Linux distributions, ImageMagick relies on librsvg to render SVG files. The exact renderer depends on how ImageMagick was built, but librsvg is the most common delegate on modern systems.

The Verdict: librsvg failed to render several of the browser-oriented filters, stripping away the glows and drop shadows.

2. Inkscape (Command Line)

Inkscape is a phenomenal vector graphics editor, but let's see how its command-line renderer handled the CSS values.

The Verdict: Inkscape ignored the rgba() attribute values, causing those fills to disappear, and failed to process the feDropShadow filter.

3. Headless Google Chrome

SVGs are fundamentally a web technology. By using Chrome's built-in headless screenshot feature, we can force a literal web browser to render the SVG exactly as intended. Unlike standalone SVG renderers, Chrome uses the same rendering engine that powers modern web pages (Blink/Skia), so CSS, filters, fonts, gradients, and SVG features behave exactly as they do in the browser.

The Verdict: The colors are pixel-perfect, and it handled the web CSS flawlessly.

Gotcha: Chrome Bottom Cutoff

There is one critical trap with Headless Chrome that cost me hours of debugging. If you wrap your SVG in a simple HTML page like this:

<style>body{margin:0;overflow:hidden}</style>
<body><!-- SVG here --></body>

Chrome treats the SVG as a responsive element. It scales the SVG to fill the viewport width, then calculates the height from the aspect ratio. If the calculated height exceeds --window-size, the bottom of the image gets silently clipped.

You end up with a screenshot that looks correct at first glance but is missing content at the bottom edge.

The fix is a single CSS rule that locks the SVG to exact pixel dimensions:

<style>
  html, body { margin:0; padding:0; overflow:hidden; }
  svg { display:block; width:1200px; height:630px; }
</style>

This forces Chrome to render the SVG at exactly 1200×630 pixels, matching the --window-size boundary perfectly. Here is the same SVG rendered again with the CSS fix applied:

No more clipping. The full image is captured edge-to-edge.

🐛 Bug:
Chrome Version matters! During testing, older Chrome versions (specifically v147.x) had a bug where SVG filters (like feDropShadow) confused the headless screenshot engine's bounding box calculations, causing the bottom of the image to clip despite the CSS fix. Upgrading to Chrome v148 or newer (tested on 148.0.7778.215 and 150.0.7871.181) resolves this layout bug entirely. If you still see clipping, check your browser version!

Why Inkscape and ImageMagick Break (Code Analysis)

After seeing Inkscape and ImageMagick fail so spectacularly on the first test, I traced the rendering failures back to the source code. The issue was not the engines; it was my browser-oriented authoring style. Here is the exact code that broke them, and the strict SVG 1.1 equivalent that fixed it.

Root Cause 1: `rgba()` CSS Colors

This is the single most common cause of invisible elements in Inkscape/ImageMagick exports.

<!-- ❌ This works in browsers but breaks in Inkscape/ImageMagick -->
<rect fill="rgba(255,255,255,0.03)" />
<stop stop-color="rgba(39, 39, 42, 0.8)" />

<!-- ✅ SVG-native equivalent that works everywhere -->
<rect fill="#ffffff" fill-opacity="0.03" />
<stop stop-color="#27272a" stop-opacity="0.8" />

rgba() is a CSS color function, not a valid SVG attribute value. Browsers support it because they blend CSS and SVG rendering. Inkscape and librsvg follow the SVG spec more strictly and either ignore these values or render them as fully transparent, making entire cards, rows, and subtle fills disappear.

Root Cause 2: `feDropShadow`

<!-- ❌ Shorthand filter. Inkscape 1.4 warns: "unknown type: svg:feDropShadow" -->
<filter id="shadow">
  <feDropShadow dx="0" dy="10" stdDeviation="15" flood-color="#000" flood-opacity="0.6"/>
</filter>

<!-- ✅ SVG 1.1 compatible equivalent -->
<filter id="shadow" x="-10%" y="-10%" width="130%" height="140%">
  <feFlood flood-color="#000000" flood-opacity="0.6" result="flood" />
  <feComposite in="flood" in2="SourceAlpha" operator="in" result="shadow-shape" />
  <feGaussianBlur in="shadow-shape" stdDeviation="15" result="shadow-blur" />
  <feOffset dx="0" dy="10" result="shadow-offset" />
  <feMerge>
    <feMergeNode in="shadow-offset" />
    <feMergeNode in="SourceGraphic" />
  </feMerge>
</filter>

feDropShadow is a newer shorthand filter supported by modern browsers but not consistently implemented by non-browser renderers. When Inkscape encounters an unknown filter type, it silently drops the entire filtered element. If your shadow filter is applied to a card group, the whole card vanishes.

Root Cause 3: Emoji and `system-ui` Fonts

<!-- Emoji renders as colored icons in Chrome, empty boxes in Inkscape/Magick -->
<text font-family="monospace" font-size="20">🎨</text>
<text font-family="monospace" font-size="20">✅</text>

<!-- system-ui is a browser-only alias. Inkscape does not recognize it -->
<text font-family="system-ui, -apple-system, sans-serif">Title</text>

Inkscape and ImageMagick have no access to color emoji fonts and don't understand browser-only font aliases like system-ui. Text positioning may shift due to different font metrics.

Phase 2: The Strict SVG 1.1 Rematch

Once I refactored the code to use strict SVG 1.1 attributes (no rgba(), verbose filter chains, and vector paths instead of emoji fonts), I ran the exact same file through Inkscape and ImageMagick again.

Here is the result.

ImageMagick (Strict SVG 1.1)

The Verdict: Flawless. The filters and gradients render perfectly.

Inkscape (Strict SVG 1.1)

The Verdict: Absolutely perfect. It is visually indistinguishable from the Chrome render.

The Ultimate Automation Script

You don't need a heavy Node.js/Puppeteer installation to use Headless Chrome. If you have Google Chrome installed on your Linux server, you can execute it directly from a Python script using subprocess.

Here is the exact Python snippet I use in my DesignOps pipeline to generate pixel-perfect WebP fallbacks from SVGs:

import os
import subprocess

svg_path = "featured.svg"
temp_png = "temp-screenshot.png"
final_webp = "social-fallback.webp"

# 1. Read the SVG content
with open(svg_path, "r") as f:
    svg_content = f.read()

# 2. Wrap it in an HTML page with CRITICAL CSS sizing.
#    Without `svg { width:1200px; height:630px; }`, Chrome treats the SVG
#    as responsive and the bottom gets clipped by the viewport!
temp_html = "temp-social.html"
html_wrapper = f"""<!DOCTYPE html>
<html>
<head>
<style>
  html, body {{ margin:0; padding:0; overflow:hidden; background:transparent; }}
  svg {{ display:block; width:1200px; height:630px; }}
</style>
</head>
<body>
{svg_content}
</body>
</html>"""
with open(temp_html, "w") as f:
    f.write(html_wrapper)

# 3. Use Headless Chrome to take a pixel-perfect screenshot.
#    The absolute file:// URI is required for Chrome to load local files.
abs_html_uri = f"file://{os.path.abspath(temp_html)}"

subprocess.run([
    "google-chrome", 
    "--headless", 
    f"--screenshot={temp_png}", 
    "--window-size=1200,630",
    "--default-background-color=00000000",
    "--disable-gpu",
    "--no-sandbox",
    abs_html_uri
], check=True)

# 4. Strip metadata and convert to lossy WebP (q=90 for optimal size)
subprocess.run(
    ["exiftool", "-all=", "-overwrite_original", temp_png],
    check=True
)
subprocess.run([
    "cwebp", 
    "-q", "90", 
    temp_png, 
    "-o", final_webp
], check=True)

# 5. Cleanup
os.remove(temp_png)
os.remove(temp_html)

Key detail: The CSS rule svg { display:block; width:1200px; height:630px; } is critical. Without it, Chrome treats the SVG as a responsive element that scales to the viewport width and calculates height from the aspect ratio, which can cause the bottom of your image to be cut off by the --window-size boundary.

Performance note: This script starts a new Chrome process for every image. For a handful of Open Graph images that is perfectly fine. For large batches, keeping Chrome alive through the Chrome DevTools Protocol or using Puppeteer/Playwright can reduce startup overhead.

Conclusion

So, which engine should you use for your DesignOps pipeline?

Option A: Headless Chrome
If you want to design SVGs like web pages (using CSS, rgba(), and emojis), you must use Headless Chrome. It gives you maximum flexibility and guarantees that what you see in the browser is what you get in the exported image.

Option B: Inkscape or ImageMagick
If you want the speed and low overhead of Inkscape or ImageMagick, you must author strict, disciplined SVG 1.1 code. No CSS colors, no shorthand filters.

The takeaway: Each renderer targets a different subset of the SVG ecosystem. Pick the engine that matches your authoring style.

Found this helpful? Follow me for more real-world Linux troubleshooting and DevOps content.

Originally published on khadirullah.com

Fixing Broken Hugo Open Graph Previews with WebP Social Fallbacks

Khadirullah Mohammad — Sat, 29 Aug 2026 07:59:44 +0000

Vector graphics (SVGs) are the holy grail of modern web design. They render with pixel-perfect clarity on any screen size, maintain incredibly small file sizes, and can be styled with CSS. However, if you rely on SVGs for your website's featured images, you'll quickly run into a frustrating problem: they completely break Open Graph (OG) social media previews.

When you share a link on LinkedIn, Twitter, or WhatsApp, their scrapers look for a rasterized image to display in the preview card. Unfortunately, none of these platforms support SVG files for Open Graph images. The result? A broken or missing preview card that severely hurts your click-through rate.

Here is the exact engineering solution I implemented to fix this on my Hugo website (using the Blowfish theme).

1. Why WebP for Social Fallbacks?

The obvious solution to the SVG problem is to provide a rasterized fallback image (like a PNG or JPEG). However, standard lossy image compression often introduces ugly color banding and blurriness around sharp text, ruining the premium feel of the vector original.

WebP is a modern image format that provides superior compression with two modes:

Lossless WebP (cwebp -lossless): Preserves exact pixel clarity, zero quality loss. For this blog post, a metadata-stripped PNG went from 135 KB to 87 KB as a lossless WebP, a 35% reduction.
Lossy WebP (cwebp -q 90): Near-identical visual quality with dramatically smaller files. The same image compressed to just ~30 KB, a 78% reduction from PNG, while remaining visually indistinguishable from the original.

Both lossless and lossy WebP are fully supported across all major social media scrapers (Twitter, LinkedIn, Facebook, WhatsApp). I recommend lossy WebP at quality 90 for social fallbacks simply because the file size savings are massive and there is no visible quality difference for OG preview cards.

Format Comparison for Open Graph Fallbacks

Feature	PNG	Lossless WebP (`-lossless`)	Lossy WebP (`-q 90`)
Social Platform Support	✅ Universal	✅ Fully Supported	✅ Fully Supported
File Size	~135 KB	~87 KB (~35% smaller)	~30 KB (~78% smaller)
Quality	Lossless	Lossless	Crisp (Indistinguishable)
Best For	Archival assets	Internal site images	Social media preview cards

When to Use Which

Lossy WebP (-q 90) is the smart choice for Open Graph social fallbacks. Social platforms downscale, re-compress, and cache your image anyway. Pixel-perfect fidelity is wasted on a preview card that renders at thumbnail size in a feed. The ~78% file size savings over PNG means faster scraper fetches and snappier link previews.

Lossless WebP (-lossless) is better for images displayed inside your blog post (screenshots, diagrams, code snippets) where readers zoom in and sharp text matters. The extra kilobytes are worth it when the image is the content itself, not just a social thumbnail.

To generate a crisp WebP social fallback, strip EXIF metadata using exiftool and convert with cwebp:

# Step 1: Strip all EXIF metadata from the rasterized PNG
exiftool -all= -overwrite_original input.png

# Step 2: Convert to WebP (lossy q=90 for optimal size)
cwebp -q 90 input.png -o social-fallback.webp

Important: Most social media platforms require OG images to be at least 1200×630 pixels. Make sure your rasterized source image is rendered at this resolution before converting to WebP.

Note: If you want to automate this conversion pipeline, check out my deep-dive Ultimate Guide to Automating SVG Rasterization to see a rendering compatibility comparison of the best command-line tools for the job.

2. The Hugo vs. Blowfish Conflict

Now that we have our crisp fallback image, we need to tell Hugo to use it for Open Graph tags, while still using featured.svg for the actual website layout.

This is where things get tricky, especially if you are using a feature-rich theme like Blowfish.

Blowfish automatically searches for images in your page bundle using greedy wildcards (like *feature* or *cover*). If you name your fallback image featured.webp alongside your featured.svg, the theme's logic might accidentally prioritize the WebP image and render it on your webpage instead of the SVG!

3. The "Social Fallback" Solution

To solve this conflict, we need a two-step approach that "blinds" the theme from accidentally using the fallback image on the frontend, while explicitly feeding it to the SEO scrapers.

Step 1: Rename the fallback image.

Instead of naming it featured.webp, name it something the theme's wildcard search won't catch, such as social-fallback.webp.

Step 2: Explicitly define the image in Front Matter.

In your Hugo blog post's Markdown file, use the images array in the YAML front matter to explicitly point the Open Graph templates to this specific file.

title: "Your Awesome Blog Post"
date: 2026-07-19
images: ["social-fallback.webp"]

The Result

With this setup:

Your website's frontend continues to perfectly render the crisp, lightweight featured.svg.
Hugo's built-in SEO templates generate <meta property="og:image" content=".../social-fallback.webp"> and <meta name="twitter:image" content=".../social-fallback.webp"> in the <head> of your HTML.
When shared on LinkedIn, Twitter, Facebook, or WhatsApp, scrapers pull the lightweight 30 KB WebP image.

You get the best of both worlds: vector rendering on your site and pristine, fast-loading social media preview cards.

4. Verification Across Platforms

After deploying changes, verify that the og:image and twitter:image tags are being picked up correctly. Here is how each platform rendered the WebP preview card for this blog:

All four platforms rendered the WebP fallback perfectly.

Tip: If any platform shows a stale or missing image, try appending a dummy query parameter (e.g. ?v=2) to your URL to force a fresh crawl. Social platform scrapers cache aggressively, and stale cache is the most common reason for previews appearing broken.

Summary Checklist

Use WebP for Open Graph social fallbacks. Lossy -q 90 gives ~78% file size savings over PNG with no visible quality loss.
Name the file social-fallback.webp to avoid Blowfish theme resource matching collisions with featured.svg.
Add images: ["social-fallback.webp"] to your YAML front matter to feed Hugo SEO templates.
Always cache-bust when testing. Append ?v=2 to your URL to force scrapers to re-fetch.

Found this helpful? Follow me for more real-world Linux troubleshooting and DevOps content.

Originally published on khadirullah.com

Git Submodule Silently Became Regular Files: How I Diagnosed and Fixed It

Khadirullah Mohammad — Fri, 28 Aug 2026 08:43:19 +0000

I needed to update my Hugo theme (Blowfish) from v2.97.0 to the latest version. The theme was supposed to be a git submodule. My .gitmodules file said so. So I ran the standard update command:

git submodule update --remote --merge

Nothing happened. No output, no errors, no update.

I tried the manual approach:

cd themes/blowfish
git pull origin main

It pulled from my site repository, not the Blowfish theme repository. Something was fundamentally broken.

This post covers how I diagnosed the problem, fixed the submodule, and then dealt with the Cloudflare Pages deployment failure that came immediately after.

Note:
Backstory: I originally set up the Blowfish theme using git submodule add, and my .gitmodules was correctly configured from the start. But at some point while troubleshooting config issues, I deleted the themes/ folder and copy-pasted the theme files back manually. I didn't realize that this single action silently destroyed the submodule. Git replaced the submodule's internal 160000 commit gitlink with a regular 040000 tree entry, turning it into ordinary tracked files. The .gitmodules file stayed behind untouched, so everything looked correct. It wasn't until I tried to update the theme months later that I discovered the mismatch.

Hugo themes support multiple installation methods: git submodule, manual copy, and Hugo modules. Manually copying files into themes/ is a perfectly valid installation method. The problem arises specifically when your repo has a .gitmodules entry claiming a submodule exists, but Git's index tracks the files as regular committed files. That state conflict is what causes git submodule update to silently do nothing.

Note:
Related: After fixing this submodule issue, I discovered that the old theme files were still bloating my git history. I wrote a separate guide on how I cleaned them up: Removing Objects From Git History with git filter-repo.

Quick Answer (TL;DR)

If git submodule update does nothing and git submodule status returns empty output, your theme was committed as regular files, not as a proper submodule.

Fix it:

# 1. Remove the fake submodule files
git rm -rf themes/blowfish

# 2. Clean cached submodule data
rm -rf .git/modules/themes/blowfish

# 3. Remove the stale .gitmodules entry
git config --file .gitmodules --remove-section submodule.themes/blowfish

# 4. Commit the removal
git add .gitmodules
git commit -m "fix: remove improperly tracked theme files"

# 5. Re-add as a proper submodule
git submodule add https://github.com/nunocoracao/blowfish.git themes/blowfish
git commit -m "feat: add blowfish theme as proper git submodule"

Then update HUGO_VERSION in your deployment platform to match the version the new theme requires.

The Symptoms

Three clues told me this wasn't a real submodule:

1. No `.git` file inside the theme directory

A properly initialized submodule has a .git file (not a directory) that points to the parent repo's .git/modules/ folder:

ls -la themes/blowfish/.git

# Expected (proper submodule):
-rw-r--r-- 1 user user 43 Jul 15 00:55 themes/blowfish/.git

# What I got:
ls: cannot access 'themes/blowfish/.git': No such file or directory

2. `git submodule status` returned nothing

git submodule status

# Expected:
 1f14448333d43207780f1a97317562a3b9ddbbd0 themes/blowfish (v2.104.0)

# What I got:
(empty, no output at all)

3. Git commands inside the theme directory operated on the parent repo

cd themes/blowfish
git remote -v

# Expected:
origin  https://github.com/nunocoracao/blowfish.git (fetch)

# What I got:
origin  https://github.com/khadirullah/khadirullah.com (fetch)

The themes/blowfish/ directory had no git identity of its own. Git treated it as part of the parent repository.

The Root Cause

Hugo themes can be installed in three ways: as a git submodule, as manually copied files, or as a Hugo module. All three are valid. The Blowfish theme documents all three methods.

In my case, I originally installed Blowfish as a proper submodule using git submodule add. But later, while troubleshooting config issues, I deleted the themes/ folder and copy-pasted the theme files back manually. What I didn't realize is that this single action silently destroyed the submodule.

Why deleting and re-adding files breaks a submodule

Internally, Git tracks a proper submodule as a special entry called a gitlink, a 160000 commit object that points to a specific commit in the external repository:

# What a proper submodule looks like in Git's index:
160000 commit abc123...    themes/blowfish
^^^^^^
# This is a gitlink. Git knows this is an external repo

When you delete a submodule directory and copy-paste the files back manually, Git replaces the gitlink with a regular tree, a 040000 tree object that treats the files as part of your main repository:

# What happened after I deleted and copy-pasted:
040000 tree 6aaaa7...    themes/blowfish
^^^^^^
# This is a regular directory. Git treats these as your own files

These mode numbers are part of Git's internal file type system (derived from Unix file permissions). Here's a quick reference:

Mode	Type	What it means
`100644`	`blob`	Regular file (not executable)
`100755`	`blob`	Executable file
`040000`	`tree`	Regular directory, files are stored in your repo
`160000`	`commit`	Gitlink (submodule), only a commit hash is stored, files live in the external repo
`120000`	`blob`	Symbolic link

You can check what mode Git is using for any path with git ls-tree HEAD <path>. If your submodule shows 040000 tree instead of 160000 commit, it's broken.

The .gitmodules file isn't affected by this. It stays behind, still claiming the submodule exists. But Git's index now disagrees, creating a silent conflict.

The result was a contradictory state:

.gitmodules said "there's a submodule at themes/blowfish"
Git's index said "these are regular tracked files"
No submodule metadata existed in .git/modules/

This kind of mismatch can happen to anyone. All it takes is deleting and re-adding theme files without using git submodule add. The .gitmodules file alone doesn't make something a submodule. Git needs the full submodule registration.

Note:
⚠️ Actions that silently destroy a submodule:

Deleting the submodule directory and copy-pasting files back

Running rm -rf themes/your-theme followed by manually adding files

Using your file manager to drag-and-drop theme files into the directory

Any operation that replaces the directory without going through git submodule add

The safe way to reinstall a submodule is to always use git submodule deinit + git rm to remove it, and git submodule add to re-add it.

flowchart TD
    A["git submodule add URL themes/blowfish"] --> B["✅ Proper submodule\n(160000 commit gitlink)"]
    C["Copy files into themes/blowfish/\n(no .gitmodules entry)"] --> D["✅ Valid manual install\n(040000 tree)"]
    E["Delete submodule dir + copy files back\n(stale .gitmodules remains)"] --> F["❌ Broken state\n(.gitmodules ≠ Git index)"]

    B --> G["git submodule update works"]
    D --> H["Update manually by\nre-downloading the theme"]
    F --> I["git submodule update\nsilently does nothing"]

    style A fill:#1e3a5f,stroke:#60a5fa,color:#e0f2fe
    style B fill:#14532d,stroke:#4ade80,color:#dcfce7
    style C fill:#1e3a5f,stroke:#60a5fa,color:#e0f2fe
    style D fill:#14532d,stroke:#4ade80,color:#dcfce7
    style E fill:#7c2d12,stroke:#f97316,color:#fed7aa
    style F fill:#7f1d1d,stroke:#ef4444,color:#fecaca
    style G fill:#14532d,stroke:#4ade80,color:#dcfce7
    style H fill:#14532d,stroke:#4ade80,color:#dcfce7
    style I fill:#7f1d1d,stroke:#ef4444,color:#fecaca

The Fix

The solution is to remove the tracked files, clean up submodule metadata, and re-add the theme as a proper submodule.

Step 1: Remove the theme files from git tracking

git rm -rf themes/blowfish

This removes all the theme files from git's index and from disk. The -rf flag handles the recursive directory removal.

Step 2: Clean cached submodule data

rm -rf .git/modules/themes/blowfish

This removes any partial submodule metadata that might exist from a previous failed initialization attempt. If the directory doesn't exist, the command silently succeeds.

Step 3: Remove the old `.gitmodules` entry

git config --file .gitmodules --remove-section submodule.themes/blowfish

This cleans up the stale submodule configuration. If you had other submodules, their entries remain untouched.

Step 4: Stage and commit the removal

git add .gitmodules
git commit -m "fix: remove improperly tracked theme files"

At this point, your repository has no theme, which is correct. The next step brings it back properly.

Step 5: Re-add as a proper submodule

git submodule add https://github.com/nunocoracao/blowfish.git themes/blowfish

This does three things:

Clones the latest version of the Blowfish repository into themes/blowfish/
Creates a .git file inside themes/blowfish/ pointing to .git/modules/themes/blowfish/
Updates .gitmodules with the correct submodule entry

Step 6: Commit the proper submodule

git commit -m "feat: add blowfish theme as proper git submodule"

Step 7: Verify

# Check the submodule is recognized
git submodule status
# Output: 1f14448333d43207780f1a97317562a3b9ddbbd0 themes/blowfish (v2.104.0)

# Check the .git file exists
ls -la themes/blowfish/.git
# Output: -rw-r--r-- 1 user user 43 Jul 15 00:55 themes/blowfish/.git

# Check the theme version
cat themes/blowfish/package.json | grep version
# Output: "version": "2.104.0",

# Build the site
hugo serve

Future Updates Now Just Work

With the submodule properly set up, updating the theme is a single command:

git submodule update --remote --merge

To pin the submodule to track the main branch (recommended for themes that follow semantic versioning):

git config -f .gitmodules submodule.themes/blowfish.branch main
git add .gitmodules
git commit -m "fix: pin blowfish submodule to main branch"

After updating, always test your site before pushing:

hugo serve

Theme updates can introduce breaking changes: new shortcodes, renamed partials, or changed CSS classes. Review the theme's changelog before updating to a major version.

Quick Diagnostic Checklist

If your Hugo theme submodule isn't behaving correctly, run through these checks:

Check	Command	Expected (working)	Broken
`.git` file exists	`ls themes/your-theme/.git`	File exists (43 bytes)	`No such file`
Submodule status	`git submodule status`	Shows commit hash + path	Empty output
Remote URL	`cd themes/your-theme && git remote -v`	Theme repo URL	Your site repo URL
Theme version	`cat themes/your-theme/package.json \	grep version`	Current version

If any of these fail, your theme is not a real submodule. Follow the fix above.

Why This Matters

Note: If you intentionally installed your Hugo theme by copying files (without submodules), this section doesn't apply to you. Manual installs work fine. You just update by re-downloading the theme. This guide is specifically for repos where .gitmodules claims a submodule exists but Git doesn't actually track one.

A broken submodule is invisible until you need it. Your site builds fine, your deployments work, everything looks normal. The problem only surfaces when you try to update, and by then, you've been shipping an outdated theme for months.

For Hugo sites specifically:

Security patches in the theme won't reach you
New shortcodes (like accordion in Blowfish v2.100+) won't be available
Hugo version compatibility fixes won't apply, and your build warnings pile up
CI/CD pipelines that run git submodule update --init --recursive will either fail silently or clone an empty directory

The "Gotcha": Cloudflare Pages Deployment Failure After Fixing

I fixed the submodule locally, everything worked in hugo serve, so I committed and pushed the fix.

And then my Cloudflare Pages deployment immediately crashed.

What happened

Locally, everything built perfectly because I was running a modern version of Hugo:

$ hugo version
hugo v0.164.0-ce2470e7012b5ab5fc4e10ebe4027e9f8d9e00dc+extended linux/amd64 BuildDate=2026-07-06T16:39:30Z VendorInfo=gohugoio

But on Cloudflare Pages, the build failed with this error:

2026-07-15T10:22:25.178761Z Detected the following tools from environment: hugo@extended_0.152.2
2026-07-15T10:22:27.074274Z WARN  Module "blowfish" is not compatible with this Hugo version
2026-07-15T10:22:27.211531Z Error: error building site: render: failed to render pages:
  execute of template failed: template: index.html:3:47:
  executing "index.html" at <site>: can't evaluate field Locale in type *langs.Language

When I updated the Blowfish theme submodule to the latest version, it pulled in code that required Hugo 0.158.0 or higher. However, in my Cloudflare Pages Settings > Environment variables, I had hardcoded HUGO_VERSION to 0.152.2 months ago.

Because Cloudflare was forced to use an older Hugo version, it couldn't understand the new Locale fields in the updated theme templates, causing the entire build to fail.

Diagnosing the failure

The first clue was in the Cloudflare Pages deployment dashboard. The build settings showed HUGO_VERSION was stuck at the old value:

GitHub's integration also reported the failure immediately:

Step-by-step fix on Cloudflare Pages

Here is exactly how to fix the Hugo version mismatch on Cloudflare Pages:

1. Go to Settings > Variables and Secrets

In the Cloudflare Pages dashboard, navigate to your project's Settings tab, then find Variables and Secrets in the sidebar. You'll see the old HUGO_VERSION value:

2. Edit the HUGO_VERSION variable

Click Edit, then update the value to match your local Hugo version. In my case, I changed it from 0.152.2 to 0.164.0:

3. Verify the saved value

After saving, confirm the variable shows the updated version:

Note:
Important: If you look at the build settings right now on a previous deployment, they may still show the old version. The new value only takes effect on the next deployment.

4. Navigate to the failed deployment

Go to the Deployments page. You'll see the failed deployment listed:

5. Retry the deployment

Click View Details on the failed deployment, then click Manage deployment and select Retry deployment:

6. Watch it build

The retried deployment will pick up the new Hugo version. You'll see the status change to "active":

7. Confirm success

After the build completes, the deployment status should show "success":

The build log confirms everything worked. All four stages (initialize, clone repo, build, deploy) passed:

Notice line 6 in the build log: "Submodule 'themes/blowfish' (https://github.com/nunocoracao/blowfish.git) registered for path 'themes/blowfish'". This confirms Cloudflare is now properly cloning the submodule during the build process. Before the fix, the theme files were just regular files in the repo, so no submodule cloning happened.

8. Verify on GitHub

GitHub's integration also updates to show the successful deployment:

9. Final state

The deployments page now shows both the failed (old) and successful (retried) deployments:

Other deployment platforms

If you're not using Cloudflare Pages, update the Hugo version in your platform's configuration:

Platform	Where to update
Cloudflare Pages	Settings → Variables and Secrets → `HUGO_VERSION`
GitHub Actions	`.github/workflows/*.yml` → `hugo-version` in the hugo-setup action
Netlify	`netlify.toml` → `HUGO_VERSION` environment variable
Vercel	`vercel.json` or Project Settings → Environment Variables

Hugo themes declare minimum version requirements in their config.toml or module.toml. When a theme updates to use new Hugo features (like the Locale field added in Hugo 0.158.0), older Hugo versions can't parse the templates.

This is especially common with actively maintained themes like Blowfish that take advantage of new Hugo APIs soon after they're released.

The timeline in my case:

Months ago: Set HUGO_VERSION=0.152.2 in Cloudflare Pages
July 2026: Fixed the submodule, which pulled Blowfish v2.104.0
Blowfish v2.104.0 requires Hugo ≥ 0.158.0
Cloudflare tried to build with Hugo 0.152.2 → template error

The fix: always check the theme's minimum Hugo version after updating, and keep your deployment platform's Hugo version in sync with your local version.

Pin your Hugo version in your repo, not just in your deployment platform:

# In your site's config/_default/config.toml or hugo.toml
[module]
  [module.hugoVersion]
    extended = true
    min = "0.158.0"

This makes Hugo print a clear warning if someone tries to build with an incompatible version, instead of failing with a cryptic template error.

You can also add a version check to your CI pipeline:

REQUIRED_VERSION="0.158.0"
CURRENT_VERSION=$(hugo version | grep -oP '\d+\.\d+\.\d+' | head -1)
if [ "$(printf '%s\n' "$REQUIRED_VERSION" "$CURRENT_VERSION" | sort -V | head -n1)" != "$REQUIRED_VERSION" ]; then
  echo "ERROR: Hugo $CURRENT_VERSION is below minimum $REQUIRED_VERSION"
  exit 1
fi

Cleaning Up the Old Theme Files From Git History

After fixing the submodule, I noticed that the old theme files (hundreds of files from the improperly committed copy) were still taking up space in my git history. Even though they were removed from the current working tree, the old commits still referenced them.

If you find yourself in the same situation and want to clean up the bloat, I've written a detailed guide on how to do it: Removing Objects From Git History with git filter-repo.

Summary

Problem	Solution
`git submodule update` does nothing	Theme was committed as regular files. Remove and re-add properly
`.gitmodules` exists but `git submodule status` is empty	The submodule was never properly registered with `git submodule add`
Deployment fails after fixing the submodule	Update `HUGO_VERSION` in your deployment platform to match the new theme's requirements
Old theme files still in git history	Use `git filter-repo` to purge them (guide)

Fix it once, and every future update is one command.

Found this helpful? Follow me for more real-world Linux troubleshooting and DevOps content.

Originally published on khadirullah.com

I Deleted Files From Git But They Were Still There: Removing Objects From Git History

Khadirullah Mohammad — Thu, 27 Aug 2026 05:25:35 +0000

This started as a simple cleanup task, but it revealed an important Git concept: deleting a file from your working tree does not remove it from repository history.

I was auditing my Hugo blog's deployment when I noticed something: 24 unused PNG screenshots were being deployed to Cloudflare Pages on every build. They were sitting inside a Hugo page bundle. Hugo page bundles can contain resource files that participate in the build pipeline. Depending on the resource type and how the files are referenced, unused assets may still be processed or copied into generated output. This can accidentally publish assets that are never referenced.

Easy fix, right? Delete the folder, commit, push. Done.

Except it's not done. Those 3MB of PNGs still live in git history. A normal clone can download the old objects as long as they remain reachable through repository references such as branches or tags. They are invisible in the current working tree, but they remain part of reachable history until the references are rewritten. They are still stored in git's object database and reachable through old history references until those references are removed and garbage collection runs.

This post covers how I found them, removed them from history, and verified they're truly gone.

Note:
When you'd actually need this in production: Someone commits an API key, a 500MB dataset, or build artifacts to the repo. Deleting the file in a new commit doesn't remove it from history. Anyone with repo access can still find it. This is the procedure to purge it completely.

Quick Answer (TL;DR)

If a secret was exposed:

Rotate/revoke it immediately
Rewrite history
Force push rewritten refs
Ask collaborators to refresh clones

For general history cleanup, deleting a file with git rm only removes it from future commits. To remove it from all Git history, run:

git filter-repo \
  --path path/to/file \
  --invert-paths \
  --force

Then force-push the rewritten history:

git push origin main --force-with-lease

--force-with-lease is safer because it prevents overwriting remote changes you do not have locally.

Finally, verify the removal and run local garbage collection. Read on for the full investigation and safety walkthrough.

Who should read this?

Choose the section based on your situation:

🟢 Accidentally committed a large file?
→ Jump to Cleanup Process

🔴 Accidentally committed a password/API key?
→ Rotate the secret first, then follow history rewrite

🔵 Want to understand Git internals?
→ Continue through the investigation section

🟣 Maintaining a production repository?
→ Read backup, tags, verification, and aftercare sections

Which tool do I need?

Situation	Tool
Remove one recent commit	`git rebase -i`
Remove file everywhere	`git filter-repo`
Replace leaked secret	`git filter-repo --blob-callback`
Huge files	`git filter-repo` or Git LFS

Why `git rm` Doesn't Fix This

When you run git rm and commit, git records: "this file no longer exists from this commit forward." But the original blob (the file's content) remains stored in Git's object database because the old commit history still references the blob. Once those references are removed, the object becomes unreachable and can eventually be removed by garbage collection.

Commit abc123 ── "Added 24 PNGs"
    └── git stored each PNG as a "blob" object in .git/objects/

Commit def456 ── "Deleted 24 PNGs"  
    └── git recorded "these files are gone from THIS version"
         BUT the blobs from abc123 still exist

Think of it like a filing cabinet where old copies remain until Git confirms nothing references them anymore and garbage collection removes them.

Which history rewrite method do you need?

Before running commands, decide what you actually want to remove.

Use git filter-repo:

git filter-repo --path path/to/file --invert-paths --force

Example:

leaked .env
accidentally committed videos
generated build files

The file itself is useful, but some content inside it must disappear.

Examples:

remove a password from config.yml
remove an API key from a JSON file
remove a private URL

For recent commits, use interactive rebase.
For many commits, use git filter-repo --blob-callback.

The complete workflow

flowchart LR
    A["🔍 Investigate"] --> B["💾 Backup"]
    B --> C["📂 Fresh Clone"]
    C --> D["🧹 filter-repo"]
    D --> E["✅ Verify"]
    E --> F["🚀 Force Push"]
    F --> G["🏷️ Update Tags"]

    style A fill:#1e3a5f,stroke:#60a5fa,color:#e0f2fe
    style B fill:#1e3a5f,stroke:#60a5fa,color:#e0f2fe
    style C fill:#1e3a5f,stroke:#60a5fa,color:#e0f2fe
    style D fill:#7c2d12,stroke:#f97316,color:#fed7aa
    style E fill:#1e3a5f,stroke:#60a5fa,color:#e0f2fe
    style F fill:#7c2d12,stroke:#f97316,color:#fed7aa
    style G fill:#1e3a5f,stroke:#60a5fa,color:#e0f2fe

Step 1: Investigate and Find the Large Files

Before rewriting history, you need to know what you're dealing with.

Find the biggest files in your entire repo history

git rev-list --objects --all \
  | git cat-file --batch-check='%(objecttype) %(objectname) %(objectsize) %(rest)' \
  | grep blob \
  | sort -k3 -n -r \
  | head -20

This command is written for Unix-like shells. Windows users may need Git Bash or PowerShell equivalents.

What each part does:

Command	Purpose
`git rev-list --objects --all`	Lists every object (blob, tree, commit) across all branches
`git cat-file --batch-check`	For each object, prints its type, hash, size, and file path
`grep blob`	Filters to only files (not directories or commits)
`sort -k3 -n -r`	Sorts by size column, numeric, reverse (biggest first)

This is the command you run when someone asks "why is our repo 2GB?" It reveals every large file ever committed, even deleted ones.

Find which commits touched a specific path

In my terminal (and the screenshots below), I originally ran a simple path match:

git log --oneline -- '*ScreenShots*'

While that works in many shells, glob matching (*) can behave inconsistently across different operating systems. A safer and more robust command for deep investigation is:

git log --all --name-only -- '**ScreenShots**'

Note:
Why is this second version better?

--all ensures you search across every branch in the repository's history (including deleted ones).

--name-only explicitly lists the modified files.

** ensures it matches folders nested deep in the repository structure regardless of your OS shell.

This tells you exactly which commits added, modified, or deleted files matching that pattern. You need this to understand what the history rewrite will affect.

Find when a file was first added

git log --diff-filter=A --oneline -- path/to/file.txt

Flag	Purpose
`--diff-filter=A`	Show only commits where the file was Added (first committed)
`--diff-filter=D`	Show only commits where the file was Deleted
`--diff-filter=M`	Show only commits where the file was Modified

This is how you trace back to the exact commit that introduced a file. Useful when you need to know "when did this secret/binary first enter the repo?"

View a file at a specific commit

# See the file contents as they were at that commit
git show abc123:path/to/file.txt

# See what changed in that specific commit
git show abc123 --stat

git show <commit>:<path> lets you read any file at any point in history without checking it out. Combine this with --diff-filter=A to find the commit, then inspect exactly what was added.

Check your repo size

du -sh .git

Run this before and after cleanup to verify the blobs are actually gone. In my case: 76MB before.

My investigation output

Here's what my actual investigation looked like:

Note:
Note on screenshots: The commit hashes visible in the screenshots below (such as the commit count and refs) reflect the repository state at the time this cleanup was performed. Since then, additional history rewrites have changed all commit hashes, which is exactly what git filter-repo does by design.

Check current status and tracking

# What files are modified, deleted, or untracked?
git status --short

# Which remote branch is your local branch tracking?
git branch -vv

git status --short gives a compact view where each line starts with a status code:

Code	Meaning
`M`	Modified
`D`	Deleted
`??`	Untracked (new file git doesn't know about)
`A`	Added (staged for commit)

git branch -vv shows whether your local branch is tracking a remote branch. After filter-repo, the tracking info is gone, so you'll need to set it up again.

Use `git filter-repo --analyze` for automated reports

Instead of building the bash pipeline manually, git filter-repo has a built-in analysis command that generates reports automatically:

git filter-repo --analyze

Notice the output says "Processed 17 commits". I ran this while my working tree was still dirty. I hadn't committed the pending changes yet. That matters because filter-repo (the actual history rewrite) requires a clean working tree. So I had to commit first, which added one more commit, bringing the total to 18.

When I ran --analyze again after committing, the analysis directory already existed from the first run:

Two things to notice here:

The Error: dir already exists error means you need --force to overwrite a previous analysis
The commit count jumped from 17 → 18 because that extra commit is the one I made to get a clean working tree before running filter-repo

This creates a .git/filter-repo/analysis/ directory with several .txt reports. The most useful ones:

# Which deleted files are eating the most space?
cat .git/filter-repo/analysis/path-deleted-sizes.txt | head -20

# Which deleted directories are the biggest offenders?
cat .git/filter-repo/analysis/directories-deleted-sizes.txt | head -20

This confirmed what the manual investigation already showed: the ScreenShots/ directory was the biggest offender among deleted content, totaling ~3MB of wasted space in the object store.

Step 2: Choose Your Tool (filter-branch vs filter-repo)

There are two tools for rewriting git history. Here's the comparison:

`git filter-branch` (the old way, deprecated)

git filter-branch --force --index-filter \
  'git rm --cached --ignore-unmatch -r path/to/folder/' \
  --prune-empty --tag-name-filter cat -- --all

Flag	What it does
`--force`	Run even if a backup from a previous filter-branch exists
`--index-filter`	Runs a git command on every commit's index without checking out files
`git rm --cached --ignore-unmatch -r`	Remove from index, don't error if file doesn't exist in that commit, recursive
`--prune-empty`	Delete commits that become empty after removing files
`--tag-name-filter cat`	Rewrite tags to point to new commit hashes
`-- --all`	Apply to all branches and refs

Problems:

Slow. Spawns a shell process per commit. Hours on large repos.
Officially deprecated. Git itself warns you to use filter-repo instead.
Creates backup refs in refs/original/ you must manually clean up.
Fragile quoting. Easy to break with special characters in paths.

`git filter-repo` (the modern way, recommended)

git filter-repo --path path/to/folder/ --invert-paths --force

Flag	What it does
`--path <path>`	Select all files matching this path
`--invert-paths`	Flip the selection: exclude instead of include
`--force`	Override safety checks and allow the rewrite to run in situations where filter-repo would normally stop

Advantages:

Often significantly faster, especially on repositories with many commits. Processes the entire history in memory.
No manual cleanup. No backup refs left behind.
Safer. git filter-repo may remove the origin remote as a safety measure, especially when it detects a clone that looks like it came directly from a remote repository. This reduces the chance of accidentally pushing rewritten history before verifying the results.
Simple syntax. One readable line.

Install it:

# Debian/Ubuntu
sudo apt install git-filter-repo

# Or via pip
pip install git-filter-repo

Before You Rewrite History: Make a Backup

Before running any destructive commands, create a backup clone of your repository. If you make a mistake, you can safely restore from this clone.

git clone --mirror https://github.com/your-username/your-repo.git repo-backup.git

Why use --mirror instead of a normal clone? A mirror clone creates a complete, 1:1 backup of the repository, including all branches, tags, remote-tracking references, and the complete object database. Unlike a regular clone, which creates a working copy and checks out only the default branch, a mirror clone copies all Git references (refs/*) and is intended specifically for backup and repository migration, making it the safest choice before rewriting history. Like all Git clones, it only backs up committed repository data, not uncommitted working directory changes.

If the rewrite unexpectedly goes wrong, you can restore from this backup:

# To get a working local copy back:
git clone repo-backup.git restored-repo

The restored repository starts as a normal clone of the mirror backup.

A mirror backup is a recovery point. Do not continue normal development inside the mirror repository.

⚠️ Warning: git push --mirror updates all references on the remote (branches, tags, and other refs) to exactly match the mirror repository. Use it only when you intentionally want to restore the entire repository to the backed-up state.

# To restore the remote repository to its original state:
cd repo-backup.git
git push --mirror https://github.com/your-username/your-repo.git

When to Actually Use This

Rewriting git history is a destructive operation. Use it only when:

Scenario	Urgency	Action
Leaked secret (API key, password)	🔴 Immediate	Rotate the secret first, then filter-repo
Huge binary (500MB video, dataset)	🟡 Moderate	Filter-repo when clone times become a problem
Legal/compliance requirement	🟡 Moderate	Filter-repo as directed
3MB of PNGs (my case)	🟢 Low	Optional cleanup, but a useful exercise for learning history rewriting

Step 3: The Cleanup Process

Here's the exact process I followed on my repo. In my case, the path to remove was content/blog/2026/amazon-linux-qemu-local-lab/media/ScreenShots/.

3.1: Commit and push all current work first

git add -A
git commit -m "your commit message"
git push origin main

Why? Before rewriting history, commit or stash any important local changes. git filter-repo performs destructive history rewriting and may refuse to run in some repository states to prevent accidental data loss. It also may remove the origin remote as a safety measure, especially when it detects that the repository appears to be a normal clone of a remote repository. Push your latest changes so they're safe on the remote before rewriting.

3.2: Run filter-repo

⚠️ Before running this command:

Create a backup
Make sure your working tree is clean
Confirm the path you are removing
Understand that commit hashes will change

git filter-repo --path content/blog/2026/amazon-linux-qemu-local-lab/media/ScreenShots/ --invert-paths --force

What happens under the hood:

Walks through every commit in your repo
For each commit, checks: "does this commit contain files in the specified path?"
If yes, rewrites that commit without those files
Every commit after it also gets a new hash (because its parent changed)
The old blobs become orphaned, and no commit points to them anymore

BEFORE:
  abc123 (has PNGs) → def456 → ghi789

AFTER:
  NEW_A (no PNGs) → NEW_B → NEW_C

Commit messages, authors, and timestamps usually remain unchanged unless explicitly modified. The edited commit receives a new hash. Every descendant commit also receives a new hash because their parent commit reference changed.

Notice the NOTICE: Removing 'origin' remote message. Since git filter-repo removed the origin remote in this case as a safety measure, we will add the remote back after verifying the rewritten history.

3.3: Add the remote back and force push

Since filter-repo removed our remotes as a safety feature, we need to add the remote back and force push the rewritten history.

git remote add origin https://github.com/your-username/your-repo.git

# In my screenshots, I used a standard force push because I was the only one working on this repo:
git push origin main --force

# However, the generally recommended, safer command is:
git push origin main --force-with-lease

Note: If your branch is protected, Git may reject the force push. You may need to temporarily allow force pushes or perform the rewrite through your repository's administrative workflow.

Why do we have to force push at all? Every commit hash changed. Git sees the rewritten history as completely different commits. Normal git push says "add my new commits on top of what's there", but there is no shared history to build on; the histories have diverged. Force pushing tells the remote: "replace the existing branch history with this rewritten version."

Why is --force-with-lease better than --force? A standard --force blindly overwrites the remote branch. If a collaborator pushed a new commit while you were running filter-repo, --force will delete their work. --force-with-lease checks the remote first and aborts if there are new commits you haven't seen yet.

The force push output looks like this:

Enumerating objects: 52, done.
Counting objects: 100% (52/52), done.
Delta compression using up to 4 threads
Compressing objects: 100% (28/28), done.
Total 52 (delta 24), reused 48 (delta 22)
remote: Resolving deltas: 100% (24/24), done.
To https://github.com/your-username/your-repo.git
 + a1b2c3d...e5f6g7h main -> main (forced update)

That last line with (forced update) confirms the rewritten branch history replaced the old one on the remote.

Important: Push rewritten tags too

Transparency Note: My personal blog repository did not use tags, so I did not need to run these tag commands during my cleanup. However, repositories that use tags need to handle them separately. A tag that still points to an old commit can keep rewritten history reachable, and updating tags may also trigger tag-based CI/CD workflows. The following is the standard procedure for synchronizing rewritten tags.

git filter-repo rewrites the commits that branches and tags reference. However, tags need separate attention because remote tag references are not updated when you only force-push a branch.

If your repository uses tags, verify them after the rewrite and push updated tags explicitly. If you only run:

git push origin main --force

only the main branch is updated. Existing remote tags will continue pointing to the old commits.

Those old tag references can keep the old history reachable, including files you intended to remove.

Push the rewritten tags:

For repositories with many tags, prefer updating specific tags rather than force-pushing the entire tag namespace.
Example: git push origin v1.0.0 --force

(If you still want to push all tags):

git push origin --force --tags

Note: This updates existing tags but does not remove remote tags that no longer exist locally.
⚠️ Avoid this on shared repositories unless you have verified the tag namespace. Updating tags can affect releases and automation.

⚠️ Does pushing tags rerun CI/CD? Yes.
A normal branch force push (git push origin main --force) usually triggers workflows configured for push events on branches. A tag update (git push origin --force --tags) can trigger workflows configured with:
on:
  push:
    tags:
      - '*'
If your pipeline publishes releases, Docker images, or deployments from tags, verify the workflow before updating tags.

This updates existing local tags on the remote, but it does not delete remote tags that were removed locally. Use git push origin --delete <tag-name> for those cases.

For example, if you had a tag:

v1.0.0 -> old commit containing deleted files

After filter-repo, your local tag becomes:

v1.0.0 -> rewritten commit without deleted files

Running:

git push origin --force --tags

updates the remote tag:

Remote before:
v1.0.0 -> old history

Remote after:
v1.0.0 -> rewritten history

Removing tags that no longer exist locally

git push --force --tags updates tags that exist locally, but it does not automatically delete remote tags that were removed during the rewrite.

If you deleted a tag locally and want it removed from the remote:

git push origin --delete <tag-name>

Example:

git push origin --delete old-release

Before:

Remote tags:
v1.0.0
v1.1.0
old-release

Local tags after cleanup:

v1.0.0
v1.1.0

After deleting:

Remote tags:
v1.0.0
v1.1.0

The old-release tag no longer keeps the old history reachable.

If you intentionally want the remote tag namespace to exactly match your local tags (deleting any remote tags that don't exist locally) you can use an explicit refspec with pruning:

🔴 Danger: This command deletes remote tags. Any tag that exists on the remote but is missing from your local repository will be permanently removed from the remote. If your local repo is not the complete source of truth for tags (e.g., other team members create release tags, or CI creates tags), you will lose those tags. Only use this if you are the sole maintainer of the tag namespace.

git push --prune origin refs/tags/*:refs/tags/*

This tells Git:

Push every local tag to the matching remote tag
Delete remote tags that no longer exist locally

This does not prune Git objects. It prunes remote tag references that are missing locally.

Example:

Remote tags before:

v1.0.0
v1.1.0
v2.0.0

Local tags after filter-repo cleanup:

v1.0.0
v2.0.0

After running the prune command:

Remote tags:
v1.0.0
v2.0.0

The remote v1.1.0 tag is deleted because it does not exist locally.

🔴 Before running this command, verify your local tags are correct. Run git tag -l locally and compare against the remote with git ls-remote --tags origin. If any tags are missing locally that should be kept, do not use this command. Use individual git push origin --delete <tag-name> instead.

⚠️ Never run git pull between filter-repo and force push. Pulling downloads the old history from the remote and tries to merge it with your rewritten history, defeating the entire purpose. If you accidentally pull, restore from a backup and redo the filter-repo step.

Even after all branches and tags are updated, hosting providers such as GitHub may retain unreachable objects for a period before garbage collection. In repositories with open or historical pull requests, additional internal references may also temporarily keep rewritten commits reachable.

Working with collaborators: Anyone who has already cloned this repository still has the old commits locally. They should re-clone the repository or carefully reset their local branches after the rewrite. Continuing to push from the old clone can accidentally reintroduce the removed history.

3.4: Restore upstream tracking

After rewriting history and re-adding the remote, verify your upstream tracking configuration. If it is missing, restore it with:

git branch --set-upstream-to=origin/main main

What this does: Tells git that your local main branch should track origin/main. Now git pull and git push work without specifying the branch every time.

You can verify tracking is restored:

git branch -vv
# Should show: main 7fe75b2 [origin/main] your latest commit message

3.5: Clean up local storage

git reflog expire --expire=now --all
git gc --prune=now --aggressive

Why? Even after filter-repo, old commits may still be reachable through reflogs until they expire. Git keeps these references as a safety mechanism, preventing garbage collection of the underlying objects. These commands:

reflog expire → clears the safety net (we've verified everything is correct)
gc --prune=now → removes unreachable objects that are no longer referenced by commits, branches, tags, or reflogs
--aggressive → recompresses all objects for optimal packing

Before running this, make sure you have a backup or are confident the rewrite is correct. After garbage collection, recovering deleted objects becomes much harder.

This removes unreachable objects from your local repository. Remote hosting services run their own garbage collection schedules, so old objects may remain temporarily on platforms such as GitHub.

Note:
Note: The --aggressive flag is optional. It performs expensive recompression and is usually unnecessary for large repositories unless you specifically want maximum packing. It can take several minutes on large repositories, so don't panic if it appears to hang. On my 18-commit repo it took about 6 seconds.

Step 4: Verify

# Check repo size (should be smaller)
du -sh .git

# Confirm files are gone from history (should return nothing)
git log --all --oneline -- 'path/to/deleted/folder/'

# Confirm no matching objects in the store
git rev-list --objects --all | grep 'path/to/deleted/folder/' 
# Should return nothing

# Confirm commit count is the same
git log --oneline | wc -l

# Verify tags point to rewritten commits
git show-ref --tags

# Optional: view all refs visually
git log --all --oneline --decorate

# Verify unreachable objects are gone locally
git fsck --full --no-reflogs

git fsck checks for unreachable objects. After a successful cleanup, the removed blobs should no longer appear as reachable objects.

If your repository uses tags, verify they point to the rewritten commits before considering the cleanup complete.

Here's my actual verification output:

My results:

Metric	Before	After
`.git` size	76 MB	73 MB
ScreenShots in history	24 files across 1 commit	None
Commit count	18	18

Why did the commit count stay the same? Although every rewritten commit received a new hash, the total number of commits stayed the same because git filter-repo rewrote existing commits rather than removing them. The count changes only if commits become empty and are removed, for example when using pruning options that remove commits with no remaining changes.

The PNG files themselves were around 3MB, and we saw a corresponding 3MB drop (76MB to 73MB) in the repository size.

Why My Repository Only Shrunk 3MB

You might wonder why removing 24 PNGs totaling 3MB only reduced the repository size from 76MB to 73MB. Git does not store files as simple 1-to-1 copies on disk. It packs objects, uses delta compression, and deduplicates identical files. In larger repositories, removing a 3MB directory does not always reduce the .git folder by exactly 3MB. The real value of this process is understanding how to remove unwanted history, especially secrets or large artifacts that can be hundreds of megabytes.

Prevention: Stop It from Happening Again

The real DevOps skill isn't the cleanup, it's making sure you never need it:

`.gitignore`

Block common offenders before they're ever committed:

# Build artifacts
public/
resources/

# Large media you don't want in git
*.psd
*.mp4
*.zip

Pre-commit hooks

Use pre-commit to block large files automatically:

# .pre-commit-config.yaml
repos:
  - repo: https://github.com/pre-commit/pre-commit-hooks
    rev: v4.6.0
    hooks:
      - id: check-added-large-files
        args: ['--maxkb=500']

Git LFS

For repos that legitimately need large files (design assets, datasets), use Git LFS to store them outside the git object database.

Advanced: Removing Specific Content from Git History

Sometimes the problem is not an entire file.

Example:

A config file contains both safe settings and a leaked password
Deleting the whole file would remove useful history
You only need to remove the sensitive line

There are two approaches depending on the situation:

Problem	Tool
Remove a folder from every commit	`git filter-repo --path --invert-paths`
Replace a leaked password everywhere	`git filter-repo --blob-callback`
Fix one recent commit before pushing	`git rebase -i`

Note:
Simpler alternative for text replacement: If you need to find-and-replace a known secret string across your entire history, git filter-repo also supports a dedicated --replace-text flag with a simple replacements file. See my companion guide: Scrubbing Secrets from Git History for a focused walkthrough of this approach.

For example, imagine you committed this file months ago:

config.txt

DATABASE_HOST=db.example.com
DATABASE_USER=app
DATABASE_PASSWORD=mysecretpassword
LOG_LEVEL=info

You still need the configuration file and its history, but the password string itself must disappear from every previous commit.

If the commit is recent enough that rewriting the following commits is practical, interactive rebase is the simplest approach. If the commit has already been pushed to a shared branch, coordinate with your team before force pushing.

⚠️ Important: Interactive rebase rewrites commit hashes. Avoid using it on shared branches unless you understand the impact. The commit hash of that commit and all later commits will change, and collaborators will need to synchronize with the rewritten history.

# Find the commit that added the lines
git log --diff-filter=A --oneline -- config.txt
# Output: abc123 feat: add database configuration

# Start interactive rebase from one commit BEFORE the target
git rebase -i abc123~1

What abc123~1 means: The ~1 suffix means "one commit before abc123." Rebase needs to start from the commit before the one you want to edit, because it replays commits starting from that point forward.

Your editor opens with a list of commits:

pick abc123 feat: add database configuration
pick def456 fix: update logging
pick ghi789 docs: update README

Change pick to edit for the target commit:

edit abc123 feat: add database configuration
pick def456 fix: update logging
pick ghi789 docs: update README

Save and close. Git pauses at that commit. Now edit the file:

# Remove the 2 offending lines from config.txt
nano config.txt

# Stage the fix
git add config.txt

# Amend the paused commit (keeps the same message)
git commit --amend --no-edit

# Continue replaying the remaining commits
git rebase --continue

What happened: Git rewound history to abc123, let you edit it, then replayed all subsequent commits on top. The rewritten history no longer contains those 2 lines.

⚠️ After any rebase that changes history, you need to force push:
git push origin main --force

If the sensitive content appears in multiple commits or you need to search-and-replace across the entire history:

git filter-repo --blob-callback '
if b"DB_PASSWORD=example_password" in blob.data:
    blob.data = blob.data.replace(
        b"DB_PASSWORD=example_password",
        b"DB_PASSWORD=REDACTED"
    )
' --force

This scans blobs during the history rewrite process and replaces the matching byte sequence wherever it appears. Use this when the sensitive value may exist across multiple commits and you do not know exactly which commits contain it.

⚠️ Security Warning: This only replaces the exact byte sequence you provide; it does not automatically detect secrets. If the value was a real password, API key, or token, rotate it immediately before or during the cleanup process. Rewriting Git history removes references from the repository, but anyone who accessed the secret before the rewrite may still have it.

Quick Reference Card

Task	Command
Find largest files in history	`git rev-list --objects --all \
Find commits that touched a path	{% raw %}`git log --all --oneline -- 'path/*'`
Check repo size	`du -sh .git`
Check file status (compact)	`git status --short`
Check branch tracking	`git branch -vv`
Find when a file was first added	`git log --diff-filter=A --oneline -- path/to/file`
View file at a specific commit	`git show <commit>:path/to/file`
Analyze repo for large/deleted files	`git filter-repo --analyze`
Verify objects exist in history	`git rev-list --objects --all \
Purge a path from all history	{% raw %}`git filter-repo --path <path> --invert-paths --force`
Re-add remote after filter-repo	`git remote add origin <url>`
Force push rewritten history	`git push origin main --force`
Restore upstream tracking	`git branch --set-upstream-to=origin/main main`
Edit a specific past commit	`git rebase -i <commit>~1`
Replace specific content across history	`git filter-repo --blob-callback '<script>' --force`
Clear reflog and garbage collect	`git reflog expire --expire=now --all && git gc --prune=now`

--aggressive can be added when deeper repacking is needed, but it is slower and usually unnecessary.

Aftercare: Repository Consumers

After rewriting history:

Developers should delete old clones or re-clone
CI runners with cached repositories should be refreshed
Docker build caches should be rebuilt if they copied repository contents
Forks and mirrors may still contain the old objects

Copy-Paste Summary

# Backup
git clone --mirror <repo-url>

# Analyze
git filter-repo --analyze

# Remove path
git filter-repo --path <path> --invert-paths --force

# Verify
git log --all -- <path>

# Push rewritten history
git push origin main --force-with-lease

Conclusion

Most developers rarely need to rewrite Git history. It becomes important when something goes wrong and you need to understand what is actually stored inside your repository.

In my case, the problem was only a few unused PNG screenshots. They were not causing a major issue, but they gave me a chance to understand how Git objects work, why deleted files can still remain in history, and how history rewriting, verification, and garbage collection fit together.

The main thing I learned is that removing a file from the current branch is not the same as removing it from repository history. Git is designed to preserve history, which is one of its strengths, but accidental commits remain part of repository history until their references are removed and garbage collection makes the objects unreachable.

A leaked secret, a large binary file, or unwanted build artifacts can become a much bigger problem than a few screenshots. In those situations, knowing how to investigate the repository, create a backup, rewrite history safely, and verify the result can save a lot of time.

History rewriting should still be treated carefully. It changes commit hashes, requires force pushing, and can affect other people working with the repository. Anyone who already cloned the repository may still have the old history locally. After a rewrite, collaborators should re-clone or carefully reset their local branches before pushing new changes. Before doing it on an important project, make sure you understand the impact and have a recovery plan.

For future projects, preventing the problem is always easier than cleaning it up later. A proper .gitignore, pre-commit checks, secret scanning, and Git LFS for large files can prevent many of these situations before they happen.

Accidental commits happen in real-world software projects. Understanding how Git stores data and knowing how to recover from these situations is a valuable skill.

References & Documentation

git-filter-repo (Official GitHub) : The recommended tool for rewriting git history
Git Documentation: git-filter-branch : The deprecated predecessor (includes the recommendation to use filter-repo)
GitHub Docs: Removing sensitive data : GitHub's official guide for purging secrets
GitHub Docs: About secret scanning : Prevent secrets from being committed in the first place
Git Internals: Git Objects : How blobs, trees, and commits work under the hood
pre-commit framework : Automate checks before every commit
Git LFS : Large File Storage for git repos

Found this helpful? Follow me for more real-world Linux troubleshooting and DevOps content.

Originally published on khadirullah.com

How to Create Linked Clones in Virt-Manager (GUI)

Khadirullah Mohammad — Wed, 26 Aug 2026 10:12:50 +0000

If you run virtual machines locally using QEMU/KVM, you should be using Linked Clones.

Instead of copying a 2GB+ master template file over and over (which takes time and wastes SSD space), a linked clone creates a tiny, empty file (often just a few kilobytes). It treats your master template as a read-only base layer.

When the VM reads data, it reads from the master.
When the VM writes data, it writes only to the new, tiny clone file.

Creating linked clones via the command line is simple (qemu-img create -b ...), but trying to do it entirely within the Virt-Manager GUI can be incredibly confusing. There is no simple "Linked Clone" button.

Here is the exact workflow to achieve this using Virt-Manager's Storage Pools and Backing Stores.

The Secret: Storage Pools

Virt-Manager is very strict about where files live. To create a linked clone in the GUI, both your master template and your new clone must live inside recognized Storage Pools.

If you just leave your master template in your ~/Downloads folder, Virt-Manager's UI will not let you select it as a backing file.

Step 1: Create a Template Pool

Let's create a dedicated, organized space for our master OS images.

Open Virtual Machine Manager.
Go to Edit (in the top menu bar) and select Connection Details.
Click the Storage tab.
Click the + (Add Pool) button at the bottom left.
Name the pool templates and set the Type to dir: Filesystem Directory. Click Forward.
Set the Target Path to a clean directory (e.g., /home/youruser/VMs/templates) and click Finish.

Now, use your regular Linux file manager (Nautilus, Dolphin, etc.) to move or copy your master .qcow2 image into that new folder.

⚠️ Crucial Warning: Never boot a VM directly from this master template! If the master file gets modified, any linked clones relying on it will break instantly. Keep this file pristine.

Step 2: Generate the Linked Clone

Now that Virt-Manager knows where your templates live, creating the clone is easy.

Still in the Connection Details -> Storage window, select your default active pool on the left (usually called default, pointing to /var/lib/libvirt/images/). This is where you want the actual VM to live and run.
Click the + (New Volume) button located above the volume list.
Configure the new volume:
- Name: my-new-vm-disk.qcow2 (or whatever you prefer)
- Format: qcow2
Look at the bottom of this window and expand the Backing Store section.
Click Browse... next to the backing store path.
In the window that pops up, select your templates pool from the left menu.
Click on your master .qcow2 file and click Choose Volume.
Click Finish.

Virt-Manager will instantly generate the tiny, linked clone disk inside your active pool. It acts as a lightweight delta layer pointing directly to your master template!

Step 3: Build the VM

Finally, let's attach our new linked clone to a VM.

Close the Connection Details window.
Click the New Virtual Machine button (top left icon).
Select Import existing disk image and click Forward.
Click Browse... and select the my-new-vm-disk.qcow2 you just generated in your active pool.
Finish the setup wizard by allocating RAM and CPUs.

That's it! You have successfully deployed a space-saving linked clone entirely within the Virt-Manager GUI.

What's Next?

If you are using this method to provision cloud images (like Ubuntu Cloud or Amazon Linux), check out my full guide on How to Run Amazon Linux 2023 Locally with QEMU/KVM and Cloud-Init to learn how to inject your SSH keys and configure your instances on first boot!

Found this helpful? Follow me for more real-world Linux troubleshooting and DevOps content.

Originally published on khadirullah.com

How to Run Amazon Linux 2023 Locally with QEMU/KVM and Cloud-Init

Khadirullah Mohammad — Thu, 09 Jul 2026 15:53:50 +0000

Cloud costs add up fast when you're learning DevOps. Running multiple virtual machines on AWS to practice Ansible, Docker, or Kubernetes means paying for every hour of compute, even when you're just experimenting.

The solution? Run the exact same Amazon Linux images that AWS uses on EC2, but locally on your machine using QEMU/KVM. No AWS account needed, no billing surprises, and your VMs boot in seconds.

Using VirtualBox, VMware, or Hyper-V? This guide demonstrates the setup with QEMU/KVM and Virt-Manager, but the Cloud-Init concepts (the user-data and meta-data files, the seed.iso creation, and the password behavior deep dive) apply identically to any hypervisor. The only difference is how you attach the disk image and CD-ROM drive in your VM manager's UI.

The catch? These cloud images don't have a default password. They expect to talk to AWS's metadata service to get their SSH keys and user configuration. On your local machine, that service doesn't exist.

This guide shows you how to solve that problem using Cloud-Init's NoCloud method.

Note: This guide is structured to help beginners get a quick win using the Virt-Manager GUI, but it concludes with an advanced, 8-test deep-dive into Cloud-Init password behavior for senior engineers.

🔥
Hardware Note: I'm running this on a modern quad-core desktop CPU with 16GB RAM. Each Amazon Linux VM idles at ~100-150MB RAM with 1 vCPU. You can comfortably run 2-3 VMs simultaneously on similar hardware.

Why Use Cloud Images Instead of Regular ISOs?

	Cloud Image (.qcow2)	Standard ISO (.iso)
Boot time	~5 seconds	10-15 minute install first
RAM at idle	~100-150 MB (headless)	300+ MB
Disk size	~1.8 GB	2-4 GB
Setup method	Cloud-Init (automated)	Manual wizard
DevOps relevance	✅ Mirrors production EC2 workflow	❌ Nobody installs EC2 from an ISO

Cloud images are pre-built, minimal, headless server images. Exactly what runs on every EC2 instance. Using them locally teaches you the same provisioning workflow that real cloud infrastructure uses.

The Problem: The Missing Metadata Service

When an EC2 instance boots on AWS, Cloud-Init reaches out to a special internal IP address (169.254.169.254) to fetch its configuration. AWS's hypervisor intercepts this request, looks up your SSH key pairs and instance settings, and streams them back to the VM.

On your local QEMU setup, that IP address goes nowhere. Cloud-Init panics, finds no configuration, and you're stuck at a login screen with no credentials.

The Solution: NoCloud

Cloud-Init has a fallback mechanism called NoCloud. Instead of reaching out to a network endpoint, it checks for a locally attached storage drive with the volume label cidata. We create a tiny virtual CD-ROM (seed.iso) containing our configuration files and mount it to the VM. Cloud-Init reads it on first boot and configures everything automatically.

Step 1: Download the Amazon Linux 2023 Image

Grab the official KVM/QCOW2 image from the Amazon Linux 2023 downloads page. Choose the kvm directory for x86_64 or kvm-arm64 for ARM-based hosts.

Step 2: Organize Your Files

Never use the downloaded file directly. QEMU modifies the .qcow2 file when the VM boots. Keep a clean master copy as a template.

# Create your lab structure
mkdir -p ~/VMs/templates ~/VMs/active-vms/vm1

# Save the original as a master template
mv ~/Downloads/al2023-kvm-*.qcow2 ~/VMs/templates/al2023-master.qcow2

# Create a working copy for your first VM
cp ~/VMs/templates/al2023-master.qcow2 ~/VMs/active-vms/vm1/amzn-devops-vm1.qcow2

💡
Pro Tip: Linked Clones. If you plan to spin up many VMs, use a linked clone instead of a full copy, saving gigabytes of disk space.

qemu-img create -f qcow2 \
  -b ~/VMs/templates/al2023-master.qcow2 \
  -F qcow2 \
  ~/VMs/active-vms/vm1/amzn-devops-vm1.qcow2

Step 3: Create the Cloud-Init Configuration

You need two plain text files. No file extensions. Name them exactly user-data and meta-data.

File 1: `user-data`

This configures your user account, SSH key, and a backup console password. Replace the SSH key placeholder with your actual public key (run cat ~/.ssh/id_ed25519.pub to get it).

#cloud-config
users:
  - name: ec2-user
    groups: wheel
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    shell: /bin/bash
    ssh_authorized_keys:
      - ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAA...YOUR_PUBLIC_KEY_HERE...
chpasswd:
  users:
    - name: ec2-user
      password: amazon
      type: text
  expire: false

💡
Using an older Cloud-Init version? Amazon Linux 2023 ships with cloud-init v22.2.2, which also supports the legacy multiline format shown below. Both formats work, but the list format above is recommended since the multiline format will be removed in a future version.

# Legacy format (deprecated, still works on v22.2.2)
chpasswd:
  list: |
    ec2-user:amazon
  expire: False

Field	Purpose
`#cloud-config`	Must be the first line. Tells Cloud-Init this is a valid config
`name: ec2-user`	Creates the default Amazon Linux user
`groups: wheel`	Adds the user to the sudo group
`sudo: ['ALL=(ALL) NOPASSWD:ALL']`	Passwordless sudo access
`ssh_authorized_keys`	Injects your host's public SSH key for password-free login
`chpasswd`	Sets a backup password (`amazon`) for console access via Virt-Manager
`expire: false`	Prevents forcing a password change on first login

💡
Don't have an SSH key? Generate one with the commands below.

ssh-keygen -t ed25519 -C "devops-local"
cat ~/.ssh/id_ed25519.pub  # Copy this entire output into user-data

File 2: `meta-data`

This sets the VM's hostname and a unique instance ID. Do NOT add #cloud-config to this file because that header is only for user-data.

local-hostname: amzn-devops-vm1
instance-id: i-devopslab01

The instance-id is mandatory. Cloud-Init will reject the NoCloud datasource without it. It can be any string, but using the AWS-style i- prefix keeps your mental model consistent with production EC2 instances.

How instance-id actually works: Cloud-Init does NOT check instance IDs across VMs; each VM is completely isolated. The check happens within a single VM only: on every boot, Cloud-Init compares the instance-id from the seed.iso against the one stored on the VM's own disk (/var/lib/cloud/data/instance-id). If they match, it skips first-boot setup. If they differ, it treats it as a brand-new deployment and re-runs provisioning. This is why changing the instance-id forces a fresh Cloud-Init run.

We still use unique IDs per VM (e.g., i-devopslab01, i-devopslab02) primarily so each VM gets a unique local-hostname on the network. Otherwise multiple VMs would show up with the same name, which gets confusing fast.

Step 4: Generate the Seed ISO

Package your configuration files into a virtual CD-ROM image. The volume label must be exactly cidata. Cloud-Init ignores anything else.

Method A: The Modern Way (cloud-localds - Recommended)
The Cloud-Init developers provide a dedicated tool for this that handles volume labels automatically.

Ubuntu/Debian: sudo apt install -y cloud-image-utils
Fedora/RHEL: sudo dnf install -y cloud-utils

cloud-localds seed.iso user-data meta-data
mv seed.iso ~/VMs/active-vms/vm1/

Method B: The Traditional Way (genisoimage - Alternative)
If you require strict ISO9660 formatting or cross-platform compatibility, use the traditional tool:

Ubuntu/Debian: sudo apt install -y genisoimage
Fedora/RHEL: sudo dnf install -y genisoimage

genisoimage -output seed.iso -volid cidata -joliet -rock user-data meta-data
mv seed.iso ~/VMs/active-vms/vm1/

(Curious about which tool is right for you? See the Appendix at the end of this article for a full breakdown of the differences.)

Your vm1 folder should now contain:

~/VMs/active-vms/vm1/
├── amzn-devops-vm1.qcow2    # The VM disk
└── seed.iso                  # Cloud-Init config disk

Step 5: Import into Virt-Manager

Create New VM: Click the top-left icon → Select Import existing disk image → Forward.
Select Disk: Click Browse → Browse Local → Choose your .qcow2 file.
Set OS Type: Type Generic Linux (or Red Hat Enterprise Linux 9) → Forward.
Allocate Resources: Memory: 1024 MiB, CPUs: 1 → Forward.
Customize: Name your VM, check "Customize configuration before install" → Finish.

Attach the Seed ISO (Critical Step)

Click Add Hardware (bottom-left).
Select Storage, change Device type to CDROM device.
Click Manage... → Browse Local → Select your seed.iso → Finish.
Click Apply, then Begin Installation.

Your VM boots in text mode. Cloud-Init reads the seed ISO, creates ec2-user, injects your SSH key, and sets the hostname within seconds.

Step 6: Connect via SSH

Find the VM's IP from Virt-Manager (lightbulb icon → NIC → IP Address), then:

ssh ec2-user@192.168.122.X

You should see the Amazon Linux welcome banner:

   ,     #_
   ~\_  ####_        Amazon Linux 2023
  ~~  \_#####\
  ~~     \###|
  ~~       \#/ ___   https://aws.amazon.com/linux/amazon-linux-2023
   ~~       V~' '->
    ~~~         /
      ~~._.   _/
         _/ _/
       _/m/'
[ec2-user@amzn-devops-vm1 ~]$

🎉 Success! If you are just starting your DevOps journey, you can stop right here! You have a fully functional Amazon Linux 2023 environment, and you can start doing your practice and deploying applications.

🔬 Advanced Reading: The rest of this guide is a technical deep-dive into how Cloud-Init handles passwords under the hood. If you want to level up your Linux troubleshooting skills, keep reading.

The Cloud-Init Deep Dive: What Really Happens to Your Password

This is where most tutorials stop. But when I tried to clean up my VM by removing the seed.iso, everything broke. I ran 8 controlled tests to map out exactly how the password-locking mechanics work under the hood.

The Discovery: How Cloud-Init Manages Passwords

By examining /var/log/cloud-init.log, I found that when Cloud-Init detects a new instance (a new instance-id it hasn't seen before), it runs a two-step process:

Step 1: passwd -l ec2-user     ← LOCKS the account first (users-groups module)
Step 2: chpasswd               ← Sets password, which UNLOCKS it (set-passwords module)

The lock happens first as a security measure. Then chpasswd runs using the password from your user-data file to unlock and set the password. This design ensures that if anything goes wrong with provisioning, the account defaults to locked rather than leaving a default password exposed.

Both modules run with frequency once-per-instance, meaning they only execute once per unique instance-id, then create a semaphore file to remember they already ran. On normal reboots where the instance-id hasn't changed, both modules are skipped entirely. Cloud-Init doesn't touch the password at all.

The 8 Tests

I ran each test on a fresh VM to eliminate leftover state. Here's what I found:

Test 1 & 2: Normal Operation ✅

Fresh boot with seed.iso → Cloud-Init detects a new instance-id, runs passwd -l (locks account), then runs chpasswd (sets password amazon, unlocking the account). Password works on console. SSH key works.

Reboot with seed.iso still attached → Password survives. Cloud-Init sees the same instance-id as the previous boot. The semaphore files exist, so all modules are skipped: no passwd -l, no chpasswd, nothing. The password in /etc/shadow stays exactly as it was.

Test 3: Removing the Seed ISO ❌

Remove seed.iso from Virt-Manager, reboot.

The password is locked. SSH key still works. Here's what the logs revealed:

Cloud-Init can't find the seed.iso, so it falls back to DataSourceNone and assigns a fallback instance-id: iid-datasource-none. Because this is a "new" instance-id, Cloud-Init treats it as a fresh deployment:

passwd -l ec2-user runs → account locked
set-passwords runs under the new instance-id, but there's no user-data to read (the seed.iso is gone) → chpasswd has no password to set
Account stays locked

The SSH key survives because it's a file (~/.ssh/authorized_keys) sitting on the VM's disk. Nothing deletes it.

Test 4: The Semaphore Trap ❌

Re-attach the same seed.iso (same instance-id), reboot.

Password is still locked. The semaphore file from the original first boot (/var/lib/cloud/instances/i-devopslab01/sem/config_set_passwords) still exists on disk. Cloud-Init sees the original instance-id, finds the semaphore, and says:

config-set-passwords already ran (freq=once-per-instance)

It skips chpasswd entirely. The account remains locked from the previous boot (when Cloud-Init ran passwd -l under the fallback instance-id) because no module runs to unlock it.

ℹ️
The Instance-ID Fix: Changing the instance-id in your meta-data to a new value (e.g., i-devopslab02) forces Cloud-Init to treat it as a brand-new deployment. It creates a fresh semaphore directory and re-runs chpasswd, restoring your password.

Test 5: The `cloud-init.disabled` File ❌

Create /etc/cloud/cloud-init.disabled, remove seed.iso, reboot.

Password is still locked. Despite cloud-init status reporting disabled, all four systemd services still executed:

cloud-init.service         → Active: active (exited)   ← Still ran!
cloud-init-local.service   → Active: active (exited)   ← Still ran!
cloud-config.service       → Active: active (exited)   ← Still ran!
cloud-final.service        → Active: active (exited)   ← Still ran!

On Amazon Linux 2023 (cloud-init v22.2.2), the cloud-init.disabled file causes Cloud-Init to self-report as disabled, but the systemd services were still enabled and still ran passwd -l.

Test 6: `systemctl disable` ❌

Use both cloud-init.disabled AND systemctl disable for all four services, remove seed.iso, reboot.

Password is still locked. Even with services marked as disabled:

cloud-init.service; disabled; preset: disabled
Active: active (exited)    ← STILL RAN despite being disabled!

Cloud-Init uses a systemd generator (cloud-init-generator) that dynamically creates service dependencies at boot time. The generator overrides the static disable setting. systemctl disable only removes boot symlinks, but generators can recreate them.

Test 7: `systemctl mask` ✅

Use both cloud-init.disabled AND systemctl mask for all four services, remove seed.iso, reboot.

sudo touch /etc/cloud/cloud-init.disabled
sudo systemctl mask cloud-init-local.service cloud-init.service cloud-config.service cloud-final.service

Password SURVIVES! All services are truly dead:

cloud-init.service         → Loaded: masked    Active: inactive (dead)
cloud-init-local.service   → Loaded: masked    Active: inactive (dead)
cloud-config.service       → Loaded: masked    Active: inactive (dead)
cloud-final.service        → Loaded: masked    Active: inactive (dead)

systemctl mask creates symlinks to /dev/null, making it impossible for any generator or dependency to start the service. No passwd -l runs, the password in /etc/shadow stays untouched, and the SSH host keys are also preserved (no more "REMOTE HOST IDENTIFICATION HAS CHANGED" errors).

I reproduced this test twice on separate fresh VMs to confirm.

Test 8: `lock_passwd: false` in user-data ❌

Add lock_passwd: false to the users block in user-data, remove seed.iso, reboot.

Password is still locked. The lock_passwd: false directive lives inside the user-data file, which lives on the seed.iso. When you remove the ISO, Cloud-Init can't read that directive, so it falls back to Amazon Linux's default config which doesn't include it.

Results Summary

Test	Method	Password	SSH Key
1	Fresh boot with seed.iso	✅ Works	✅ Works
2	Reboot, seed.iso attached	✅ Works	✅ Works
3	Remove seed.iso	❌ Locked	✅ Works
4	Re-attach same seed.iso	❌ Locked	✅ Works
5	`cloud-init.disabled` file	❌ Locked	✅ Works
6	`disabled` + `systemctl disable`	❌ Locked	✅ Works
7	`disabled` + `systemctl mask`	✅ Works	✅ Works
8	`lock_passwd: false` in user-data	❌ Locked	✅ Works

The Definitive Answer

There are three ways to handle seed.iso removal. Pick whichever fits your workflow:

Option 1: Just leave the seed.iso attached (zero effort)

Don't remove it. It uses zero CPU, negligible disk space, and Cloud-Init stays happy on every reboot. This is the simplest option if you don't mind the ISO sitting in your VM's hardware list.

Option 2: Remove seed.iso, then reset the password (easiest fix)

This is the simplest solution if you want a clean VM without the ISO attached:

# 1. Remove seed.iso from Virt-Manager, then boot the VM
# 2. Password won't work on console, but SSH still works:
ssh ec2-user@192.168.122.X
# 3. Manually reset the password:
sudo passwd ec2-user
# Enter your new password (e.g., 'amazon') twice
# 4. Done. Password now survives all future reboots.

Why this works: When you removed the seed.iso and rebooted, Cloud-Init assigned the fallback instance-id iid-datasource-none and created semaphore files for it. From this point on, the instance-id never changes (there's no seed.iso to provide a different one). So on every future reboot, Cloud-Init sees the same iid-datasource-none, finds the existing semaphores, and skips all modules, including passwd -l. Your manually set password in /etc/shadow stays untouched.

Option 3: Mask Cloud-Init, then remove seed.iso (most thorough)

If you want to completely prevent Cloud-Init from ever running again:

# SSH into the VM after first boot
sudo touch /etc/cloud/cloud-init.disabled
sudo systemctl mask cloud-init-local.service cloud-init.service cloud-config.service cloud-final.service
sudo poweroff
# Now remove seed.iso from Virt-Manager's CD-ROM drive

This is the nuclear option: Cloud-Init is permanently dead. Password, SSH keys, and host keys all stay exactly as they were. Use this if you're converting the VM into a long-lived server where Cloud-Init has no purpose.

Creating Additional VMs

For every new VM, you need three unique items:

Item	VM 1	VM 2	VM 3
Folder	`~/VMs/active-vms/vm1/`	`~/VMs/active-vms/vm2/`	`~/VMs/active-vms/vm3/`
`local-hostname`	`amzn-devops-vm1`	`amzn-devops-vm2`	`amzn-devops-vm3`
`instance-id`	`i-devopslab01`	`i-devopslab02`	`i-devopslab03`

Your user-data stays the same since your SSH key doesn't change.

# Quick setup for a second VM
mkdir -p ~/VMs/active-vms/vm2
cp ~/VMs/templates/al2023-master.qcow2 ~/VMs/active-vms/vm2/amzn-devops-vm2.qcow2

# Create meta-data with new hostname and instance-id
cat > meta-data << EOF
local-hostname: amzn-devops-vm2
instance-id: i-devopslab02
EOF

# Regenerate seed.iso
genisoimage -output ~/VMs/active-vms/vm2/seed.iso -volid cidata -joliet -rock user-data meta-data

All VMs on Virt-Manager share the same virtual network bridge (virbr0, subnet 192.168.122.0/24) by default, so they can talk to each other and your host machine out of the box.

Automating the Entire Process

Here's a shell script that replaces the entire manual Virt-Manager workflow:

#!/bin/bash
set -e

TEMPLATE_DIR="$HOME/VMs/templates"
MASTER_TEMPLATE="$TEMPLATE_DIR/al2023-master.qcow2"
USER_DATA_TEMPLATE="$TEMPLATE_DIR/user-data"
BASE_DIR="$HOME/VMs/active-vms"
FULL_COPY=false

# Parse arguments
for arg in "$@"; do
    case "$arg" in
        --full-copy) FULL_COPY=true; shift ;;
    esac
done

if [ "$#" -ne 2 ]; then
    echo "Usage: $0 [--full-copy] <vm-name> <instance-id>"
    echo "Example: $0 amzn-devops-vm2 devopslab02"
    echo ""
    echo "Options:"
    echo "  --full-copy  Create a full disk copy (~1.8 GB) instead of a linked clone (~200 KB)"
    echo "               Linked clones are faster and save disk space, but depend on the master template."
    echo "               Use --full-copy if you plan to move or delete the master template later."
    exit 1
fi

VM_NAME="$1"
INSTANCE_ID="$2"
VM_DIR="$BASE_DIR/$VM_NAME"

# Safety checks
if [ ! -f "$MASTER_TEMPLATE" ]; then
    echo "ERROR: Master template not found: $MASTER_TEMPLATE"
    echo "Download from: https://cdn.amazonlinux.com/al2023/os-images/latest/kvm/"
    exit 1
fi

if [ ! -f "$USER_DATA_TEMPLATE" ]; then
    echo "ERROR: user-data template not found: $USER_DATA_TEMPLATE"
    echo "Create your user-data file at: $USER_DATA_TEMPLATE"
    exit 1
fi

if virsh dominfo "$VM_NAME" &>/dev/null; then
    echo "ERROR: VM '$VM_NAME' already exists. Choose a different name or remove it first:"
    echo "  virsh destroy $VM_NAME && virsh undefine $VM_NAME"
    exit 1
fi

echo "=== Provisioning $VM_NAME ==="

mkdir -p "$VM_DIR"

# Generate unique meta-data
cat <<EOF > "$VM_DIR/meta-data"
local-hostname: $VM_NAME
instance-id: i-$INSTANCE_ID
EOF

# Copy user-data template
cp "$USER_DATA_TEMPLATE" "$VM_DIR/user-data"

# Generate seed ISO
genisoimage -output "$VM_DIR/seed.iso" -volid cidata -joliet -rock "$VM_DIR/user-data" "$VM_DIR/meta-data"

# Create VM disk (linked clone by default, full copy with --full-copy)
if [ "$FULL_COPY" = true ]; then
    echo "Creating full disk copy (~1.8 GB)..."
    cp "$MASTER_TEMPLATE" "$VM_DIR/$VM_NAME.qcow2"
else
    echo "Creating linked clone (~200 KB, depends on master template)..."
    qemu-img create -f qcow2 -b "$MASTER_TEMPLATE" -F qcow2 "$VM_DIR/$VM_NAME.qcow2"
fi

# Deploy headless
virt-install \
  --name "$VM_NAME" \
  --memory 1024 \
  --vcpus 1 \
  --disk path="$VM_DIR/$VM_NAME.qcow2",device=disk,bus=virtio \
  --disk path="$VM_DIR/seed.iso",device=cdrom \
  --os-variant generic \
  --network network=default \
  --import \
  --noautoconsole

echo "=== $VM_NAME deployed! ==="
echo "Find IP: virsh domifaddr $VM_NAME"
echo "Connect: ssh ec2-user@<ip>"

Save as deploy.sh, run chmod +x deploy.sh, and deploy VMs in seconds:

# Linked clone (default, fast, ~200 KB per VM)
./deploy.sh amzn-devops-vm2 devopslab02

# Full disk copy (~1.8 GB per VM, independent of master template)
./deploy.sh --full-copy amzn-devops-vm3 devopslab03

Common SSH Errors

"WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!"

Your host computer remembers a previous VM that used the same IP. Fix:

ssh-keygen -R 192.168.122.X
ssh ec2-user@192.168.122.X    # Type 'yes' when prompted

This happens frequently when you create and destroy VMs that reuse local IPs. For temporary lab VMs:

ssh -o StrictHostKeyChecking=no ec2-user@192.168.122.X

What's Next

With your VM running:

# Update packages
sudo dnf makecache && sudo dnf update -y

# Install Docker
sudo dnf install -y docker
sudo systemctl enable --now docker
sudo usermod -aG docker ec2-user

# Reconnect SSH, then test
docker run -d -p 8080:80 nginx
# Visit http://192.168.122.X:8080 from your host browser

Quick Reference Card

Task	Command
Generate SSH key	`ssh-keygen -t ed25519 -C "devops-local"`
View your public key	`cat ~/.ssh/id_ed25519.pub`
Create seed ISO	`genisoimage -output seed.iso -volid cidata -joliet -rock user-data meta-data`
Clear stale SSH key	`ssh-keygen -R <ip-address>`
Check password status	`sudo passwd -S ec2-user`
Safely disable Cloud-Init	`sudo touch /etc/cloud/cloud-init.disabled && sudo systemctl mask cloud-init-local.service cloud-init.service cloud-config.service cloud-final.service`
Find VM IP address	`virsh domifaddr <vm-name>`
Destroy VM	`virsh destroy <vm-name> && virsh undefine <vm-name>`

Appendix: cloud-localds vs genisoimage

cloud-localds and genisoimage serve the same core purpose for virtual machines: packaging configuration files (like user-data and meta-data) into an ISO image that cloud-init reads. However, cloud-localds is an automated, high-level wrapper specifically for cloud-init, while genisoimage is a low-level, traditional CD/DVD authoring tool.

(Fun Fact: cloud-localds is actually just a wrapper script! Under the hood, it automatically constructs and runs the genisoimage or mkisofs command for you, saving you from having to memorize the correct volume labels and ISO extensions).

Key Differences

Feature	cloud-localds	genisoimage
Purpose	Designed specifically to build NoCloud seed images.	General-purpose filesystem (ISO9660/Joliet/Rock Ridge) creation.
Usage	Requires minimal parameters; automatically sets the correct volume ID and file structure.	Requires exact flags (-volid cidata, -joliet, -rock) to be properly recognized by cloud-init.
Flexibility	Highly opinionated; outputs the ISO directly formatted for local hypervisors (like QEMU).	Granular control over file placement, boot catalogs, and disk attributes.

When to Use Each

Use cloud-localds if:

You are setting up a local testing environment in QEMU/KVM and need to quickly spin up a cloud image.
You want a simple command-line interface that handles volume labeling (cidata) automatically.
You are building a .img or .raw disk file directly, instead of strict ISO formats.

Use genisoimage if:

You require cross-platform compatibility (e.g., building an ISO image on macOS using a wrapper like hdiutil, or using mkisofs on Linux).
You need an exact, standard .iso file format for hypervisors like VMware or Proxmox.

Examples

Using cloud-localds:

cloud-localds -v --network-config=network-config.yaml seed.img user-data.yaml meta-data.yaml

Using genisoimage:

genisoimage -output seed.img -volid cidata -rational-rock -joliet user-data meta-data network-config

References & Documentation

Amazon Linux 2023 KVM Images (Official Download) - Official Amazon CDN for QCOW2 images
Using Amazon Linux 2023 Outside of Amazon EC2 - AWS documentation for running AL2023 locally
AL2023 KVM Requirements - Official KVM hardware and software requirements
Cloud-Init NoCloud Datasource - Official NoCloud documentation covering user-data, meta-data, and instance-id requirements
Cloud-Init Module Reference: Set Passwords - Documentation for the chpasswd and set-passwords module behavior
Cloud-Init Boot Stages - How Cloud-Init's systemd services and generator work
systemctl mask vs disable (Arch Wiki) - Explanation of why mask prevents generators from overriding service state
cloud-localds Manual (Debian Manpage) - Official manual page for the cloud-localds wrapper utility.
genisoimage man page - Manual for the legacy ISO generation tool.

Found this helpful? Follow me for more real-world Linux troubleshooting and DevOps content.

Originally published on khadirullah.com

How I Recovered My Linux GUI After a Full Disk Killed LightDM

Khadirullah Mohammad — Fri, 03 Jul 2026 13:13:39 +0000

I booted my Linux system and instead of the usual login screen — nothing. Just a blinking cursor on tty1. No graphical desktop, no LightDM greeter, no error dialog. Just a text terminal.

This is the story of how I diagnosed the problem from tty1, discovered the real root cause, and fixed it — all without reinstalling anything.

Setup: Debian 13 (Trixie) with Cinnamon desktop, running as a KVM virtual machine with a 28 GB virtual disk and LightDM as the display manager.

🚨 Spoiler: It wasn't a display manager bug. It was a full disk pretending to be a GUI problem.

The Symptom: Stuck on tty1

After logging in on tty1, I tried switching to the graphical session with Ctrl+Alt+F7. Instead of the desktop, I saw:

Failed to start postgresql@17-main.service

PostgreSQL failing was a red herring — it had nothing to do with the GUI. But it was a hint that something deeper was wrong.

Step 1: Check the Display Manager

First, I checked whether the display manager service was running:

systemctl status display-manager

Result: The service was failed but enabled — meaning it was supposed to start at boot, but something prevented it.

Next, I confirmed which display manager was in use:

ls -l /etc/systemd/system/display-manager.service

... -> /lib/systemd/system/lightdm.service

So the system uses LightDM. Let's dig deeper.

Step 2: Investigate LightDM

sudo systemctl status lightdm

LightDM was in a failed state. I tried reconfiguring it:

sudo dpkg-reconfigure lightdm

Then attempted a restart:

sudo systemctl restart lightdm

Still failing. The status output wasn't giving much detail, so I checked the full logs:

sudo systemctl status lightdm --no-pager
sudo journalctl -xeu lightdm --no-pager

The journal showed:

lightdm.service: Triggering OnFailure=dependencies

That's a generic systemd message — it means the service failed, but the reason could be anything. Time to look at the system itself.

Step 3: The Real Culprit — Disk Full

df -h /

Filesystem   Size   Used   Avail   Use%   Mounted on
/dev/vda1    28G    27G    0       100%   /

The root filesystem was completely full. Zero bytes available.

This is the kind of failure that breaks everything silently. When Linux can't write to disk, services fail in confusing ways — no clear error, just "failed."

Why a Full Disk Breaks the GUI

LightDM, Xorg, and the login session all need to write files at startup:

Component	What It Writes	Where
LightDM	Session state, PID files	`/var/run/`, `/tmp/`
Xorg	Display server logs, sockets	`/var/log/`, `/tmp/.X11-unix/`
systemd	Service status, runtime data	`/run/`
Login session	Authentication tokens	`/var/`, `/tmp/`

When any of these writes fail, the service crashes. But the error message rarely says "disk full" — it usually says something vague like "failed to start" or "OnFailure=dependencies."

Step 4: Free Up Disk Space

With the root cause identified, I needed to free space from the terminal. Here's what I did:

Clean the APT Package Cache

sudo apt clean

This removes all cached .deb files from /var/cache/apt/archives/. On a system that's had many package installations, this directory can grow to several gigabytes without you realizing it.

Remove Unused Packages

sudo apt autoremove --purge

This removes orphaned dependencies — packages that were installed automatically but are no longer needed by anything. The --purge flag also removes their configuration files.

Check System Journal Size

journalctl --disk-usage
sudo journalctl --disk-usage

I checked how much space system logs were consuming. If the journal had been consuming significant space, I could have reduced it with sudo journalctl --vacuum-size=200M.

Investigate Where Space Was Going

I also ran a few diagnostic commands to understand the disk usage:

sudo du -xh / --max-depth=1 2>/dev/null | sort -h

Results after cleanup:

/opt    704M
/var    3.2G
/home   5.9G
/usr    8.1G

And checked for deleted files still held open by processes:

sudo lsof +L1

What does this command actually do?

lsof = List Open Files — shows every file currently opened by any running process
+L1 = filter to show only files with a link count less than 1 — meaning the file has been deleted from the filesystem but a process still has it open
sudo = run as root so we can see files opened by all processes, not just our own

Why this matters for disk space

On Linux, deleting a file with rm only removes the directory entry (the name). If a process still has that file open, the actual data stays on disk until the process closes it or is restarted.

This creates a confusing gap between two tools:

Tool	What it reports	Sees deleted-but-open files?
`df -h`	Filesystem-level usage	✅ Yes — counts them as used space
`du -sh /`	Directory-level usage	❌ No — can't see files with no name

So if df says 27 GB used but du only adds up to 18 GB, the difference is likely ghost files — deleted but still consuming space because a process holds them open.

Real-world example

# A log file grows to 5 GB
# You delete it to free space:
rm /var/log/huge.log

# But the service that was writing to it still has it open
# df still shows 5 GB used — space NOT freed!

# lsof +L1 reveals it:
sudo lsof +L1
# COMMAND   PID   USER       FD  TYPE  SIZE/OFF  NLINK  NAME
# postgres  1234  postgres    5w  REG  5368709120  0    /var/log/huge.log (deleted)

# Fix: restart the service to release the file handle
sudo systemctl restart postgresql
# NOW the 5 GB is actually freed

How to read the output

COMMAND   PID   USER       FD   TYPE   SIZE/OFF  NLINK  NAME
postgres  1234  postgres    5w  REG    5368709120  0    /var/log/huge.log (deleted)

Column	Meaning
`COMMAND`	The process holding the file open
`PID`	Process ID — use this to restart or kill it
`SIZE/OFF`	How much space the deleted file is still consuming
`NLINK`	Link count — `0` means deleted from filesystem
`NAME`	Original path, tagged with `(deleted)`

How to fix it

Restart the process that's holding the deleted file:

sudo systemctl restart <service-name>

Once the process releases the file handle, the space is truly freed.

Step 5: Verify and Restart

After cleanup:

df -h /

Filesystem   Size   Used   Avail   Use%   Mounted on
/dev/vda1    28G    19G    8.4G    69%    /

From 100% → 69%. That's roughly 8 GB freed — most of it from the APT cache.

Now I restarted LightDM:

sudo systemctl restart lightdm

The GUI came back immediately. LightDM loaded, the login screen appeared, and the desktop session started normally — no reboot needed.

The Exact Commands I Used

Here's the complete sequence from my shell history — nothing more, nothing less:

# 1. Diagnose the display manager
systemctl status display-manager                          # 1941
ls -l /etc/systemd/system/display-manager.service          # 1942

# 2. Investigate LightDM
sudo systemctl status lightdm                              # 1943
sudo dpkg-reconfigure lightdm                              # 1944
sudo systemctl restart lightdm                             # 1945
sudo systemctl status lightdm                              # 1946
sudo systemctl status lightdm --no-pager                   # 1947
sudo journalctl -xeu lightdm --no-pager                    # 1948

# 3. Discover the root cause
df -h /                                                    # 1949

# 4. Free disk space
sudo apt clean                                             # 1950
journalctl --disk-usage                                    # 1951
sudo journalctl --disk-usage                               # 1952
sudo apt autoremove --purge                                # 1953

# 5. Investigate disk usage
du -sh ~/.cache/pip                                        # 1954
sudo du -xh / --max-depth=1 2>/dev/null | sort -h          # 1955
sudo lsof +L1                                              # 1956

# 6. Verify and recover
df -h /                                                    # 1957
sudo systemctl restart lightdm                             # 1958

What Actually Fixed It

Let's be precise. The fix was not restarting LightDM. The fix was freeing disk space so that LightDM could start.

Action	Effect
`sudo apt clean`	Removed cached `.deb` files (likely several GB)
`sudo apt autoremove --purge`	Removed unused packages and configs
Disk: 100% → 69%	~8 GB freed
`sudo systemctl restart lightdm`	Restarted the now-functional service

No reboot was needed. Once disk space was available, a simple service restart brought the GUI back immediately.

The PostgreSQL failure I saw earlier? Also caused by the full disk — it just happened to display first because it starts before LightDM in the boot sequence.

Lessons Learned

1. "Service failed" Often Means "Disk Full"

When multiple services fail at boot with vague errors, check df -h / before anything else. A full disk is one of the most common — and most misleading — causes of service failures on Linux.

2. APT Cache Can Grow Silently

Every time you run apt install, the downloaded .deb file is cached in /var/cache/apt/archives/. Over months of installing packages, this can consume gigabytes. Unlike pip or npm, APT doesn't auto-clean by default.

Prevention: Add periodic cleanup to your routine:

# Clean APT cache
sudo apt clean

# Remove unused packages
sudo apt autoremove --purge

# Limit journal size
sudo journalctl --vacuum-size=500M

3. `du` and `df` Can Disagree

If df shows more used space than du reports, look for deleted-but-open files:

sudo lsof +L1

This finds files that have been deleted from the filesystem but are still held open by a running process. The space won't be freed until the process is killed or restarted.

4. Keep a Safety Margin

A Linux system needs free disk space to function. When the root filesystem hits 100%, things break in unpredictable ways — services fail, logs can't be written, even apt can't install fixes.

Rule of thumb: Keep at least 2–5 GB free on the root filesystem at all times.

5. Monitor Disk Usage

A simple check you can run anytime:

df -h /

For a more detailed breakdown:

sudo du -xh / --max-depth=1 2>/dev/null | sort -h

Quick Reference: Emergency Disk Cleanup

If you're stuck on tty with a full disk, here's a fast recovery checklist:

# 1. Confirm disk is full
df -h /

# 2. Quick wins (usually frees 1-5+ GB)
sudo apt clean
sudo apt autoremove --purge
sudo journalctl --vacuum-size=200M

# 3. Find biggest directories
sudo du -xh / --max-depth=1 2>/dev/null | sort -h

# 4. Find biggest files
sudo find / -xdev -type f -size +500M -ls 2>/dev/null | sort -k7 -n

# 5. Check for deleted-but-open files
sudo lsof +L1

# 6. Check common caches
du -sh ~/.cache/pip 2>/dev/null
du -sh ~/.cache/huggingface 2>/dev/null
du -sh ~/.ollama 2>/dev/null

# 7. Verify space freed
df -h /

# 8. Restart services or reboot
sudo systemctl restart lightdm   # or your display manager
sudo reboot

Final Thought

This wasn't a LightDM bug. It wasn't a driver issue. It wasn't a broken configuration.

It was a full disk — and every confusing error message was just a downstream symptom of that one simple problem.

In my case, just two commands did the job:

sudo apt clean
sudo apt autoremove --purge

That freed ~8 GB of cached packages and unused dependencies — more than enough to get the system working again.

But What If Cleanup Isn't Enough?

Not every full-disk situation can be solved by clearing caches. If your disk is genuinely running out of space because your data, applications, or workloads have outgrown it, cleanup will only buy you time. In those cases, you'll need to extend the disk itself:

Scenario	Solution
Cloud VM (AWS, GCP, Azure, etc.)	Expand the volume in the cloud console, then run `sudo growpart /dev/vda 1` and `sudo resize2fs /dev/vda1`
Local VM (VirtualBox, KVM, Proxmox)	Increase the virtual disk size in the hypervisor settings, then grow the partition from inside the VM
LVM	Add a new physical volume or extend the logical volume with `lvextend` + `resize2fs`
Physical server	Add a new disk, mount it, and move large directories (like `/home` or `/var`) to it

The key takeaway: always diagnose first. Run df -h / to confirm the disk is full, then decide — is this a cleanup problem or a capacity problem?

If apt clean and autoremove free enough space, great. If not, it's time to give your system more room to breathe.

Found this helpful? Follow me for more real-world Linux troubleshooting and DevOps content.

Originally published on khadirullah.com

Charting New Waters: Building a Custom Coral Source Spec for Internal Enterprise APIs

Khadirullah Mohammad — Sun, 31 May 2026 08:34:33 +0000

⚔️ This post is my submission for the "Chart New Waters" bounty in the Pirates of the Coral-bean hackathon by WeMakeDevs.

Source Spec: payment-api.yaml

The Problem: Enterprises Don't Just Use Public SaaS

When building an Enterprise Agent, there's one massive architectural challenge: enterprises don't just use GitHub, Sentry, and Slack.

A real enterprise relies on hundreds of internal, private microservices — custom payment gateways, user management APIs, internal inventory systems, health monitoring dashboards. These services:

❌ Have no public API documentation
❌ Are not listed in any marketplace
❌ Have no native Coral connector
❌ Sit behind VPNs and private networks

If your incident correlation engine can only query public SaaS tools, you're missing half the picture.

🔥 This is where Coral's Custom Source Specs save the day. One YAML file turns any REST API into a SQL table.

Why Not Just Use Sentry? (Active vs Passive Monitoring)

You might wonder: "If the payment API already sends errors to Sentry, and Coral already reads Sentry, why do we need to query the API directly?"

The answer highlights a critical DevOps distinction:

Passive Monitoring (Sentry):
  Error occurs → Sentry captures snapshot → "What BROKE 5 min ago?"

Active Monitoring (Custom Source):
  Coral queries /api/health → Real-time response → "Is the API UP right now?"

Type	Tool	Question It Answers	When
Passive	Sentry	"What broke 5 minutes ago?"	After the fact
Active	Custom Source Spec	"Is the service alive RIGHT NOW?"	Real-time

During a major outage at 2AM, a DevOps engineer's first question is: "Is the API completely dead, or is it recovering?" Sentry can't answer that — it only logs past errors. The custom source can.

With our spec, the Incident Investigator can execute powerful logic:

"I see the database error in Sentry from 5 minutes ago. Let me instantly query payment_api.health to check if the service is currently online and what the response time is."

How Custom Source Specs Work

Think of a custom source spec as a translator between SQL and HTTP.

The YAML file maps:

SQL table name → HTTP endpoint
SQL columns → JSON response fields
SQL query → HTTP request (method, path, headers)

From the user's perspective, they just write SQL. Coral handles the HTTP call, JSON parsing, and column mapping automatically.

Building the Source Spec: Step by Step

The API We're Connecting

Our demo-payment-api is a Flask microservice with these endpoints:

Endpoint	Returns
`GET /api/health`	Service health, response times, endpoint status
`GET /api/payments`	List of processed and pending payments

In a real enterprise, this could be any internal microservice — a user management API, an inventory system, a billing platform. The pattern is the same.

Step 1: Define the Source Identity

Every custom source needs a name, version, and the Coral DSL version:

name: payment_api          # This becomes the SQL schema: payment_api.health
version: 0.1.0
dsl_version: 3             # Current Coral DSL version
backend: http              # We're connecting to an HTTP API
description: "Internal Payment API for processing and tracking mock payments"
base_url: "http://localhost:5001"   # Where the API lives

ℹ️ The name is critical — it becomes the SQL schema prefix. After registration, you'll query tables as payment_api.health, payment_api.payments, etc.

Step 2: Set Up Authentication

Even internal APIs need authentication. We configure Bearer token auth:

inputs:
  PAYMENT_API_TOKEN:
    kind: secret
    hint: "Bearer token for the Payment API"

auth:
  type: HeaderAuth
  headers:
    - name: Authorization
      from: template
      template: "Bearer {{input.PAYMENT_API_TOKEN}}"

The inputs section tells Coral to ask for a PAYMENT_API_TOKEN environment variable when adding the source. This keeps secrets out of the YAML file itself.

Step 3: Map Endpoints to SQL Tables

This is the magic part. For each API endpoint, we create a SQL table definition:

Table 1: health — Maps GET /api/health

The API returns JSON like:

[
  {"endpoint": "/api/health", "status": "healthy", "response_time_ms": 12, "timestamp": "2026-05-29T18:00:00Z"},
  {"endpoint": "/api/payments", "status": "healthy", "response_time_ms": 45, "timestamp": "2026-05-29T18:00:00Z"}
]

We map it in YAML:

tables:
  - name: health
    description: "Payment service health status and uptime metrics"
    request:
      method: GET
      path: /api/health
    response:
      rows_path: []          # Response IS the array (no wrapper key)
    columns:
      - name: endpoint
        type: Utf8
        expr: { kind: path, path: [endpoint] }
      - name: status
        type: Utf8
        expr: { kind: path, path: [status] }
      - name: response_time_ms
        type: Int64
        expr: { kind: path, path: [response_time_ms] }
      - name: timestamp
        type: Utf8
        expr: { kind: path, path: [timestamp] }

Key details:

rows_path: [] means the JSON response IS the array (not wrapped in a key like {"data": [...]})
Each column's expr: { kind: path, path: [...] } extracts a specific JSON field
type maps JSON types to SQL types (Utf8 = string, Int64 = integer, Float64 = decimal)

Table 2: payments — Maps GET /api/payments

  - name: payments
    description: "List of processed and pending payments"
    request:
      method: GET
      path: /api/payments
    response:
      rows_path: [data]      # Rows are inside the "data" key
    columns:
      - name: id
        type: Utf8
        expr: { kind: path, path: [id] }
      - name: amount
        type: Float64
        expr: { kind: path, path: [amount] }
      - name: currency
        type: Utf8
        expr: { kind: path, path: [currency] }
      - name: status
        type: Utf8
        expr: { kind: path, path: [status] }
      - name: customer
        type: Utf8
        expr: { kind: path, path: [customer] }

Notice rows_path: [data] — this tells Coral the rows are nested inside a "data" key in the JSON response.

Step 4: Add Test Queries

Good source specs include test queries that Coral runs during registration to validate everything works:

test_queries:
  - SELECT status, endpoint FROM payment_api.health LIMIT 1
  - SELECT id, amount, status FROM payment_api.payments LIMIT 1

Step 5: Register with Coral

# 1. Lint the YAML to check for syntax errors
coral source lint coral-config/payment-api.yaml

# 2. Add the source (providing the auth token)
PAYMENT_API_TOKEN=mock_token_123 coral source add --file coral-config/payment-api.yaml

# 3. Query it like a database!
coral sql "SELECT endpoint, status, response_time_ms FROM payment_api.health"

✅ From this moment on, Coral treats your private microservice exactly the same as GitHub or Sentry.

The Complete Source Spec

Here's the full payment-api.yaml:

name: payment_api
version: 0.1.0
dsl_version: 3
backend: http
description: "Internal Payment API for processing and tracking mock payments"
base_url: "http://localhost:5001"

inputs:
  PAYMENT_API_TOKEN:
    kind: secret
    hint: "Bearer token for the Payment API"

auth:
  type: HeaderAuth
  headers:
    - name: Authorization
      from: template
      template: "Bearer {{input.PAYMENT_API_TOKEN}}"

test_queries:
  - SELECT status, endpoint FROM payment_api.health LIMIT 1
  - SELECT id, amount, status FROM payment_api.payments LIMIT 1

tables:
  - name: health
    description: "Payment service health status and uptime metrics"
    request:
      method: GET
      path: /api/health
    response:
      rows_path: []
    columns:
      - name: endpoint
        type: Utf8
        expr: { kind: path, path: [endpoint] }
      - name: status
        type: Utf8
        expr: { kind: path, path: [status] }
      - name: response_time_ms
        type: Int64
        expr: { kind: path, path: [response_time_ms] }
      - name: timestamp
        type: Utf8
        expr: { kind: path, path: [timestamp] }

  - name: payments
    description: "List of processed and pending payments"
    request:
      method: GET
      path: /api/payments
    response:
      rows_path: [data]
    columns:
      - name: id
        type: Utf8
        expr: { kind: path, path: [id] }
      - name: amount
        type: Float64
        expr: { kind: path, path: [amount] }
      - name: currency
        type: Utf8
        expr: { kind: path, path: [currency] }
      - name: status
        type: Utf8
        expr: { kind: path, path: [status] }
      - name: customer
        type: Utf8
        expr: { kind: path, path: [customer] }

The Result: True Enterprise Correlation

With this custom source, the DevOps Incident Investigator can now:

Query internal API health in real-time:

   SELECT endpoint, status, response_time_ms FROM payment_api.health

Cross-reference with Sentry errors:

   -- "Is the payment API healthy after this Sentry error appeared?"
   SELECT h.status, h.response_time_ms, i.title AS error
   FROM payment_api.health h, sentry.issues i
   WHERE i.level = 'error'
   LIMIT 5

Extend to ANY internal service: The same YAML pattern works for user-management APIs, inventory systems, billing platforms — any service with an HTTP endpoint.

Why This Matters for Enterprises

Every enterprise has internal tools that are:

Private — behind VPNs, not publicly accessible
Undocumented — no OpenAPI spec, no marketplace listing
Critical — the payment gateway, the user auth service, the config management system

Without custom source specs, an incident investigator is blind to these services. With them, any REST API becomes a SQL table in minutes.

The pattern is always the same:

Write a YAML file mapping endpoints → tables
Run coral source add --file your-spec.yaml
Query with SQL: SELECT * FROM your_api.your_table

One YAML file. Any API. Full SQL access.

Reproduce It Yourself

# 1. Clone both repos
git clone https://github.com/khadirullah/devops-incident-investigator
git clone https://github.com/khadirullah/demo-payment-api

# 2. Start the payment API
cd demo-payment-api && pip install -r requirements.txt && python3 app.py
# Running on http://localhost:5001

# 3. Register the custom source with Coral
cd ../devops-incident-investigator
PAYMENT_API_TOKEN=mock_123 coral source add --file coral-config/payment-api.yaml

# 4. Query your internal API with SQL!
coral sql "SELECT * FROM payment_api.health"
coral sql "SELECT id, amount, status FROM payment_api.payments LIMIT 5"

ℹ️ Official Guide: How to write a custom source spec →

🔗 View on GitHub

Built as part of the DevOps Incident Investigator for the Pirates of the Coral-bean hackathon by WeMakeDevs.

#ChartNewWaters #Coral #DevOps #CustomSourceSpec #PiratesOfTheCoralBean

Originally published on khadirullah.com

Building a DevOps Incident Investigator with Coral SQL — From 15 Minutes to 15 Seconds

Khadirullah Mohammad — Sun, 31 May 2026 02:37:19 +0000

🏴‍☠️ Built for the Pirates of the Coral-bean hackathon by WeMakeDevs | May 25–31, 2026

TL;DR

Built a DevOps Incident Investigator using Coral SQL that correlates GitHub PRs, Sentry incidents, and Slack incident context using a single SQL query.

Coral turns operational debugging into a single SQL query across distributed systems.

Results:

📉 Incident triage reduced from ~15 minutes to ~15 seconds.
🪸 Custom Coral source spec created for internal APIs.
🤖 AI-generated root cause analysis and Slack alerts.
💻 Includes both a CLI and a Web Dashboard.

Built in 4 days for the Pirates of the Coral-bean hackathon.

The Problem

As a DevOps engineer, I've experienced the pain of incident investigation firsthand — switching between GitHub, Sentry, and Slack at 2 AM trying to figure out what broke production.

GitHub — "Which PR was deployed last?"
Sentry — "What errors are spiking?"
Slack — "What's the team saying?"

That's 3 tabs, 3 APIs, and 15 minutes of context-switching before you even understand what happened. This is what modern incident response looks like without unified observability.

The Solution

When I discovered Coral, an open-source tool that lets you query any API with SQL, I knew exactly what to build.

One command. Three sources. Full incident picture.

python3 investigator.py --query all --owner khadirullah --repo demo-payment-api

The Magic: Cross-Source JOINs

Coral allows querying GitHub and Sentry together in a single statement. For this prototype, I use deployment timing to surface likely suspect PRs that were merged around the same time incidents first appeared.

SELECT p.number AS pr_number, p.title AS pr_title,
       p.user__login AS pr_author, p.merged_at,
       i.short_id AS sentry_id, i.title AS error_title,
       i.level AS error_level, i.first_seen AS error_first_seen
FROM github.pulls p
JOIN sentry.issues i ON p.merged_at IS NOT NULL
WHERE p.owner = 'khadirullah' AND p.repo = 'demo-payment-api'
  AND p.state = 'closed'
ORDER BY i.first_seen DESC, p.merged_at DESC LIMIT 15

🔥 This is the money shot. PRs merged on the same day as errors appeared, side by side. "PR #820 merged at 12:18 PM, errors started at 12:20 PM" — instant suspect identification.

Why I Chose Coral (and Why It's a Game-Changer)

Normally, if you want to pull data from GitHub, Sentry, and Slack, you end up installing multiple SDKs (PyGithub, sentry-sdk, slack-sdk), learning different APIs, handling pagination and retries yourself, and writing glue code to correlate everything manually.

(Note: SDKs like sentry-sdk are excellent for emitting telemetry from applications, but they are still siloed when it comes to querying and correlating operational data across multiple platforms during an incident.)

What Coral Replaced

Before Coral:

GitHub SDK ──┐
Sentry SDK ──┤──→ Custom Glue Code + Pagination + Rate Limiting
Slack SDK  ──┘

After Coral:

Coral SQL ──→ SELECT * FROM github JOIN sentry JOIN slack

By treating the entire operational stack as a unified query layer, I was able to build my agent using far fewer SDKs and almost no glue code. Coral handles the authentication, pagination, and schema mapping locally, allowing me to focus on the actual business logic: writing cross-source JOINs and generating AI analysis.

Day 1: Setting Up Coral & Connecting Sources

Installing Coral

curl -fsSL https://withcoral.com/install.sh | sh
coral --version
# coral 0.3.0+96d61f7

One command. That's it.

Connecting GitHub

ℹ️ For the GitHub token, I created a classic PAT at github.com/settings/tokens with zero scopes selected. For public repos, you don't need any permissions — the token just bumps your API rate limit from 60 to 5,000 requests per hour.

GITHUB_TOKEN=ghp_XXXXX coral source add github

362 tables connected instantly — issues, pull requests, commits, repos, actions.

Connecting Sentry

Finding the Sentry token wasn't obvious. Sentry doesn't have a simple "API Tokens" page — you create tokens through Custom Integrations:

Settings → Custom Integrations → Create New Integration (Internal Integration)
Name it coral-hackathon with minimal read-only permissions (Project: Read, Issue & Event: Read, Organization: Read)
Copy the generated token and connect:

SENTRY_TOKEN=sntrys_XXXXX SENTRY_ORG=my-org coral source add sentry

12 tables connected — events, issues, projects, deployments.

I verified the tables were available with a quick SELECT schema_name, table_name FROM coral.tables.

Connecting Slack

Slack was the trickiest. Coral's pre-filled link creates a Slack app, but it only sets up User Token Scopes with PKCE — tokens aren't displayed in the UI.

⚠️ The fix: Add Bot Token Scopes instead of User Token Scopes. In OAuth & Permissions, scroll to Bot Token Scopes and add: channels:history, channels:read, groups:history, groups:read, users:read. Then reinstall the app — a Bot User OAuth Token (xoxb-...) appears!

coral source add --interactive slack
# Paste xoxb-... token when prompted

All Sources Connected! 🎉

Source	Tables	What It Gives Us
GitHub	362	PRs, commits, issues, actions
Sentry	12	Errors, events, projects
Slack	2	Channels, users
Total	376	All queryable with SQL

Architecture

Here's how the entire system fits together:

Two interfaces (CLI + Web Dashboard) sit on top of Coral SQL, which unifies 4 data sources into a single query layer. The AI layer uses Gemini Flash for root cause analysis and natural language → SQL generation.

Day 2: Writing SQL Queries & Building the CLI

Generating Test Data for Sentry

First roadblock: Sentry was empty. I wrote generate_errors.py to send 10 realistic DevOps errors.

Errors include: ConnectionError (PostgreSQL max connections), MemoryError (OOM kill), RuntimeError (K8s CrashLoopBackOff), TimeoutError (30s timeout), and more.

The SQL Queries

Query 1: Deployments — "What was recently deployed?"

SELECT number, title, user__login AS author, merged_at
FROM github.pulls
WHERE owner = 'khadirullah' AND repo = 'demo-payment-api'
  AND merged_at IS NOT NULL
ORDER BY merged_at DESC LIMIT 10

Query 2: Incidents — "What errors are happening?"

SELECT short_id, title, level, count AS event_count, first_seen, last_seen
FROM sentry.issues
ORDER BY last_seen DESC LIMIT 10

Query 3: Correlation — "Suspect deployment identification"

(As shown in the introduction, this cross-source JOIN correlates GitHub PRs with Sentry errors based on timestamps).

Query 4: Team Overview — Another cross-source JOIN:

SELECT u.name AS username, u.real_name, u.is_admin,
       c.name AS channel_name, c.num_members
FROM slack.users u CROSS JOIN slack.channels c
WHERE u.deleted = false AND c.is_archived = false

Building the CLI

The CLI wraps everything in a clean, colorful interface with zero pip dependencies — Python stdlib only (subprocess, argparse, json, urllib):

python3 investigator.py --query all \
  --owner khadirullah --repo demo-payment-api \
  --slack-token $SLACK_TOKEN

Troubleshooting Gotchas

⚠️ Sentry Project ID: My first query failed with "Invalid project parameter. Values must be numbers." I was using the slug (python) but Sentry wants the numeric ID. Fixed with:
SELECT id, slug, name FROM sentry.projects
⚠️ Slack Bot Permissions: Got not_in_channel error when fetching messages. The bot had channels:history but wasn't in the channel. Then hit missing_scope — needed channels:join scope. Quick fix: add scope → reinstall app → copy new token.

Day 3: Web Dashboard + AI Integration

Why Build a Dashboard?

A CLI is functional, but judges have 5 minutes. They need to see the data. I built a Flask dashboard with a dark glassmorphism design.

The Dashboard

Features:

Stats Row — Live counters for deployments, incidents, correlations, risky PRs
Incident Timeline — Horizontal scrolling event sequence
Deployments Table — Merged PRs with authors and timestamps
Sentry Incidents — Severity-colored cards with "🤖 Analyze" button
Correlation View — Visual PR ↔ Error mapping
Slack Messages — Chat-style from #incidents
Risky PRs — Risk-scored with green/yellow/red bars
Team Overview — Member cards from Slack users × channels JOIN

AI Root Cause Analysis

Click 🤖 Analyze on any Sentry error.

For example, PROD-41A (PostgreSQL max connections):

Root Cause: PR #487 introduced a new connection pooling layer that eagerly opens connections on pod startup. With 4 pods each opening 25 connections, the default max_connections=100 limit is immediately exhausted.

Immediate Fix: 1) Roll back PR #487, 2) Increase max_connections to 200, 3) Restart affected pods

Confidence: 94% (Coral system score)

(Note: This example is generated from demo data and illustrates the style of analysis produced by the assistant.)

In live mode with a Gemini API key, it calls Gemini Flash in real-time. Pre-computed demos ensure instant results without API waits.

Settings & Token Security

All 4 API tokens encrypted with Fernet symmetric encryption before writing to disk
"Delete All Tokens" does a 3-pass random overwrite
Demo → Live toggle with per-panel badges

Demo Mode

The dashboard starts in Demo Mode — pre-loaded data, zero setup. Switching to live is 4 clicks:

⚙️ Settings → enter tokens
💾 Save
Click DEMO badge → LIVE
🔄 Refresh

If a live call fails, panels gracefully fall back to demo data.

Day 4: Competitive Upgrades — Going Beyond

Custom Source Spec: Querying Internal APIs

Enterprises don't just use public SaaS tools. A real incident investigator needs to query internal microservices. This is where Coral's Custom Source Specs shine.

I built a companion project (demo-payment-api) and wrote a YAML spec to teach Coral how to talk to it:

name: payment_api
version: 0.1.0
dsl_version: 3
backend: http
base_url: "http://localhost:5001"

auth:
  type: HeaderAuth
  headers:
    - name: Authorization
      template: "Bearer {{input.PAYMENT_API_TOKEN}}"

tables:
  - name: health
    request:
      method: GET
      path: /api/health
    columns:
      - name: status
        type: Utf8
        expr: { kind: path, path: [status] }
      - name: response_time_ms
        type: Int64
        expr: { kind: path, path: [response_time_ms] }

The YAML acts as a "translator" — it tells Coral how to map SQL concepts (tables, columns) to HTTP concepts (endpoints, JSON paths):

User ──→ Coral: SELECT status FROM payment_api.health
Coral ──→ YAML: Look up "health" table
YAML ──→ Coral: GET /api/health
Coral ──→ API: HTTP GET http://localhost:5001/api/health
API ──→ Coral: {"status": "healthy", "response_time_ms": 42}
Coral ──→ User: | status  | response_time_ms |
                | healthy |       42         |

# Lint the spec
coral source lint coral-config/payment-api.yaml

# Add to Coral
PAYMENT_API_TOKEN=mock_123 coral source add --file coral-config/payment-api.yaml

# Query like a database!
coral sql "SELECT * FROM payment_api.health"

ℹ️ This proves the tool can connect to any internal enterprise service — not just the big SaaS providers. Read more in our dedicated Chart New Waters deep-dive.

Natural Language to SQL (`/api/ask`)

Type a question in plain English → AI generates Coral SQL → executes it → returns results:

User: "Show me critical errors from last week"
  → Gemini Flash
  → SELECT * FROM sentry.issues WHERE level = 'error' LIMIT 10
  → Coral SQL Engine
  → Results Table

The backend feeds the live Coral schema (all tables + columns) to Gemini, so it generates accurate SQL every time.

Automated Slack Alerts

Investigation is only half the battle — communication is the other half. After AI generates a root cause analysis, users can click "📢 Send to Slack" to push the report directly to #incidents:

🚨 Sentry Error → 🤖 AI Analysis → 📢 Send to Slack → #incidents channel → Team sees report

If the token only has read permissions, the UI catches the missing_scope error and gracefully prompts the user to add chat:write scope.

Handling Slack Messages (TVF vs Custom Fallback)

Coral exposes Slack messages via a function-like table interface (TVF), which requires passing the exact channel ID directly into the SQL query: SELECT * FROM slack.messages(channel => 'C12345678').

However, I needed something more robust for an automated dashboard. If the bot isn't already in the incident channel, the Slack API returns a strict not_in_channel error. Instead of failing, I deliberately built a custom Python fallback that catches this error, dynamically forces the bot to join the channel, fetches the messages, and then uses Coral to pull the slack.users table to map the raw User IDs (e.g., UXXXXXXX) to real human names.

Fetching the Team Overview

With the messages handled resiliently, I still needed to display the active Incident Response team. Instead of making separate API calls for users and channels, I let Coral grab the entire team landscape in a single round-trip using a CROSS JOIN:

SELECT u.name AS username, u.real_name, u.is_admin,
       c.name AS channel_name, c.num_members
FROM slack.users u CROSS JOIN slack.channels c
WHERE u.deleted = false AND c.is_archived = false

A quick Python iteration separates the results, giving us the full team and channel rosters instantly without juggling multiple Slack SDK requests.

Why Sentry (Not Just SonarQube & Trivy)

A question I got: "Why do you need Sentry if you have SonarQube and Trivy?" The answer:

Before Deploy:  SonarQube → "Code COULD break"
                Trivy     → "Has known CVEs"
                      ↓
                   Deploy
                      ↓
After Deploy:   Sentry    → "App JUST broke for 1,203 users"

Tool	Stage	Catches
SonarQube	Pre-deploy (CI)	Code smells, potential bugs
Trivy	Pre-deploy (CI)	Known CVEs in dependencies
Sentry	Post-deploy (Runtime)	Real crashes, right now, with stack traces + user impact

The Incident Investigator bridges all three stages — correlating code changes (GitHub) with runtime errors (Sentry) and team communication (Slack).

Setting Up Slack Alerts (Webhooks vs Bot Tokens)

We use both approaches:

Feature	Incoming Webhook	Bot Token (xoxb-)
Direction	App → Slack (one-way)	App ↔ Slack (two-way)
Use case	Post alerts	Read messages + respond
Setup	Just a URL	OAuth scopes
Best for	Simple alerting	Complex integrations

Webhook → Demo Payment API posts error alerts to #incidents
Bot Token → Incident Investigator reads messages from #incidents via Coral SQL

The complete pipeline:

Error in Payment API
  ├──→ Sentry captures exception ──→ Coral SQL queries ──┐
  └──→ Slack webhook fires ──→ #incidents channel ──────┤
                                                         ├──→ Incident Investigator
  GitHub PRs ──────────────────────────────────────────┘       │
                                                               ▼
                                                        Gemini AI Analysis
                                                               │
                                                               ▼
                                                        📢 Push to Slack #incidents

What I Learned

About Coral

Zero-scope GitHub tokens work — just bumps your rate limit for public repos
Sentry wants numeric project IDs — not slugs, query sentry.projects first
Slack's Coral source is limited — only channels and users, no messages. But the bot token works with direct API calls
Cross-source JOINs are the killer feature — GitHub × Sentry in one SQL statement is genuinely powerful

About DevOps Incident Response

The hardest part of incident response isn't fixing the problem — it's finding the right information. We spend more time context-switching than debugging. The investigator answers three questions fast:

What was deployed recently?
What errors appeared after deployment?
What's the team saying about it?

That's 80% of the first 15 minutes of any incident.

About Hackathons

Ship fast, iterate later — the Discord advice was spot-on
Document as you go — writing the blog alongside the code was more efficient
Errors are content — every missing_scope became a blog section
Keep scope tight, then expand — CLI first, dashboard second, AI third

Limitations

Correlation heuristic: GitHub ↔ Sentry correlation is currently based on deployment timing, not deterministic tracing.
AI as an assistant: Root cause analysis is AI-assisted and should be treated as a hypothesis, not absolute truth.
Slack capabilities: Slack messages are retrieved through a Python fallback integration because the native Coral Slack source currently exposes only users and channels.
API rate limits: In a high-traffic live incident, direct API calls to Slack and GitHub could hit rate limits, requiring an intermediate caching layer.
Schema mismatches: Edge cases in custom internal APIs may occasionally fail to map perfectly to Coral's static YAML types without custom data coercions.

The Result

In testing, the workflow reduced incident investigation from roughly 15 minutes of manual context-switching to less than 15 seconds for an initial deployment-to-error correlation query.

The Impact

Before (The Old Way):

❌ Open GitHub to check recent PRs
❌ Open Sentry to check recent errors
❌ Open Slack to read team discussions
❌ Manually correlate timestamps across 3 tabs

After (The Coral Way):

✅ Run one command or view one dashboard
✅ Instantly see PRs and Errors side-by-side
✅ Get AI-generated root cause analysis

Technical Achievements

Feature	Details
6 SQL Queries	Deployments, incidents, correlation, risky PRs, team, health
2 Cross-Source JOINs	GitHub × Sentry, Slack users × channels
1 Custom Source Spec	`payment-api.yaml` for internal microservice
AI Analysis	Gemini-powered root cause + fix suggestions
NL-to-SQL	Ask questions in English, get Coral SQL results
2 Interfaces	CLI (zero deps) + Web Dashboard (Flask)
Security	Fernet symmetric encryption at rest
Docker + CI	Production-ready packaging + GitHub Actions

Try It

git clone https://github.com/khadirullah/devops-incident-investigator
cd devops-incident-investigator
pip install flask google-generativeai cryptography
python3 app.py
# Open http://localhost:5000 — works instantly with demo data!

🔗 View on GitHub

Future Work

Direct Release Correlation: Correlate Sentry releases directly to GitHub commits using commit SHAs.
More Sources: Add Grafana and Kubernetes log sources to the SQL engine.
Automated Postmortems: Use the AI layer to generate and publish full incident postmortems automatically.

Built with Coral for the Pirates of the Coral-bean hackathon by WeMakeDevs.

Follow the journey: khadirullah.com | GitHub

Originally published on khadirullah.com

Setting Up Custom Domain Email with SPF, DKIM, and DMARC

Khadirullah Mohammad — Tue, 19 May 2026 13:30:00 +0000

A complete guide to setting up professional email on your custom domain — with SPF, DKIM, and DMARC explained from the ground up.

When I bought my domain, one of the first things I wanted was a professional email address — contact@yourdomain.com instead of a generic Gmail address. But I also wanted to make sure nobody could spoof my domain to send fake emails pretending to be me.

This post covers everything I did: setting up Zoho Mail, configuring DNS records in Cloudflare, and implementing SPF, DKIM, and DMARC — the three protocols that prove your emails are legitimate.

Why Custom Domain Email?

Option	What It Looks Like	Impression
Gmail	@gmail.com	"Just another person"
Custom domain	contact@yourdomain.com	"Professional, owns their infrastructure"

For a DevOps engineer's website, having a custom domain email shows you understand DNS, mail infrastructure, and security — which is literally part of the job.

Why Zoho Mail?

Provider	Free Tier	Why I Chose/Skipped
Google Workspace	No free tier anymore	Costs $6/month per user
Microsoft 365	No free tier	Costs $6/month per user
ProtonMail	Custom domain on paid plan only	Costs $4/month
Zoho Mail	✅ Free for 5 users, 5GB	Free, custom domain. Note: Web/App only (no IMAP)
Cloudflare Email Routing	Free	Forwarding only — can't send FROM your domain

Zoho Mail gives you a real mailbox on your custom domain for free for up to 5 users. You can send and receive emails as anything@yourdomain.com, though the free plan requires using the Zoho Mail website or mobile app (IMAP/POP for third-party apps like Outlook or Apple Mail is not included). You can upgrade to a paid plan to add IMAP/POP and other features.

Step 1: Add Your Domain to Zoho

Go to Zoho Mail → Sign up for the free plan
Add your domain — Zoho will ask you to verify ownership
Zoho gives you a TXT record to add to your DNS — this proves you own the domain

In Cloudflare DNS:

Type	Name	Content
TXT	`@`	`zoho-verification=zb12345678.zmverify.zoho.com`

After adding this record, click "Verify" in Zoho. Once verified, Zoho gives you the remaining DNS records.

Step 2: MX Records (Mail Exchange)

MX records tell the internet where to deliver emails for your domain. When someone sends an email to contact@yourdomain.com, the sending mail server looks up the MX records for yourdomain.com to find out which mail server should receive it.

Zoho provides these MX records:

Type	Name	Mail Server	Priority
MX	`@`	`mx.zoho.com`	10
MX	`@`	`mx2.zoho.com`	20
MX	`@`	`mx3.zoho.com`	50

Priority determines the order — the sending server tries priority 10 first (mx.zoho.com). If that's down, it tries priority 20, then 50. This gives you redundancy.

Add all three in Cloudflare DNS.

ℹ️ Note: These are the MX records for Zoho's US/global data center (zoho.com). If you signed up at zoho.eu or zoho.in, your MX servers will be different (e.g., mx.zoho.eu). Always use the exact values Zoho provides during setup.

⚠️ Important: Make sure the proxy status for MX records is set to DNS only (gray cloud), not Proxied (orange cloud). Email traffic cannot go through Cloudflare's proxy.

Step 3: SPF (Sender Policy Framework)

What Is SPF?

SPF answers the question: "Which mail servers are allowed to send email on behalf of my domain?"

Without SPF, anyone in the world could send an email that says From: contact@yourdomain.com — and the receiving server would have no way to verify if it's real. SPF fixes this by publishing a list of authorized mail servers in your DNS.

How It Works

1. You send an email from contact@yourdomain.com via Zoho
2. Zoho sends it with a Return-Path (envelope sender) on your domain
3. The receiving mail server looks up the SPF record for that envelope domain
4. SPF record says: "Only Zoho's servers are allowed to send for this domain"
5. Mail server checks: Did this email actually come from Zoho's IP addresses?
   → Yes → SPF passes
   → No  → SPF fails (mark as suspicious or reject)

ℹ️ Technical detail: SPF validates the envelope sender (Return-Path), not the From: header you see in your email client. For simple setups like Zoho, these match your domain — but the distinction matters when you add third-party senders and is the reason DMARC alignment exists as a separate check (explained below).

The Record

Zoho provides this SPF record:

Type	Name	Content
TXT	`@`	`v=spf1 include:zoho.com ~all`

ℹ️ Regional note: Just like MX records, the SPF include: domain varies by Zoho region. If you signed up at zoho.eu, use include:zoho.eu. If zoho.in, use include:zoho.in. Always use the exact SPF record Zoho provides during setup.

Breaking it down:

Part	Meaning
`v=spf1`	This is an SPF record (version 1)
`include:zoho.com`	Allow any server that Zoho authorizes
`~all`	Soft-fail everything else (mark as suspicious but don't reject)

ℹ️ ~all vs -all: The ~ (tilde) means "soft fail" — unauthorized emails are marked suspicious but still delivered. The - (hyphen) means "hard fail" — unauthorized emails are rejected outright. I started with ~all to make sure legitimate emails weren't accidentally blocked. Once you're confident everything works, you can switch to -all for stricter enforcement.

ℹ️ DevOps gotcha: SPF has a 10 DNS lookup limit. Each include: in your SPF record triggers DNS lookups, and nested includes count too. Zoho's include:zoho.com uses a few of those. If you later add services like SendGrid, Mailchimp, or AWS SES, you can easily exceed this limit — causing SPF to permanently fail. Use tools like dmarcian SPF Surveyor to audit your lookup count.

Step 4: DKIM (DomainKeys Identified Mail)

What Is DKIM?

DKIM answers the question: "Was this email actually sent by who it claims, and was it tampered with in transit?"

DKIM uses cryptographic signatures. When Zoho sends an email from your domain, it signs the email with a private key that only Zoho has. The receiving server verifies the signature using a public key that you publish in your DNS.

How It Works

1. You send an email from contact@yourdomain.com via Zoho
2. Zoho signs selected email headers (like From, Subject, Date) and a hash of the body with its private key
3. Zoho adds the signature to the email header as "DKIM-Signature"
4. Receiving mail server looks up the DKIM public key in your DNS
5. Server verifies: Does the signature match the email content?
   → Yes → Email is authentic and untampered
   → No  → Email was forged or modified in transit

The Record

Zoho gives you a DKIM TXT record to add. It looks something like:

Type	Name	Content
TXT	`zmail._domainkey`	`v=DKIM1; k=rsa; p=MIGfMA0GCS...` (long public key)

The zmail._domainkey is the selector — it tells receiving servers where to find the public key for Zoho-signed emails.

You get this record from Zoho's admin panel: Email Admin → Domain → Email Authentication → DKIM. Zoho generates the key pair and gives you the public key to put in DNS. You just copy-paste it into Cloudflare.

Step 5: DMARC (Domain-based Message Authentication, Reporting & Conformance)

What Is DMARC?

DMARC answers the question: "Does the domain in the From: header actually match the domain verified by SPF or DKIM — and what should happen if it doesn't?"

DMARC ties SPF and DKIM together into a unified policy. It tells receiving servers:

Check SPF — did the email come from an authorized server?
Check DKIM — is the signature valid?
Check alignment — does the authenticated domain match the From: header domain?
If neither SPF nor DKIM passes with alignment, apply the policy (nothing, quarantine, or reject)
Send me reports about pass/fail results

Why Alignment Matters

SPF and DKIM alone have a gap: an attacker could set up their own domain with valid SPF and DKIM, but forge your domain in the From: header that the recipient sees. Both SPF and DKIM would pass — for the attacker's domain — but the recipient would still see your spoofed address.

DMARC closes this gap by requiring alignment: the domain authenticated by SPF or DKIM must match the domain in the visible From: header.

Check	What Must Align
SPF alignment	`Return-Path` (envelope sender) domain must match `From:` header domain
DKIM alignment	`d=` domain in the DKIM signature must match `From:` header domain

DMARC passes if at least one of SPF or DKIM passes its check and is aligned with the From: domain. This means an email can fail SPF but still pass DMARC if DKIM passes and is aligned (or vice versa).

For a simple Zoho setup, alignment works automatically — Zoho uses your domain for both the envelope sender and DKIM signature. But if you ever add third-party email services (like Mailchimp or SendGrid), you'll need to make sure they can send with proper alignment, or those emails will fail DMARC.

The Record

Type	Name	Content
TXT	`_dmarc`	`v=DMARC1; p=none; rua=mailto:dmarc@yourdomain.com`

Breaking it down:

Part	Meaning
`v=DMARC1`	This is a DMARC record (version 1)
`p=none`	Policy: don't take action on failures (just monitor)
`rua=mailto:dmarc@yourdomain.com`	Send aggregate reports to this email

⚠️ Heads up: DMARC aggregate reports are not human-readable emails. They arrive as gzipped XML attachments that look like gibberish if you open them directly. You'll need a tool to parse them — services like DMARC Analyzer, Postmark DMARC, or dmarcian can ingest these reports and turn them into dashboards you can actually read.

ℹ️ Other useful DMARC tags you'll encounter:

ruf=mailto:... — Forensic (per-failure) reports with details about individual failures. Few providers send these due to privacy concerns, but it doesn't hurt to include.

adkim=r / aspf=r — Alignment mode: r for relaxed (subdomains can align, e.g., mail.yourdomain.com matches yourdomain.com), s for strict. Defaults are relaxed, which is correct for most setups.

pct=100 — Percentage of messages the policy applies to. Useful for gradually rolling out a stricter policy (e.g., pct=10 to apply p=quarantine to only 10% of failing messages at first).

DMARC Policy Levels

Policy	What Happens on Failure	When to Use
`p=none`	Nothing — just collect reports	Start here. Monitor for 2-4 weeks.
`p=quarantine`	Failed emails go to spam folder	After confirming legitimate emails pass
`p=reject`	Failed emails are rejected entirely	Maximum protection — use after `quarantine` works

💡 Tip: I started with p=none to monitor and make sure my legitimate emails from Zoho were passing SPF and DKIM checks. Once I confirmed everything was working, I could tighten the policy. Don't jump straight to p=reject — you might accidentally block your own emails.

Step 6: Email Aliases and Routing

The Setup

Instead of creating multiple Zoho accounts (the free tier allows up to 5 users, but paid plans charge per user), I use aliases to keep costs down and management simple:

Address	Purpose	Type
`yourname@yourdomain.com`	Main email (admin account)	Primary
`contact@yourdomain.com`	Displayed on website	Alias → forwards to primary

How Aliases Work

When someone sends an email to contact@yourdomain.com, Zoho delivers it to my primary inbox. When I reply, I can choose to reply as contact@yourdomain.com — so the person never sees my primary address.

The Security Catch with Aliases

There is one important detail to note: Zoho allows you to log into your account using any of your aliases as the username.

This can be both good and bad:

The Advantage: It's convenient. You don't have to remember your primary admin email; you can just log in using your public alias.
The Security Risk: If you created a private, hard-to-guess admin email to protect your account, that security is bypassed because attackers can simply use your public alias (like contact@yourdomain.com) to attempt to log into your admin panel.

🛡️ Security Tip: Use Group Aliases — To solve this login vulnerability, you can use Group Aliases instead of regular user aliases. Group aliases are strictly blocked from being used to sign in. With a bit of extra configuration in Zoho, you can still send and receive emails using the group alias, giving you the routing benefits without exposing your account to login attacks.

Folder-Based Rules

I also set up rules in Zoho to automatically organize incoming email:

Rule	Action
Email to `contact@`	Move to "Website" folder
Email from GitHub	Move to "GitHub" folder
Email from LinkedIn	Move to "LinkedIn" folder

This keeps my inbox clean and organized without manual sorting.

How All the Records Work Together

Here's what happens when someone receives an email "from" my domain:

Verifying Your Setup

After adding all the records, verify everything works:

Check DNS Records

# MX records
dig MX yourdomain.com +short
# Expected: 10 mx.zoho.com, 20 mx2.zoho.com, 50 mx3.zoho.com

# SPF
dig TXT yourdomain.com +short
# Expected: "v=spf1 include:zoho.com ~all"

# DKIM
dig TXT zmail._domainkey.yourdomain.com +short
# Expected: "v=DKIM1; k=rsa; p=MIGf..."

# DMARC
dig TXT _dmarc.yourdomain.com +short
# Expected: "v=DMARC1; p=none; rua=mailto:..."

Online Tools

🔗 MXToolbox — checks MX, SPF, DKIM, DMARC, and blacklist status
🔗 Mail Tester — send a test email and get a deliverability score out of 10
🔗 DMARC Analyzer — parse DMARC aggregate reports

Send a Test Email

Send an email from your custom domain to a Gmail address. In Gmail, open the email → click the three dots → "Show original". Look for:

SPF:   PASS
DKIM:  PASS
DMARC: PASS

✅ If all three show PASS, your authentication setup is complete. But read the next section — authentication alone doesn't guarantee inbox delivery.

But Your Emails Can Still Land in Spam

⚠️ Important: Setting up SPF, DKIM, and DMARC is essential — but it doesn't guarantee inbox delivery. These records prove authentication (that you are who you say you are), but major email providers like Gmail and Outlook also evaluate reputation and content.

Here are the other factors that can send your perfectly authenticated emails straight to spam:

1. Spammy Subject Lines or Body Content

Email providers run your email through content filters. If your subject line looks like FREE MONEY — ACT NOW!!! or your body is stuffed with sales language, excessive links, or all-caps text, it gets flagged regardless of your DNS setup. Write emails like a human, not a marketer.

2. IP or Domain Reputation

Every mail server has an IP address, and that IP has a reputation score. If the IP your emails are sent from has been used for spam in the past (even by other users on the same shared server), your emails inherit that bad reputation. You can check your IP's reputation using tools like MXToolbox Blacklist Check or Google Postmaster Tools.

3. Domain and IP Warming

This is the one most people don't know about. When you start sending emails from a brand new domain or IP address, email providers have zero trust in you. You have no sending history, no reputation — you're an unknown.

If you suddenly send 500 emails from a new domain, Gmail will almost certainly flag them. The solution is called warming — you start by sending a small number of emails (5-10 per day) and gradually increase the volume over 2-4 weeks. This lets providers build trust in your sending patterns over time. On shared hosting like Zoho, the IP addresses are already established — it's your domain reputation that starts from zero.

ℹ️ For a personal domain like mine, IP warming isn't a big concern since I only send a few emails a day. But if you're setting up email for a **business or newsletter, IP warming is critical.**

4. Recipients Marking You as Spam

This is the most brutal one. If enough people who receive your emails click the "Report Spam" button, email providers learn that people don't want your emails. Once your spam complaint rate crosses a threshold (Google's threshold is roughly 0.3%), your future emails start landing in spam for everyone — even people who want them.

This is why every newsletter has an unsubscribe link. It's better for someone to unsubscribe than to hit "Report Spam."

💡 Tip: Bottom line: SPF, DKIM, and DMARC get you through the authentication door. But content quality, sender reputation, IP warming, and user engagement determine whether you make it to the inbox or the spam folder. Think of DNS records as your ID card — they prove who you are, but they don't guarantee you'll be invited in.

Summary of All DNS Records

Here's the complete set of DNS records I added in Cloudflare for email:

Type	Name	Content	Proxy
MX	`@`	`mx.zoho.com` (priority 10)	DNS only
MX	`@`	`mx2.zoho.com` (priority 20)	DNS only
MX	`@`	`mx3.zoho.com` (priority 50)	DNS only
TXT	`@`	`v=spf1 include:zoho.com ~all`	—
TXT	`zmail._domainkey`	`v=DKIM1; k=rsa; p=MIGf...`	—
TXT	`_dmarc`	`v=DMARC1; p=none; rua=mailto:...`	—
TXT	`@`	`zoho-verification=...`	—

What I Learned

SPF, DKIM, and DMARC are not optional — without them, your emails land in spam or get rejected by Gmail/Outlook. Most people skip these and wonder why their emails aren't delivered.
Zoho gives you the records — you just paste them — I didn't generate any keys manually. Zoho's admin panel provides every DNS record you need. Your job is to copy them into your DNS provider (Cloudflare in my case) correctly.
Start with p=none for DMARC — don't jump to p=reject immediately. Monitor first, make sure legitimate emails pass, then tighten the policy.
Aliases are powerful — one Zoho account can receive email at multiple addresses. No need to create separate accounts.
DNS propagation takes time — after adding records, wait 15-30 minutes (sometimes up to 48 hours) before testing. Don't panic if verification fails immediately.
MX records must be DNS-only in Cloudflare — if you accidentally proxy them (orange cloud), email delivery breaks. Always set MX records to gray cloud (DNS only).

A Quick Warning: My Personal Choice on Using Custom Domain Email for Logins

💀 Warning: This section is not a general recommendation — it's my personal threat model and decision based on how I use custom domains. If you're only using your domain for professional or business email, you can skip this — but I recommend reading it anyway.

When I first set all this up, my immediate thought was: Awesome, I'm going to create a custom alias for every platform I use. github@yourdomain.com, linkedin@yourdomain.com... you get the idea. It felt super organized.

But after thinking about it for a while, a darker thought crossed my mind: What happens if I die?

Or even if I just forget to renew the domain?

A custom domain is basically a subscription. If I'm not around to keep paying for it, the domain will eventually expire. And once it expires, anyone on the internet can buy it. If someone else does manage to buy my expired domain, they can set up an email server and start receiving all my messages. They could hit "Forgot Password" on my GitHub, get the reset link, and attempt to take over my account.

This creates a dependency chain: accounts → email → domain ownership. Break any link, and everything downstream is at risk.

The Objections I Considered

"Can't I just pay for 10 years upfront or use auto-renew?"
You could! You can prepay for a decade or put a credit card on auto-renew. Those are good practices, but they still aren't bulletproof. Credit cards expire, banks block transactions, and 10 years isn't forever.

"What about leaving a Digital Will?"
You might think that leaving a "Digital Will" with instructions for your family to maintain and renew the domain solves the problem. While it's a nice thought, relying on it for your core security is a bad idea. Your family simply won't care about maintaining your infrastructure as much as you do.

More importantly, they probably don't have the deep technical knowledge required to navigate domain registrars, DNS records, and email hosting. Expecting them to manage all of that — or expecting them to spend money hiring a professional to do it for them while they are grieving — is highly unrealistic.

"But I'll be dead, why do I care?"
You might be thinking this — and honestly, if you don't have anything important tied to the email, maybe you don't need to care! But if you are a developer distributing software that other people rely on, an attacker could push malware or spyware under your name. Furthermore, if you've used that email for government IDs or highly private, sensitive information, you probably don't want a random stranger gaining access to your digital life — because it could ultimately be used to scam, extort, or hurt your family. Who knows what a malicious actor might do with that kind of leverage?

"But what about MFA?"
You might argue that setting up Two-Factor Authentication (2FA/MFA) would stop an attacker from getting in, even if they have access to the email. And technically, you're right. But why even put yourself in that situation? Why rely on a secondary defense mechanism when your primary one (your email) is compromised?

Furthermore, if an attacker is triggering password resets, your accounts will likely get flagged and locked down. Recovering a locked account through customer support is already an incredibly tedious and stressful process. Now imagine trying to prove to support that it's really you, when you don't even own the email address associated with the account anymore! It's a nightmare waiting to happen.

Important Context

This is not a flaw in custom domains themselves. They are widely used in companies, teams, and organizations safely because they have renewal processes, domain management policies, and ownership continuity. The risk is mainly relevant for individuals managing everything alone over long time periods.

What I Changed

Because of this risk, I completely backtracked. I moved all my critical platform logins back to standard public emails like Gmail or Outlook, and now keep my custom domain strictly for professional use:

Purpose	Email	Why
Public-facing (on this website)	`contact@yourdomain.com`	Professional impression
Replies and correspondence	`yourname@yourdomain.com`	Clean sender identity
Account recovery and critical logins	Gmail / Outlook	Can't be bought out from under me

This gives me a balance between professionalism and long-term account safety.

🛡️ Security Tip: If you still want to use your custom domain for everything: That's completely valid — just make sure to add a secure public email (like ProtonMail, Outlook, or Gmail) as a secondary recovery email on all your platforms, enable auto-renew on your domain, and keep your payment methods updated. That way, if your domain host goes down, or you forget to renew, or something bad happens, you still have a reliable backdoor to recover your accounts.

Topic	Resource
SPF specification	RFC 7208 — Sender Policy Framework
DKIM specification	RFC 6376 — DomainKeys Identified Mail
DMARC specification	RFC 7489 — Domain-based Message Authentication
Beginner-friendly explainers	Cloudflare Learning Center — DNS Email Security
Google's sender requirements	Google Email Sender Guidelines
Microsoft's sender requirements	Microsoft Anti-Spam Policies
DMARC report analysis	Postmark's Free DMARC Monitoring

How to Block Internet Access for Any Linux App (While Keeping LAN)

Khadirullah Mohammad — Wed, 08 Apr 2026 12:55:49 +0000

Ever wanted Jellyfin to stay off the internet? Or Chromium to only work on your local network? Maybe you want to test how an app behaves offline — without actually pulling the Ethernet cable.

This guide shows you how to block outbound internet for any specific app on Linux while keeping localhost and your home LAN fully functional.

I'll cover five approaches, from a quick 2-minute wrapper script to a production-hardened Chromium setup that survives apt upgrades. Then I'll show you the fundamental security flaw that most guides never mention — and what to use instead when it actually matters.

🔥 Safety First: Take a Snapshot!
You are modifying core network firewall rules. A simple typo can easily break your internet connection or lock you out of your server. It is highly recommended to take a VM/System Snapshot before starting. If a snapshot is not possible, please take a manual backup of your UFW rules first. Reverting a snapshot takes 10 seconds; troubleshooting a broken firewall can take hours.

Why UFW?

You might wonder why this guide uses UFW instead of raw nftables or iptables. The answer is simple: safety for beginners. If something goes wrong — you accidentally lock yourself out of the network, or an app stops working — you can just run sudo ufw disable or even sudo apt remove ufw to instantly restore full connectivity. With raw nftables, one wrong rule can leave you debugging kernel tables for an hour. UFW is a thin wrapper over iptables/netfilter — same power, much easier to roll back.

How Does This Actually Work?

Every time a process opens a network socket, the Linux kernel stamps it with the process's UID (User ID) and GID (Group ID). The firewall — specifically netfilter, which UFW sits on top of — can inspect those stamps on outgoing packets and decide: accept or reject.

That's the entire trick:

Mark the app's processes with a specific UID or GID
Write firewall rules that allow that UID/GID to reach LAN addresses but reject everything else

For services (Jellyfin, Syncthing), we match by UID because they already run as dedicated users. For desktop apps (Firefox, Chromium), we match by GID using a no-internet group.

Which Approach Should You Use?

Your Situation	Best Option	Difficulty
"I just want to test this quickly"	Option A — Wrapper script	⭐ Easy
Desktop GUI app (Firefox, KeePassXC)	Option B — setgid on ELF	⭐⭐ Medium
System service (Jellyfin, Syncthing)	Option C — UID owner-match	⭐ Easy
Chromium or Electron apps	Option D — dpkg-divert	⭐⭐⭐ Advanced
You don't use UFW	Option E — Direct iptables/nftables	⭐⭐ Medium
Need real enforcement	Bypass-Proof Alternatives — Firejail / namespaces	⭐⭐ Medium

Quick Glossary

Term	Meaning	How to check
UID	User Identifier (numeric)	`id -u username`
GID	Group Identifier (numeric)	`getent group groupname`
EGID	Effective GID — the runtime GID the kernel actually uses for socket ownership	`ps -eo egid,egroup,cmd`
UFW	Uncomplicated Firewall — Debian/Ubuntu frontend for iptables	`sudo ufw status`
`sg`	Run a command with a different primary group	`sg groupname command`
`dpkg-divert`	Debian tool to relocate a package-managed file so your file can sit at the original path	`dpkg-divert --list`
conntrack	Connection tracking — lets the firewall allow replies to established connections	—
owner-match	iptables module that matches packets by the UID/GID of the process that created the socket	—

Before You Start: Back Up Everything

If you didn't take a VM or system snapshot, you must back up your current firewall state. Take 30 seconds to save your current rules so you can easily revert them later:

sudo iptables-save > ~/iptables.before
sudo mkdir -p ~/ufw_rules_backup
sudo cp /etc/ufw/before.rules ~/ufw_rules_backup/before.rules.backup
sudo cp /etc/ufw/before6.rules ~/ufw_rules_backup/before6.rules.backup

If anything goes wrong:

sudo cp ~/ufw_rules_backup/before.rules.backup /etc/ufw/before.rules
sudo cp ~/ufw_rules_backup/before6.rules.backup /etc/ufw/before6.rules
sudo ufw reload

The Firewall Rules (The Core of Everything)

Every option below ends up using the same firewall rules. The only difference is how you mark the app. Here's what the rules look like — you'll paste these into /etc/ufw/before.rules.

Where Exactly to Paste

Open the file and look for the *filter section at the top:

*filter
:ufw-before-input - [0:0]
:ufw-before-output - [0:0]
:ufw-before-forward - [0:0]
← YOUR RULES GO HERE, right after these lines

Your file should initially look like this:

For Desktop Apps (GID Match)

Once you paste your rules into the editor, it should look exactly like this:

Replace GID with your actual numeric group ID:

# --- BEGIN no-internet block (IPv4) ---
-A ufw-before-output -m owner --gid-owner GID -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -m owner --gid-owner GID -d 127.0.0.0/8 -j ACCEPT
-A ufw-before-output -m owner --gid-owner GID -d 10.0.0.0/8 -j ACCEPT
-A ufw-before-output -m owner --gid-owner GID -d 172.16.0.0/12 -j ACCEPT
-A ufw-before-output -m owner --gid-owner GID -d 192.168.0.0/16 -j ACCEPT
-A ufw-before-output -m owner --gid-owner GID -j LOG --log-prefix "Blocked noinet: "
-A ufw-before-output -m owner --gid-owner GID -j REJECT
# --- END no-internet block (IPv4) ---

Do the same in /etc/ufw/before6.rules (use ufw6-before-output, allow ::1 and fe80::/10):

# --- BEGIN no-internet block (IPv6) ---
-A ufw6-before-output -m owner --gid-owner GID -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw6-before-output -m owner --gid-owner GID -d ::1 -j ACCEPT
-A ufw6-before-output -m owner --gid-owner GID -d fe80::/10 -j ACCEPT
# Optional: uncomment for mDNS / DLNA / SSDP LAN service discovery
# -A ufw6-before-output -m owner --gid-owner GID -d ff00::/8 -j ACCEPT
-A ufw6-before-output -m owner --gid-owner GID -j LOG --log-prefix "Blocked noinet v6: "
-A ufw6-before-output -m owner --gid-owner GID -j REJECT
# --- END no-internet block (IPv6) ---

For Services (UID Match — `/etc/ufw/before.rules`)

Same structure, but use --uid-owner with the service's numeric UID:

# --- BEGIN service UID block (IPv4) ---
-A ufw-before-output -m owner --uid-owner UID -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw-before-output -m owner --uid-owner UID -d 127.0.0.0/8 -j ACCEPT
-A ufw-before-output -m owner --uid-owner UID -d 10.0.0.0/8 -j ACCEPT
-A ufw-before-output -m owner --uid-owner UID -d 172.16.0.0/12 -j ACCEPT
-A ufw-before-output -m owner --uid-owner UID -d 192.168.0.0/16 -j ACCEPT
-A ufw-before-output -m owner --uid-owner UID -j LOG --log-prefix "Blocked uid: "
-A ufw-before-output -m owner --uid-owner UID -j REJECT
# --- END service UID block (IPv4) ---

And in /etc/ufw/before6.rules:

# --- BEGIN service UID block (IPv6) ---
-A ufw6-before-output -m owner --uid-owner UID -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A ufw6-before-output -m owner --uid-owner UID -d ::1 -j ACCEPT
-A ufw6-before-output -m owner --uid-owner UID -d fe80::/10 -j ACCEPT
-A ufw6-before-output -m owner --uid-owner UID -j LOG --log-prefix "Blocked uid v6: "
-A ufw6-before-output -m owner --uid-owner UID -j REJECT
# --- END service UID block (IPv6) ---

Why This Order?

The rules are evaluated top-to-bottom, first match wins:

RELATED,ESTABLISHED — Don't break existing connections mid-stream
Loopback (127.x) — App can still talk to localhost
LAN ranges (10.x, 172.16.x, 192.168.x) — App can reach your home network
LOG — Audit blocked attempts in /var/log/kern.log or journalctl
REJECT — Everything else (the actual internet) gets blocked

Safe Way to Edit

⚠️ Warning: Always backup your original firewall rules to a safe, persistent location (like your root directory) before editing. Temporary files in /tmp/ are wiped upon every reboot!

Don't edit the live file directly. Backup, copy to a temp file, edit, test, then apply:

# 1. Create a permanent backup
sudo cp /etc/ufw/before.rules /root/before.rules.backup

# 2. Copy to a temporary file for editing
sudo cp /etc/ufw/before.rules /tmp/before.rules.edit
sudo nano /tmp/before.rules.edit                          # paste your rules

# 3. Syntax check (safe, doesn't apply)
sudo iptables-restore --test < /tmp/before.rules.edit     

# 4. Apply the rules
sudo mv /tmp/before.rules.edit /etc/ufw/before.rules
sudo chown root:root /etc/ufw/before.rules
sudo chmod 644 /etc/ufw/before.rules
sudo ufw reload

Option A: Quick Wrapper Script

Time: 2 minutes · Best for: Testing, quick experiments

This is the fastest way. You create a tiny script that launches any app under a no-internet group.

Setup

# Create the group
sudo groupadd -f no-internet
getent group no-internet    # note the GID (e.g., 1001)

# Add your user to the group so 'sg' doesn't prompt for a password
sudo usermod -aG no-internet $USER

3. Create the wrapper script

sudo tee /usr/local/bin/no-internet > /dev/null <<'EOF'
#!/bin/bash
exec sg no-internet "$@"
EOF
sudo chmod 755 /usr/local/bin/no-internet

Add the GID firewall rules to UFW and reload.

Usage

no-internet firefox &
no-internet steam &
no-internet keepassxc &

Verify It Works

# Should be BLOCKED:
sg no-internet -c 'curl -I -m 10 https://example.com' && echo "FAIL" || echo "BLOCKED ✓"

# Should still work:
sg no-internet -c 'curl -I -m 10 http://192.168.1.1' && echo "LAN works ✓" || echo "FAIL"

Downside: If you launch the app from the desktop menu, it won't use the wrapper. You'd need to edit the .desktop file:

cp /usr/share/applications/firefox.desktop ~/.local/share/applications/
nano ~/.local/share/applications/firefox.desktop
# Change: Exec=firefox %u
# To:     Exec=/usr/local/bin/no-internet firefox %u

Option B: setgid on the Binary

Time: 5 minutes · Best for: Desktop apps you always want restricted

Instead of a wrapper, you set the GID flag directly on the app's binary. Every time it runs — from the menu, terminal, wherever — it automatically gets the no-internet group.

Find the Real Binary

This is important. Many apps have wrapper scripts. You need the actual ELF binary (Executable and Linkable Format — the compiled program file that Linux actually runs):

which firefox                              # might be /usr/bin/firefox
readlink -f "$(which firefox)"             # resolves symlinks
file "$(readlink -f "$(which firefox)")"   # should say "ELF 64-bit"

If file says "shell script" or "Python script", dig deeper — that script calls the real binary somewhere.

Apply setgid

sudo chown root:no-internet /path/to/real/elf/binary
sudo chmod 750 /path/to/real/elf/binary
sudo chmod g+s /path/to/real/elf/binary    # the magic: setgid bit

Now every process spawned from this binary inherits EGID = no-internet, which the firewall matches.

Verify

firefox & sleep 1
ps -eo pid,uid,egid,cmd | grep firefox
# EGID column should show your no-internet GID number

Rollback

sudo chmod g-s /path/to/real/elf/binary
sudo chown root:root /path/to/real/elf/binary
sudo chmod 755 /path/to/real/elf/binary

⚠️ Caveat: This doesn't work on Snap or Flatpak apps — they run in sandboxes with their own network stack. For Flatpak, use Flatseal (GUI) to toggle off "Network" permissions, or run flatpak override --user --unshare=network com.app.Name. For Snap, use snap connections app-name and snap disconnect app-name:network to revoke the network plug. Or install the app as a native .deb.

Option C: Service UID Match

Time: 3 minutes · Best for: Daemons like Jellyfin, Syncthing, qBittorrent

Services already run as dedicated system users. You just match their UID in the firewall. This is the strongest of the five options because a service can't change its own UID.

Find the UID

id -u jellyfin    # e.g., 112

Add UID Rules to UFW

Same as the GID rules above, but use --uid-owner 112 instead of --gid-owner. Paste into before.rules and before6.rules, then:

sudo ufw reload

Test

# Internet should be blocked:
sudo -u jellyfin curl -I -m 10 https://example.com && echo "FAIL" || echo "BLOCKED ✓"

# LAN should work (reaches a local Python HTTP server):
sudo -u jellyfin curl -I -m 10 http://192.168.1.10 && echo "LAN works ✓" || echo "FAIL"

The Ultimate Proof: LAN vs Internet

One of the best ways to verify your setup is to try reaching an external site and a local IP in the same process. Here is the result of that test:

Don't Forget: Allow Incoming on the Service Port

If your UFW default is "deny incoming" (it should be), LAN clients can't reach your service unless you explicitly allow the port:

sudo ufw allow from 192.168.0.0/16 to any port 8096 proto tcp

For Custom Services Without a Dedicated User

sudo adduser --system --group --no-create-home --shell /usr/sbin/nologin myservice
sudo passwd -l myservice
id -u myservice    # use this UID in rules

Option D: dpkg-divert + Wrapper

Time: 15 minutes · Best for: Chromium, Electron, multi-process apps

Note: dpkg-divert is a Debian/Ubuntu tool. If you're on Fedora, Arch, or another distro, you'll need to manually relocate the binary instead — the firewall rules themselves are distro-agnostic.

Chromium is special. It spawns renderer processes, GPU processes, utility processes — all from different code paths. A simple setgid on one binary won't catch them all.

The solution: use Debian's dpkg-divert to relocate the real binary, then put a wrapper at the original path. Every invocation — menu, terminal, child processes — goes through your wrapper.

The Full Setup

# 1. Create the group
sudo groupadd -f no-internet
getent group no-internet    # note the GID

# Add your user to the group so 'sg' doesn't prompt for a password
sudo usermod -aG no-internet $USER

# 2. Divert the real binary to a new location
sudo mkdir -p /usr/lib/chromium
sudo dpkg-divert --local --add --rename \
  --divert /usr/lib/chromium/chromium.distrib /usr/bin/chromium

# 3. Reinstall so the diverted file lands at the new path
sudo apt install --reinstall chromium

# 4. Lock down the real binary
sudo chown root:no-internet /usr/lib/chromium/chromium.distrib
sudo chmod 0750 /usr/lib/chromium/chromium.distrib

# 5. Put a shell wrapper at the original path
sudo tee /usr/bin/chromium > /dev/null <<'EOF'
#!/bin/bash
exec sg no-internet /usr/lib/chromium/chromium.distrib "$@"
EOF
sudo chmod 0755 /usr/bin/chromium

Add the GID firewall rules, reload UFW, and test.

Option D Variant: Compiled C Wrapper

Instead of a shell wrapper, you can compile a minimal C binary. It avoids spawning an extra bash process and the binary isn't human-readable (though strings will still reveal the path — see Security Limitations below).

Save as /tmp/sg-wrapper.c:

/* sg-wrapper.c — execv /bin/sg no-internet -- /usr/lib/chromium/chromium.distrib */
#define _GNU_SOURCE
#include <errno.h>
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

int main(int argc, char *argv[]) {
    const char *group = "no-internet";
    const char *sg_path = "/bin/sg";
    const char *real_binary = "/usr/lib/chromium/chromium.distrib";

    int extra = argc - 1;
    /* count: sg_path + group + "--" + real_binary + extra_args + NULL */
    int sg_argc = 1 + 1 + 1 + 1 + extra + 1;
    char **sg_argv = calloc(sg_argc, sizeof(char *));
    if (!sg_argv) { fprintf(stderr, "calloc failed\n"); return 127; }

    int i = 0;
    sg_argv[i++] = (char *)sg_path;
    sg_argv[i++] = (char *)group;
    sg_argv[i++] = (char *)"--";
    sg_argv[i++] = (char *)real_binary;
    for (int j = 1; j < argc; ++j) sg_argv[i++] = argv[j];
    sg_argv[i] = NULL;

    execv(sg_path, sg_argv);
    fprintf(stderr, "execv(%s) failed: %s\n", sg_path, strerror(errno));
    /* free is technically unreachable if execv succeeds, but kept for completeness */
    free(sg_argv);
    return 126;
}

Compile and install:

gcc -O2 -s -o /tmp/sg-wrapper /tmp/sg-wrapper.c
sudo mv /tmp/sg-wrapper /usr/bin/chromium
sudo chown root:no-internet /usr/bin/chromium
sudo chmod 2751 /usr/bin/chromium    # setgid(2) + rwx(7) + r-x(5) + --x(1)

Surviving `apt upgrade`

Package updates can overwrite your changes. Protect them:

# Tell dpkg to enforce ownership/permissions
sudo dpkg-statoverride --add root no-internet 0750 /usr/lib/chromium/chromium.distrib

# Create a script that reapplies permissions
sudo tee /usr/local/sbin/reapply-noinet.sh > /dev/null <<'EOF'
#!/usr/bin/env bash
set -euo pipefail
GROUP=no-internet
[ -e /usr/bin/chromium ] && chown root:$GROUP /usr/bin/chromium && chmod 2751 /usr/bin/chromium || true
[ -e /usr/lib/chromium/chromium.distrib ] && chown root:$GROUP /usr/lib/chromium/chromium.distrib && chmod 0750 /usr/lib/chromium/chromium.distrib || true
EOF
sudo chmod 755 /usr/local/sbin/reapply-noinet.sh

# Hook it into APT so it runs after every package update
sudo tee /etc/apt/apt.conf.d/99-reapply-noinet > /dev/null <<'EOF'
DPkg::Post-Invoke {"[ -x /usr/local/sbin/reapply-noinet.sh ] && /usr/local/sbin/reapply-noinet.sh";};
EOF

Rollback

sudo rm -f /usr/bin/chromium
sudo dpkg-divert --remove --rename /usr/bin/chromium
sudo apt install --reinstall chromium

Option E: Raw iptables / nftables

Best for: Systems that don't use UFW, or if you prefer direct control.

iptables

GID=1001  # your no-internet group ID

sudo iptables -I OUTPUT 1 -m owner --gid-owner $GID -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
sudo iptables -I OUTPUT 2 -m owner --gid-owner $GID -d 127.0.0.0/8 -j ACCEPT
sudo iptables -I OUTPUT 3 -m owner --gid-owner $GID -d 10.0.0.0/8 -j ACCEPT
sudo iptables -I OUTPUT 4 -m owner --gid-owner $GID -d 172.16.0.0/12 -j ACCEPT
sudo iptables -I OUTPUT 5 -m owner --gid-owner $GID -d 192.168.0.0/16 -j ACCEPT
sudo iptables -A OUTPUT -m owner --gid-owner $GID -j LOG --log-prefix "NOINTERNET: "
sudo iptables -A OUTPUT -m owner --gid-owner $GID -j REJECT

Persist with:

sudo apt install iptables-persistent
sudo netfilter-persistent save

nftables

Add to /etc/nftables.conf:

table inet lanlock {
  chain output {
    type filter hook output priority 0;
    meta skgid 1001 ct state related,established accept
    meta skgid 1001 ip daddr 127.0.0.0/8 accept
    meta skgid 1001 ip daddr 10.0.0.0/8 accept
    meta skgid 1001 ip daddr 172.16.0.0/12 accept
    meta skgid 1001 ip daddr 192.168.0.0/16 accept
    meta skgid 1001 ip6 daddr ::1 accept
    meta skgid 1001 ip6 daddr fe80::/10 accept
    meta skgid 1001 counter log prefix "NOINTERNET: "
    meta skgid 1001 drop
  }
}

sudo nft -f /etc/nftables.conf
sudo systemctl enable --now nftables

The Security Flaw Nobody Talks About

Now that you know how to set this up, let's talk about when it's actually enough — because the GID-based approach (Options A, B, and D) has a fundamental bypass that most guides never mention.

The Problem: EGID vs Supplementary Groups

The firewall's --gid-owner match checks the process's EGID (Effective Group ID) — not its supplementary group list. Here's what that means in practice:

How the app is launched	Process EGID	Firewall matches?	Internet?
Via wrapper (`sg no-internet ...`)	`no-internet` (1001)	✅ Yes	❌ Blocked
Directly (`/usr/lib/chromium/chromium.distrib`)	User's primary group (1000)	❌ No	✅ Full access

When a user runs a binary directly, their primary group becomes the EGID. The no-internet supplementary group membership is irrelevant to the firewall.

And there's a catch-22: sg (which the wrapper uses) requires the user to be a member of the no-internet group. But if they're a member, they also have permission to execute the chmod 0750 binary directly — bypassing the wrapper entirely.

"What If I Hide the Binary Path?"

You might think: "I'll compile the wrapper as a C binary so users can't read the script to find the real path." That doesn't work either:

Attempt	Why it fails
Compiled C wrapper	`strings /usr/bin/chromium` reveals the embedded path
Random filename	`ps aux` and `/proc/PID/exe` expose it at runtime
setgid on the binary itself	Chromium and Firefox refuse to run with setgid (browser security feature)

So When IS the GID Approach Good Enough?

✅ Self-discipline — you want YOUR OWN app to stop phoning home (telemetry, metadata downloads, auto-updates)
✅ Services and daemons — Option C uses UID matching, which IS unbypassable since processes can't change their own UID
✅ Non-technical users — people who won't think to look for the diverted binary

When You Need Something Stronger

❌ Technical users who actively want to bypass your restrictions
❌ Multi-user machines where you're enforcing policy
❌ Any scenario where "security through obscurity" isn't acceptable

For those cases, keep reading.

Bypass-Proof Alternatives (Not-Tested By Me)

When the GID approach isn't enough, here are three methods that provide real, kernel-enforced isolation.

Note: I haven't personally tested these alternatives end-to-end. They're included for completeness based on documentation and community guides. If you try any of these and find issues (or get them working), feel free to reach out.

Alternative 1: Separate User + UID Match

Run the app as a completely separate user. UID matching cannot be bypassed — a user can't change their own UID.

# Create a restricted user
sudo adduser --disabled-password --gecos "" --shell /usr/sbin/nologin chromium-user
sudo passwd -l chromium-user
id -u chromium-user    # use this UID in UFW rules (same format as Option C)

# Allow X11 display access
xhost +SI:localuser:chromium-user

# Launch
sudo -u chromium-user chromium

Tradeoffs: You lose your keyring, D-Bus session, bookmarks, and cookies from your main user. Wayland compositors may block other users entirely. But the network restriction is absolute.

Alternative 2: Firejail (Easiest True Isolation)

Firejail uses kernel network namespaces under the hood. No firewall rules needed — the app physically cannot see the external network.

sudo apt install firejail

# No network at all — this works reliably
firejail --net=none chromium

⚠️ My experience: firejail --net=none works perfectly — the app has zero network access. However, I was unable to get LAN-only mode working using the theoretical setup for reference, but your mileage may vary.

LAN-only (theoretical):

firejail --netfilter=/etc/firejail/lan-only.net chromium

Create /etc/firejail/lan-only.net:

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT DROP [0:0]
-A OUTPUT -d 127.0.0.0/8 -j ACCEPT
-A OUTPUT -d 10.0.0.0/8 -j ACCEPT
-A OUTPUT -d 172.16.0.0/12 -j ACCEPT
-A OUTPUT -d 192.168.0.0/16 -j ACCEPT
-A OUTPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
COMMIT

Alternative 3: Network Namespaces (Manual, Full Control)

For maximum control, create a network namespace directly. No extra packages needed.

# Create a namespace with no external network
sudo ip netns add no-inet

# Run the app inside it
sudo ip netns exec no-inet sudo -u $USER chromium

# Optional: Add LAN-only access via a veth pair
sudo ip link add veth-host type veth peer name veth-jail
sudo ip link set veth-jail netns no-inet
sudo ip addr add 192.168.100.1/24 dev veth-host
sudo ip link set veth-host up
sudo ip netns exec no-inet ip addr add 192.168.100.2/24 dev veth-jail
sudo ip netns exec no-inet ip link set veth-jail up
sudo ip netns exec no-inet ip link set lo up

Quick Comparison

Threat Model	Best Solution
Block your own apps from phoning home	GID wrapper (Option A/D) — simple, good enough
Block a daemon/service	UID owner-match (Option C) — unbypassable
Restrict technical/untrusted users	Separate user + UID match (Alt 1)
True network sandbox, easy setup	Firejail (Alt 2)
Full manual control, no dependencies	Network namespace (Alt 3)
Enterprise/production	AppArmor + containers

Troubleshooting

"sg: no such group"
→ Group doesn't exist yet. Run sudo groupadd -f no-internet.

Internet is still working after adding rules
→ Double-check the numeric UID/GID in your rules matches reality. Make sure you pasted the block right after the :ufw-before-output line, not at the bottom. Run sudo ufw reload.

UFW reload fails
→ Syntax error in your rules. Test before applying: sudo iptables-restore --test < /etc/ufw/before.rules. If it fails, restore your backup.

It works, but breaks after reboot
→ You might have iptables-persistent installed, which conflicts with UFW. Remove it: sudo apt remove iptables-persistent. Let UFW handle everything.

setgid isn't working
→ You probably applied it to a shell script wrapper, not the real ELF binary. Use readlink -f $(which app) and file to find the actual binary.

Snap/Flatpak apps are unaffected
→ They run in sandboxes with their own network stack. Flatpak: Use Flatseal (GUI) to toggle off "Network" permissions, or run flatpak override --user --unshare=network com.app.Name. Snap: Use snap connections app-name and snap disconnect app-name:network to revoke the network plug. Or install the app as a native .deb.

DNS seems to leak
→ systemd-resolved runs on 127.0.0.53. Since we allow 127.0.0.0/8, DNS resolves even for blocked apps — but the actual connections still get rejected.

Testing Checklist

After setting up any option, run through this:

# 1. Group exists and GID is correct?
getent group no-internet
# Expected: no-internet:x:<GID>:

# 2. Service UID correct? (Option C only)
id -u jellyfin
# Expected: numeric UID, e.g., 107

# 3. File ownership and permissions correct? (Options B/D)
stat -c "%n: %U %G %a" /usr/lib/chromium/chromium.distrib /usr/bin/chromium
# Expected: real binary → root:no-internet 0750, wrapper → per your policy

# 4. Running processes have correct EGID/UID?
ps -eo pid,ppid,uid,euid,gid,egid,cmd | egrep 'chromium|jellyfin|firefox'
# Look for: EGID == no-internet GID (Options A/B/D) or UID == service UID (Option C)

# 5. Internet blocked?
sg no-internet -c 'curl -I -m 10 https://example.com' && echo "FAIL" || echo "BLOCKED ✓"
# For services:
sudo -u jellyfin curl -I -m 10 https://example.com && echo "FAIL" || echo "BLOCKED ✓"

# 6. LAN still works?
sg no-internet -c 'curl -I -m 10 http://192.168.1.1' && echo "LAN works ✓" || echo "FAIL"

# 7. Check firewall logs (if LOG rules added)
sudo journalctl -k --since "10 minutes ago" | grep -i 'Blocked\|NOINTERNET'

Emergency Rollback

If something goes wrong, these commands restore everything:

# Restore UFW backups
sudo cp /root/before.rules.bak /etc/ufw/before.rules
sudo cp /root/before6.rules.bak /etc/ufw/before6.rules
sudo ufw reload

# If you need immediate connectivity recovery
sudo iptables -I OUTPUT 1 -m owner --gid-owner <GID> -j ACCEPT
# Remove when fixed:
sudo iptables -D OUTPUT -m owner --gid-owner <GID> -j ACCEPT

# Last resort — disable the entire firewall
sudo ufw disable
# Fix your rules, then: sudo ufw enable

# Undo dpkg-divert (Option D)
sudo dpkg-divert --remove --rename /usr/bin/chromium
sudo apt install --reinstall chromium

🎬 Watch it in Action: Full GUI Demo

This walkthrough puts the system to the test using a real-world browser (Google Chrome). Here’s exactly what you’ll see:

🚫 The Block
Chrome tries to reach Google—and fails instantly while the firewall rules are active.

🌐 LAN Routing
Despite the block, Chrome successfully loads a local dashboard on your LAN, proving internal traffic still works flawlessly.

🎛️ The Control
With a simple toggle:

ufw disable → restores full internet access
ufw enable → locks everything down again

Curious to see it all in action? Watch the full high-resolution 75-second demo:

🔗 https://khadirullah.com/blog/block-internet-linux-apps/#watch-it-in-action-full-gui-demo

✨ A complete visual walkthrough of the interface, behavior, and control flow—from block to restore.

Summary

The GID-based approach (Options A–E) is a clean, elegant way to restrict app networking — and it's good enough for most personal use cases. If you want to stop Jellyfin from downloading metadata, or prevent a game from phoning home, it works perfectly.

But if you need real enforcement against users who know their way around Linux, the GID approach has a fundamental EGID bypass. For those cases, use UID matching (unbypassable for services), Firejail (easiest for desktop apps), or network namespaces (maximum control).

Tested on Debian 13 (Trixie) with UFW. Should work on any Debian/Ubuntu-based distro with kernel 4.x+.

DEV Community: Khadirullah Mohammad

Accidentally Pushed a Password? How to Scrub Secrets from Git History

⚠️ First Response: Invalidate and Backup Immediately

When NOT to rewrite history

Is it too late? (The Wayback Machine Anxiety)

Why Interactive Rebase Is Usually the Wrong Tool

The Right Way: Global Text Replacement

1. Create a Replacements File

2. Verify and Execute the Scrub

Bonus: Scrubbing Commit Messages

The Butterfly Effect (Hashes)

Advanced Tips and Common Pitfalls

1. The Exact Match Trap

2. Deep Search: -S vs -G

3. The "No Modified Files" Illusion

4. Git Submodules Are Separate

5. Where do the old commits go? (Garbage Collection)

The Final Step: Force Pushing

1. Reconnect the Remote

2. Verification (Diff)

3. The Force Push (All Branches and Tags!)

4. Notify Collaborators (Crucial!)

5. Rotate Dependent Secrets

After the Push: GitHub's Server Cache

Prevention: Stop It From Happening Again

The Ultimate Guide to Automating SVG Rasterization: Inkscape vs ImageMagick vs Headless Chrome

Phase 1: The Browser-Oriented SVG Test

1. ImageMagick (librsvg)

2. Inkscape (Command Line)

3. Headless Google Chrome

Gotcha: Chrome Bottom Cutoff

Why Inkscape and ImageMagick Break (Code Analysis)

Root Cause 1: rgba() CSS Colors

Root Cause 2: feDropShadow

Root Cause 3: Emoji and system-ui Fonts

Phase 2: The Strict SVG 1.1 Rematch

ImageMagick (Strict SVG 1.1)

Inkscape (Strict SVG 1.1)

The Ultimate Automation Script

Conclusion

Fixing Broken Hugo Open Graph Previews with WebP Social Fallbacks

1. Why WebP for Social Fallbacks?

Format Comparison for Open Graph Fallbacks

When to Use Which

2. The Hugo vs. Blowfish Conflict

3. The "Social Fallback" Solution

The Result

4. Verification Across Platforms

Summary Checklist

Git Submodule Silently Became Regular Files: How I Diagnosed and Fixed It

Quick Answer (TL;DR)

The Symptoms

1. No .git file inside the theme directory

2. git submodule status returned nothing

3. Git commands inside the theme directory operated on the parent repo

The Root Cause

Why deleting and re-adding files breaks a submodule

The Fix

Step 1: Remove the theme files from git tracking

Step 2: Clean cached submodule data

Step 3: Remove the old .gitmodules entry

Step 4: Stage and commit the removal

Step 5: Re-add as a proper submodule

Step 6: Commit the proper submodule

Step 7: Verify

Future Updates Now Just Work

Quick Diagnostic Checklist

Why This Matters

The "Gotcha": Cloudflare Pages Deployment Failure After Fixing

What happened

Diagnosing the failure

Step-by-step fix on Cloudflare Pages

Other deployment platforms

Cleaning Up the Old Theme Files From Git History

Summary

I Deleted Files From Git But They Were Still There: Removing Objects From Git History

Quick Answer (TL;DR)

Who should read this?

Which tool do I need?

Why git rm Doesn't Fix This

2. Deep Search: `-S` vs `-G`

Root Cause 1: `rgba()` CSS Colors

Root Cause 2: `feDropShadow`

Root Cause 3: Emoji and `system-ui` Fonts

1. No `.git` file inside the theme directory

2. `git submodule status` returned nothing

Step 3: Remove the old `.gitmodules` entry

Why `git rm` Doesn't Fix This

Use `git filter-repo --analyze` for automated reports

`git filter-branch` (the old way, deprecated)

`git filter-repo` (the modern way, recommended)

`.gitignore`

File 1: `user-data`

File 2: `meta-data`

Test 5: The `cloud-init.disabled` File ❌

Test 6: `systemctl disable` ❌

Test 7: `systemctl mask` ✅

Test 8: `lock_passwd: false` in user-data ❌