DEV Community: Rez Moss

Beyond Vulnerability Scanning, How SBOM Diff Exposes Shadow Dependencies in Your Supply Chain

Rez Moss — Wed, 04 Feb 2026 22:16:19 +0000

Log4Shell in December 2021 was patched within days. Finding where Log4j existed in production took much longer.

Most of affected Java projects had Log4j as an indirect dependency bundled with something else, not chosen directly. 80% of those had it five or more levels deep in their dependency tree. Security teams spent the holidays grepping through JARs/build and running recursive searches across hundreds of repos, trying to answer a question that should have been trivial: what's in our software?

The patch was available. The problem was inventory

Transitive Dependencies

You add express to a Node.js project. Express needs body-parserBody-parser needs raw-body m Raw-body needs unpipe One package becomes a tree.

Direct dependencies: What you put in package.json
Transitive dependencies: Everything those packages pull in

An Ivanti report found over 90% of dependencies in modern apps are indirect. The ratio varies by ecosystem, Maven amplifies dependencies more than npm, which amplifies more than PyPI,but the pattern holds,youe direct dependencies might number in the dozens. Your full dependency tree often numbers in the hundreds.

Each package in that tree has its own maintainers, release practices, and security posture. Some are maintained by teams at major companies. Others are side projects that haven't seen a commit in three years. From the perspective of your running application, they're all equally trusted.

The event-stream attack in 2018 exploited this. A maintainer handed off a popular npm package to someone who offered to help. The new maintainer added flatmap-stream as a dependency, that package contained obfuscated code targeting Copay bitcoin wallets, two millions weekly downloads continued for months before anyone noticed

t malicious code wasnt in event-stream itself, It was one level deeper, hidden in a transitive dependency that nobody was watching.

Why I Built This?

Package managers install dependencies, They resolve version conflicts and download packages. They dont tell you what changed between yesterday and today. Running npm install twice produces the same result, but you cant easily compare Tuesdays dependency tree to Monday's

Vulnerability scanners find known CVEs, They match package versions against databases of disclosed vulnerabilities. They cant tell you about packages that arent vulnerable yet, packages that might become the next Log4j. By the time a CVE exists, youre in reactive mode.

Lock files pin versions. They ensure reproducible builds and prevent unexpected updates. They dont help you understand what 47 new packages just entered your codebase when someone added a new dependency, The lock file diff shows hundreds of lines of JSON..

Good luck reviewing that in a PR

Whats missing is comparison, generate a snapshot of your dependencies today. Generate another tomorrow. Diff them. See exactly what changed: new packages, removed packages, version updates, license changes

Thats what SBOMs enable. A Software Bill of Materials lists every component in your software, direct and transitive, with versions, licenses, and hashes. Compare two SBOMs and you get a clear picture of what's different

I wanted a tool to diff SBOMs, flag suspicious changes, and enforce policies in CI. I didnt find one that worked the way I wanted, so I built sbomlyze

Setup

# Install to ./bin
curl -sSfL https://raw.githubusercontent.com/rezmoss/sbomlyze/main/install.sh | sh

# Install to /usr/local/bin (requires sudo)
curl -sSfL https://raw.githubusercontent.com/rezmoss/sbomlyze/main/install.sh | sudo sh -s -- -b /usr/local/bin

# Install specific version
curl -sSfL https://raw.githubusercontent.com/rezmoss/sbomlyze/main/install.sh | sh -s -- -v 0.2.0

$ sbomlyze --version
sbomlyze v0.2.3

You'll need an SBOM generator. I use Syft, which supports most package ecosystems and container images (they’re best)

curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin

Syft outputs CycloneDX or SPDX formats. sbomlyze reads both.

Example: Adding Express

Heres a minimal example showing what happens when you add a single dependency.

V1 - lodash only:

package.json
---------------
{
  "name": "test-app",
  "version": "1.0.0",
  "dependencies": {
    "lodash": "^4.17.21"
  }
}
--------------

npm install
syft . -o cyclonedx-json > ../v1-sbom.json

V2 - add express:

package.json
---------------
{
  "name": "test-app",
  "version": "1.0.0",
  "dependencies": {
    "lodash": "^4.17.21",
    "express": "^4.18.2"
  }
}
---------------

npm install
syft . -o cyclonedx-json > ../v2-sbom.json

Diff:

sbomlyze v1-sbom.json v2-sbom.json

Output:

📊 Drift Summary:
  ⚠️  Integrity drift: 1 components (hash changed without version change!)

+ Added (68):
  + accepts 1.3.8
  + array-flatten 1.1.1
  + body-parser 1.20.4
  + bytes 3.1.2
  + call-bind-apply-helpers 1.0.2
  + call-bound 1.0.4
  + content-disposition 0.5.4
  + content-type 1.0.5
  + cookie 0.7.2
  + cookie-signature 1.0.7
  + debug 2.6.9
  + depd 2.0.0
  + destroy 1.2.0
  + dunder-proto 1.0.1
  + ee-first 1.1.1
  + encodeurl 2.0.0
  + es-define-property 1.0.1
  + es-errors 1.3.0
  + es-object-atoms 1.1.1
  + escape-html 1.0.3
  + etag 1.8.1
  + express 4.22.1
  + finalhandler 1.3.2
  + forwarded 0.2.0
  + fresh 0.5.2
  + function-bind 1.1.2
  + get-intrinsic 1.3.0
  + get-proto 1.0.1
  + gopd 1.2.0
  + has-symbols 1.1.0
  + hasown 2.0.2
  + http-errors 2.0.1
  + iconv-lite 0.4.24
  + inherits 2.0.4
  + ipaddr.js 1.9.1
  + math-intrinsics 1.1.0
  + media-typer 0.3.0
  + merge-descriptors 1.0.3
  + methods 1.1.2
  + mime 1.6.0
  + mime-db 1.52.0
  + mime-types 2.1.35
  + ms 2.0.0
  + negotiator 0.6.3
  + object-inspect 1.13.4
  + on-finished 2.4.1
  + parseurl 1.3.3
  + path-to-regexp 0.1.12
  + proxy-addr 2.0.7
  + qs 6.14.1
  + range-parser 1.2.1
  + raw-body 2.5.3
  + safe-buffer 5.2.1
  + safer-buffer 2.1.2
  + send 0.19.2
  + serve-static 1.16.3
  + setprototypeof 1.2.0
  + side-channel 1.1.0
  + side-channel-list 1.0.0
  + side-channel-map 1.0.1
  + side-channel-weakmap 1.0.2
  + statuses 2.0.2
  + toidentifier 1.0.1
  + type-is 1.6.18
  + unpipe 1.0.0
  + utils-merge 1.0.1
  + vary 1.1.2

! Duplicates in second SBOM (1):
  ! ms: [2.0.0 2.1.3]

One line change in package.json added 68 packages,the project went from 3 components to 71

In a PR t diff shows "added express to dependencies" The 67 transitive packages dont show up anywhere in the code review, theyre invisible unless you generate SBOMs and compare them

This is typical. Most frameworks pull in dozens of transitive dependencies. React, Angular, Django, Rails,they all have deep dependency trees. Every one of those packages runs in production with the same privileges as your own code.

Understanding the Output

Added components lists every new package in your dependency tree, each one now runs in production. Some youve heard of (body-parser, cookie). Others you havent (dunder-proto, gopd, es-object-atoms). They all have equal access to your applications memory, network, and filesystem.

Duplicates flag packages that appear multiple times with different versions, in this output, ms shows up as both 2.0.0 and 2.1.3. Different parts of Express need different versions, and npm installs both. This works at runtime, but creates complications. If ms gets a CVE, you need to trace through your dependency tree to find every path that pulls it in, then update the packages along those paths. With one version, thats straightforward. With multiple versions nested at different depths, it gets tedious

Integrity drift catches components where the hash changed but the version stayed the same. This can happen legitimately a package maintainer rebuilds and republishes without bumping the version, it can also indicate tampering. Someone compromises a registry account, pushes modified code under an existing version number. The version looks unchanged, but the code is different. Integrity drift flags this for investigation

Policy Enforcement

You can define rules and fail builds that violate them. Policies let you codify organizational standards and enforce them automatically

policy.json

{
  "max_added": 5,
  "max_removed": 3,
  "max_changed": 10,
  "deny_licenses": ["GPL-3.0", "AGPL-3.0"],
  "require_licenses": false,
  "deny_duplicates": true
}

sbomlyze v1-sbom.json v2-sbom.json --policy policy.json

Output when violated

❌ Policy Errors (2):
  [max_added] too many components added: 68 > 5
  [deny_duplicates] found 1 duplicate components in result

Exit code 1 fails CI

Field	Function
`max_added`	Cap on new packages
`max_removed`	Cap on removed packages
`max_changed`	Cap on version changes
`deny_licenses`	Block specific licenses
`require_licenses`	Require license info
`deny_duplicates`	Fail on duplicate versions

The thresholds depend on your codebase, A greenfield project might set max_added: 10, A established codebase with lockeddown dependencies might set max_added: 3, start loose, observe what normal PRs look like, then tighten

License blocking matters for compliance. Some organizations cant ship GPL code in proprietary products, some cant use AGPL in SaaS applications. The policy catches these before merge rather than during legal review months later.

Duplicate blocking is stricter,many legitimate dependency trees have duplicates. You might start with this disabled and enable it later as you clean up your dependency tree.

CI Integration

Here's a GitHub Actions workflow that generates SBOMs for both the base branch and the PR branch, then diffs them

name: SBOM Diff

on:
  pull_request:
    branches: [main]

jobs:
  sbom-diff:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0

      - name: Setup Go
        uses: actions/setup-go@v5
        with:
          go-version: '1.21'

      - name: Install tools
        run: |
          go install github.com/rezmoss/sbomlyze/cmd/sbomlyze@latest
          curl -sSfL https://raw.githubusercontent.com/anchore/syft/main/install.sh | sh -s -- -b /usr/local/bin

      - name: Generate baseline SBOM
        run: |
          git checkout origin/main
          npm ci
          syft . -o cyclonedx-json > baseline.json

      - name: Generate PR SBOM
        run: |
          git checkout ${{ github.head_ref }}
          npm ci
          syft . -o cyclonedx-json > pr.json

      - name: Diff
        run: sbomlyze baseline.json pr.json --policy policy.json

The workflow checks out both branches, generates an SBOM for each, and compares them,if the policy fails, the PR fails Developers see exactly which packages changed and why the build broke

You can also run this without a policy file to get visibility without enforcement,the diff output appears in the CI logs. Reviewers can check it alongside the code diff

JSON Output

For scripting and custom tooling, sbomlyze outputs JSON:

sbomlyze v1-sbom.json v2-sbom.json --json > diff.json

jq '.diff.added | length' diff.json
jq '.diff.added[].name' diff.json
jq '.diff.duplicates' diff.json

You can build custom checks on top of this,flag pkgs from specific maintainers. Alert on packages with low download counts. Cross reference against an internal allowlist. The JSON gives you the data; you write the logic

Container Images

Container images contain more than your application code, the base image brings OS packages libc, openssl, coreutils. Your application adds its runtime and dependencies, a typical container has hundreds of components from multiple ecosystems

Syft scans container images directly

syft nginx:1.24-alpine -o cyclonedx-json > nginx-1.24.json
syft nginx:1.27-alpine -o cyclonedx-json > nginx-1.27.json
sbomlyze nginx-1.24.json nginx-1.27.json

nginx:1.24-alpine has 1,393 components. nginx:1.27-alpine has 1,047,the ver bunp changed hundreds of packages, Alpine base image updates, library upgrades, removed pkgs

These changes are invisible if you only track application dependencies,you Dockerfile says FROM nginx:1.27-alpine,the diff shows one line changed, the actual change to your deployed software is hundreds of pkgs.

sbomlyze works at the SBOM level, so it handles whatever Syft detects OS packages, application dependencies, binaries, configuration files, all in one diff

Single SBOM Stats

When you want an overview of one SBOM without comparison

sbomlyze v2-sbom.json

📦 SBOM Statistics
==================

Total Components: 71

By Package Type:
  npm          70
  unknown      1

Licenses:
  With license:    69
  Without license: 2

  Top Licenses:
    MIT                            66
    ISC                            2
    BSD-3-Clause                   1

Integrity:
  With hashes:    1
  Without hashes: 70

⚠️  Duplicates Found: 1
  ms: [2.0.0 2.1.3]

High "without license" counts suggest packages that may be unmaintained or poorly documented, legitimate pkgs usually have license files, missing licenses can also indicate compliance risk, youre shipping code with unclear legal status.

Low hash coverage limits your ability to verify integrity. Without hashes, you cant detect if a package was modified after it was published

Unexpected package types are worth investigating. If your Node.js application shows Python packages, something pulled them in. Maybe a build tool, maybe a testing framework, maybe something that shouldn't be there.

Recommendations

Store SBOMs for each release, you need baselines to diff against. Include SBOM generation in your release pipeline. Archive them alongside your build artifacts

syft . -o cyclonedx-json > sbom-v${VERSION}.json

When an incident happens, you can compare the affected version against previous versions,you can answer "when did this package enter our dependency tree?" without recreating old builds.

Diff on every PR Even without blocking, visibility helps, reviewers can see the full impact of dependency changes. "Added axios" becomes "added axios and 12 transitive dependencies" That context changes the review

Document big additions, When you approve Express (68 packages), write down why link to the security review, note who approved it, six months later when someone asks about the dependency, the answer exists

Combine with vulnerability scanning SBOM diffing shows what changed. Vulnerability scanning shows whats vulnerable. Different questions, both useful. Diffing catches new dependencies before they accumulate CVEs. Scanning catches existing dependencies when CVEs are disclosed.

Review quarterly Look through your dependency tree. Research packages you dont recognize. Check maintenance status,last commit date, open issues, bus factor. Identify candidates for replacement or removal.

Summary

# Generate
syft . -o cyclonedx-json > sbom.json

# Compare
sbomlyze baseline.json current.json

# Enforce
sbomlyze baseline.json current.json --policy policy.json

A Note on Tooling

sbomlyze is something I built for my own daytoday workflow, plenty of other SBOM and dependency analysis tools exist, some may fit your needs better, some worse. I’m not claiming nothing else works or that this is the only solution. I looked around, didn't find something that matched how I wanted to work, and built my own. If its useful to you, great. If you find something else that works better for your situation then use that.

If youre deeper into this space than I am, or you think theres a better approach, I'd genuinely like to hear it, open an issue, send a PR, or just tell me I'm wrong. I'm not attached to being right, I'm attached to solving the problem. If someone points me to a tool that does this better, I'll use it and recommend it.

Repo : https://github.com/rezmoss/sbomlyze

How to Compare Container SBOMs and Detect Drift Between Image Versions

Rez Moss — Fri, 09 Jan 2026 15:27:25 +0000

Every time you pull a container image, you’re trusting hundreds of packages you didn’t explicitly choose. The base image brings its OS packages, your dependencies bring their dependencies, and somewhere in that stack is the next CVE waiting to be discovered.

An SBOM (Software Bill of Materials) inventories everything inside a container. But a single snapshot only tells you what’s there now. The real value comes from comparing them: what changed? What got added? What got removed?

I went looking for a tool that could diff SBOMs, flag integrity drift, enforce policies, and slot into CI without friction. Found plenty of SBOM generators, but nothing that handled comparison the way I needed. So I built sbomlyze for myself. Turns out others had the same gap.

This post covers using sbomlyze to answer those questions,from basic diffs to drift detection, policy enforcement, and CI integration.

Read article here https://rezmoss.com/blog/compare-container-sbom-detect-drift-image-versions/

Basic Conversions - Atoi and Itoa 1/7

Rez Moss — Sat, 16 Aug 2025 15:27:25 +0000

Converting between strings and integers is a fundamental operation in Go programming. Whether you're parsing user input, reading configuration files, or handling API responses, you'll frequently need to transform string representations of numbers into integers and vice versa. Go's strconv package provides two essential functions for these conversions: Atoi for string-to-integer conversion and Itoa for integer-to-string conversion.

These functions handle the heavy lifting of conversion while providing proper error handling mechanisms. Understanding how they work and when they fail will make your code more robust and help you avoid common pitfalls that can crash your applications or produce unexpected results.

Let's dive into how these functions work, explore their edge cases, and establish patterns for handling conversion errors effectively.

Understanding Atoi: String to Integer Conversion

The Atoi function converts a string representation of a number into an integer. Its signature is straightforward: func Atoi(s string) (int, error). The function returns both the converted integer and an error value, following Go's explicit error handling pattern.

Here's a basic example:

package main

import (
    "fmt"
    "strconv"
)

func main() {
    // Simple conversion
    str := "42"
    num, err := strconv.Atoi(str)
    if err != nil {
        fmt.Printf("Conversion failed: %v\n", err)
        return
    }
    fmt.Printf("Converted %s to %d\n", str, num)
}

Atoi handles positive and negative integers seamlessly:

// Positive and negative numbers
examples := []string{"123", "-456", "0", "+789"}

for _, s := range examples {
    if num, err := strconv.Atoi(s); err == nil {
        fmt.Printf("%s -> %d\n", s, num)
    }
}

strconv.Atoi does not trim leading or trailing whitespace. Any surrounding or internal whitespace causes a syntax error. If you expect whitespace, trim it yourself (e.g., with strings.TrimSpace) before calling Atoi:

// Whitespace handling
s := "  42  "
// Atoi fails with whitespace, so we must trim it first.
num, err := strconv.Atoi(strings.TrimSpace(s))
fmt.Printf("'%s' -> %d, %v\n", s, num, err) // "  42  " -> 42, nil

// Internal whitespace will still fail
num, err = strconv.Atoi("4 2")
fmt.Printf("'4 2' -> %d, %v\n", num, err) // '4 2' -> 0, strconv.Atoi: parsing "4 2": invalid syntax

When working with user input or external data, always check the error return value. Atoi will fail on any string that doesn't represent a valid integer, including empty strings, non-numeric characters, or numbers outside the integer range.

Atoi Edge Cases and Limitations

Understanding where Atoi breaks down is crucial for writing reliable code. The function has several edge cases that can catch developers off guard if not properly handled.

Empty and Invalid Strings

Atoi cannot convert empty strings or strings containing non-numeric characters:

// These will all fail
invalidInputs := []string{
    "",           // Empty string
    "abc",        // Letters
    "12.34",      // Not valid: Atoi only supports base-10 integers, not floats
    "1,234",      // Numbers with commas
    "0x1A",       // Hexadecimal
    "1e5",        // Scientific notation
    "1_234",      // Underscores not allowed in Atoi (only in ParseInt with base 0)
}

for _, input := range invalidInputs {
    _, err := strconv.Atoi(input)
    if err != nil {
        fmt.Printf("'%s' failed: %v\n", input, err)
    }
}

Integer Overflow

Atoi returns an int, whose size depends on your platform (32 bits on 32-bit systems, 64 bits on 64-bit systems). If the string represents a number outside this range, Atoi does not silently overflow—it returns a strconv.ErrRange error.

// This will fail on all systems (value exceeds int64 max, so Atoi returns ErrRange)
// Note: On overflow Atoi returns ErrRange and the value is clamped to int’s limits.
largeNumber := "9223372036854775808" 
_, err := strconv.Atoi(largeNumber)
if err != nil {
    fmt.Printf("Overflow error: %v\n", err)
}


// Fails ONLY on 32-bit (but OK on 64-bit)
_, err = strconv.Atoi("2147483648")           // ErrRange on 32-bit, OK on 64-bit

Leading Zeros and Signs

Atoi handles leading zeros and explicit positive signs correctly, but there are nuances:

// These work as expected
fmt.Println(strconv.Atoi("007"))    // 7, nil
fmt.Println(strconv.Atoi("+42"))    // 42, nil
fmt.Println(strconv.Atoi("-0"))     // 0, nil

// Multiple signs fail
fmt.Println(strconv.Atoi("++42"))   // 0, error
fmt.Println(strconv.Atoi("--42"))   // 0, error

Whitespace Behavior

Atoi does not trim whitespace. Any whitespace must be removed before parsing:

// Whitespace edge cases
// Fails due to leading/trailing whitespace
fmt.Println(strconv.Atoi("\t42\n"))  // 0, error

// Correct way is to trim first
fmt.Println(strconv.Atoi(strings.TrimSpace("\t42\n"))) // 42, nil

// Internal whitespace always fails
fmt.Println(strconv.Atoi("4\t2"))    // 0, error

// Space between sign and number fails
fmt.Println(strconv.Atoi("- 42"))    // 0, error

These edge cases highlight why error checking is essential when using Atoi in production code.

Understanding Itoa: Integer to String Conversion

The Itoa function provides the reverse operation of Atoi, converting integers to their string representation. Unlike Atoi, Itoa has a simpler signature: func Itoa(i int) string. Notice there's no error return value—integer to string conversion cannot fail.

Here's basic usage:

package main

import (
    "fmt"
    "strconv"
)

func main() {
    // Simple conversion
    num := 42
    str := strconv.Itoa(num)
    fmt.Printf("Converted %d to '%s'\n", num, str)
}

Itoa handles the full range of integer values, including negative numbers and zero:

// Various integer types
numbers := []int{0, -1, 42, -999, 2147483647, -2147483648}

for _, num := range numbers {
    str := strconv.Itoa(num)
    fmt.Printf("%d -> '%s'\n", num, str)
}

The function produces clean output without unnecessary formatting. It always returns the plain base-10 string form of the integer (no leading zeros, no explicit + for positives, no formatting symbols)

// Clean output examples
fmt.Println(strconv.Itoa(42))     // "42"
fmt.Println(strconv.Itoa(-42))    // "-42"
fmt.Println(strconv.Itoa(0))      // "0"
fmt.Println(strconv.Itoa(7))      // "7"

Itoa is particularly useful when you need to concatenate numbers with strings or prepare data for output:

// Practical usage
userID := 1234
message := "User " + strconv.Itoa(userID) + " logged in"
fmt.Println(message) // "User 1234 logged in"

// Building file names
fileNum := 5
filename := "data_" + strconv.Itoa(fileNum) + ".txt"
fmt.Println(filename) // "data_5.txt"

Since Itoa cannot fail, you don't need to worry about error handling when converting integers to strings. This makes it straightforward to use in any context where you need a string representation of a number.

Error Handling Patterns

When working with Atoi, proper error handling is essential for building robust applications. The errors returned by Atoi provide specific information about what went wrong, allowing you to respond appropriately to different failure scenarios.

Basic Error Checking Pattern

The most common pattern is immediate error checking after conversion:

func parseUserInput(input string) (int, error) {
    num, err := strconv.Atoi(input)
    if err != nil {
        return 0, fmt.Errorf("invalid number format: %w", err)
    }
    return num, nil
}

// Usage
if result, err := parseUserInput("123abc"); err != nil {
    log.Printf("Parse error: %v", err)
    // Handle error appropriately
} else {
    fmt.Printf("Successfully parsed: %d\n", result)
}

Providing Default Values

Sometimes you want to provide a fallback value when conversion fails:

func parseWithDefault(s string, defaultValue int) int {
    if num, err := strconv.Atoi(s); err == nil {
        return num
    }
    return defaultValue
}

// Usage examples
port := parseWithDefault(os.Getenv("PORT"), 8080)
timeout := parseWithDefault(config.Timeout, 30)

Validating Input Ranges

Combine conversion with validation to ensure numbers fall within acceptable ranges:

func parseAge(s string) (int, error) {
    age, err := strconv.Atoi(s)
    if err != nil {
        return 0, fmt.Errorf("age must be a number: %w", err)
    }
    if age < 0 || age > 150 {
        return 0, fmt.Errorf("age must be between 0 and 150, got %d", age)
    }
    return age, nil
}

Batch Processing with Error Collection

When processing multiple values, collect errors rather than stopping on the first failure:

func parseNumbers(inputs []string) ([]int, []error) {
    var results []int
    var errors []error

    for i, input := range inputs {
        if num, err := strconv.Atoi(input); err != nil {
            errors = append(errors, fmt.Errorf("index %d: %w", i, err))
        } else {
            results = append(results, num)
        }
    }

    return results, errors
}

Type-Specific Error Messages

Examine the specific error type to provide more helpful feedback:

func parseWithDetailedError(s string) (int, error) {
    num, err := strconv.Atoi(s)
    if err != nil {
        if numErr, ok := err.(*strconv.NumError); ok {
            switch numErr.Err {
            case strconv.ErrSyntax:
                return 0, fmt.Errorf("'%s' is not a valid number format", s)
            case strconv.ErrRange:
                return 0, fmt.Errorf("'%s' is too large to fit in an integer", s)
            }
        }
        return 0, fmt.Errorf("conversion failed: %w", err)
    }
    return num, nil
}

These patterns help you handle conversion failures gracefully while providing meaningful feedback to users and maintaining application stability.

When Conversion Fails

Understanding the specific scenarios where Atoi fails helps you anticipate problems and design better error handling strategies. The function returns a *strconv.NumError that contains detailed information about what went wrong.

Syntax Errors

The most common failure occurs when the string doesn't represent a valid number format:

func demonstrateSyntaxErrors() {
    syntaxErrors := []string{
        "hello",      // Non-numeric characters
        "12.34",      // Decimal point
        "1,234",      // Comma separator
        "12abc",      // Mixed numeric/alphabetic
        "0xFF",       // Hexadecimal notation
        "1e10",       // Scientific notation
        "",           // Empty string
    }

    for _, input := range syntaxErrors {
        _, err := strconv.Atoi(input)
        if err != nil {
            fmt.Printf("'%s': %v\n", input, err)
        }
    }
}

Range Errors

Numbers that exceed the integer limits for your platform will trigger range errors:

func demonstrateRangeErrors() {
    // These will cause range errors on most systems
    largeNumbers := []string{
        "9223372036854775808",  // Exceeds int64 max
        "-9223372036854775809", // Exceeds int64 min
    }

    for _, input := range largeNumbers {
        _, err := strconv.Atoi(input)
        if err != nil {
            if numErr, ok := err.(*strconv.NumError); ok {
                if numErr.Err == strconv.ErrRange {
                    fmt.Printf("Range error for '%s': number too large\n", input)
                }
            }
        }
    }
}

Recovery Strategies

When conversions fail, you have several options for recovery:

// Strategy 1: Use alternative parsing functions
func parseFlexible(s string) (int, error) {
    if num, err := strconv.Atoi(s); err == nil {
        return num, nil
    }

    f, err := strconv.ParseFloat(s, 64)
    if err != nil {
        return 0, fmt.Errorf("cannot parse %q: %w", s, err)
    }

    if math.IsNaN(f) || math.IsInf(f, 0) || f < float64(math.MinInt) || f > float64(math.MaxInt) {
        return 0, fmt.Errorf("cannot represent %q as int", s)
    }

    return int(f), nil // Safe: finite and in range
}


// Strategy 2: Clean input before parsing
func parseWithCleaning(s string) (int, error) {
    // Trim leading and trailing whitespace, which Atoi does not handle.
    cleaned := strings.TrimSpace(s)

    // Note: Removing characters like commas is context-dependent and may not be
    // safe for international number formats. Use with caution.
    // For example, in the US, "1,234" is 1234, but in Germany, "1,234" is 1.234.
    cleaned = strings.ReplaceAll(cleaned, ",", "")

    return strconv.Atoi(cleaned)
}

// Strategy 3: Graceful degradation
func parseWithFallback(s string, fallback int) int {
    if num, err := strconv.Atoi(s); err == nil {
        return num
    }
    log.Printf("Failed to parse '%s', using fallback %d", s, fallback)
    return fallback
}

Production Error Handling

In production applications, log conversion failures for debugging while providing user-friendly error messages:

func handleConversionError(input string, err error) {
    // Log detailed error for developers
    log.Printf("Conversion failed: input='%s', error=%v", input, err)

    // Provide user-friendly feedback
    var userMessage string
    if numErr, ok := err.(*strconv.NumError); ok {
        switch numErr.Err {
        case strconv.ErrSyntax:
            userMessage = "Please enter a valid number"
        case strconv.ErrRange:
            userMessage = "Number is too large"
        default:
            userMessage = "Invalid input format"
        }
    } else {
        userMessage = "Unable to process input"
    }

    fmt.Printf("Error: %s\n", userMessage)
}

Understanding these failure modes and implementing appropriate recovery strategies makes your applications more resilient and user-friendly when dealing with unpredictable input data.

Testing File System Code: Mocking, Stubbing, and Test Patterns 9/9

Rez Moss — Sat, 02 Aug 2025 16:15:00 +0000

Testing Philosophy with File System Interfaces

File system code presents unique testing challenges that many developers struggle with. The traditional approach of writing tests that directly interact with the real file system creates brittle, slow tests that depend on external state. A better approach treats the file system as a dependency that can be abstracted, controlled, and tested in isolation.

Dependency Injection Benefits

The key to testable file system code lies in dependency injection. Instead of hardcoding calls to os.Open() or ioutil.ReadFile(), inject the file system interface as a dependency. This seemingly simple change transforms your code from untestable to highly testable.

// Problematic: Direct file system dependency
func ProcessConfigFile(path string) (*Config, error) {
    data, err := os.ReadFile(path)
    if err != nil {
        return nil, err
    }
    return parseConfig(data)
}

// Better: File system as injected dependency
func ProcessConfigFile(fsys fs.FS, path string) (*Config, error) {
    data, err := fs.ReadFile(fsys, path)
    if err != nil {
        return nil, err
    }
    return parseConfig(data)
}

This pattern allows you to pass different file system implementations during testing while maintaining the same interface in production code. The production code receives os.DirFS(".") while tests receive mock implementations.

Testable File System Operations

When designing file system operations for testability, focus on three core principles: interface segregation, explicit dependencies, and predictable behavior.

Interface segregation means depending only on the file system capabilities you actually need. If your function only reads files, depend on fs.FS rather than a broader interface that includes write operations. This makes your intentions clear and simplifies testing.

// Only needs read access
func LoadTemplate(fsys fs.FS, name string) (*Template, error) {
    content, err := fs.ReadFile(fsys, name)
    if err != nil {
        return nil, fmt.Errorf("loading template %s: %w", name, err)
    }
    return parseTemplate(content)
}

// Needs write access - use different interface
func SaveReport(fsys WriteFS, path string, report *Report) error {
    data, err := report.Marshal()
    if err != nil {
        return err
    }
    return fsys.WriteFile(path, data, 0644)
}

Explicit dependencies eliminate hidden file system interactions. Every file system operation should flow through the injected interface, making the code's dependencies obvious to both humans and tests.

Isolation from Real File System

Test isolation prevents tests from interfering with each other and with the host system. When tests manipulate real files, they can leave behind state that affects subsequent test runs or, worse, modify important system files.

Proper isolation means your tests never touch the real file system unless specifically testing integration points. Unit tests work entirely with mock file systems, while integration tests use temporary directories that are cleaned up automatically.

func TestProcessConfigFile(t *testing.T) {
    // Isolated test using mock file system
    testFS := fstest.MapFS{
        "config.json": &fstest.MapFile{
            Data: []byte(`{"setting": "value"}`),
        },
    }

    config, err := ProcessConfigFile(testFS, "config.json")
    assert.NoError(t, err)
    assert.Equal(t, "value", config.Setting)
}

This approach gives you complete control over the file system state during tests. You define exactly what files exist, their contents, permissions, and modification times. Tests become deterministic and fast because they don't perform actual disk I/O.

The isolation also extends to error conditions. Mock file systems let you simulate permission errors, disk full conditions, and network failures that would be difficult or impossible to reproduce consistently with real file systems.

Testing file system code well requires thinking about the file system as just another dependency to be managed, not as an unchangeable part of the environment. This mindset shift enables the testing patterns and techniques that follow in the subsequent sections.

Using testing/fstest.TestFS

Go's testing/fstest package provides powerful tools for validating file system implementations and ensuring your code works correctly with any fs.FS implementation. The TestFS function acts as a comprehensive test suite that verifies file system behavior against the expected interface contracts.

Validating Custom FS Implementations

When you create custom file system implementations, fstest.TestFS becomes your verification tool. It runs a battery of tests that check edge cases, error conditions, and specification compliance that you might miss in manual testing.

func TestMyCustomFS(t *testing.T) {
    // Create your custom file system
    customFS := &MyCustomFS{
        files: map[string][]byte{
            "file1.txt":     []byte("content1"),
            "dir/file2.txt": []byte("content2"),
            "empty.txt":     []byte(""),
        },
    }

    // Validate it conforms to fs.FS interface expectations
    if err := fstest.TestFS(customFS, "file1.txt", "dir/file2.txt", "empty.txt"); err != nil {
        t.Fatal(err)
    }
}

The function tests various scenarios: opening files that exist and don't exist, reading directories, handling path separators correctly, and ensuring error messages follow Go's conventions. It catches subtle bugs that surface only in specific conditions.

For read-write file systems, create additional validation that goes beyond the basic fs.FS interface:

func TestWritableFS(t *testing.T) {
    fsys := NewMemoryFS()

    // First validate basic fs.FS compliance
    err := fstest.TestFS(fsys, "test1.txt", "dir/test2.txt")
    require.NoError(t, err)

    // Then test write operations
    testWriteOperations(t, fsys)
    testConcurrentAccess(t, fsys)
    testErrorConditions(t, fsys)
}

Test Coverage for Edge Cases

fstest.TestFS excels at uncovering edge cases that developers typically overlook. It tests paths with different separators, attempts to open directories as files, checks behavior with empty files, and validates that file info structures contain correct data.

The tool systematically tests path handling, which is notoriously error-prone:

// These are all tested automatically by fstest.TestFS
testCases := []string{
    "normal-file.txt",
    "path/to/nested/file.txt", 
    "file-with-dots..txt",
    "file.with.multiple.dots",
    "UPPERCASE.TXT",
    "file with spaces.txt",
}

It also validates that your file system handles the distinction between files and directories correctly, ensures that fs.FileInfo returns accurate information, and checks that error types match Go's standard patterns.

The edge case testing extends to boundary conditions. What happens when you try to read beyond a file's end? How does your implementation handle zero-length files? Does it correctly report file modification times? TestFS checks all of these scenarios.

Performance Testing Patterns

While fstest.TestFS focuses on correctness, performance testing requires additional patterns. Create benchmarks that measure your file system implementation under realistic workloads:

func BenchmarkFileSystemOperations(b *testing.B) {
    fsys := createLargeTestFS(1000) // 1000 files
    filenames := getRandomFilenames(fsys, 100)

    b.ResetTimer()
    b.RunParallel(func(pb *testing.PB) {
        for pb.Next() {
            name := filenames[rand.Intn(len(filenames))]
            if file, err := fsys.Open(name); err == nil {
                io.Copy(io.Discard, file)
                file.Close()
            }
        }
    })
}

Performance testing should cover different access patterns: sequential reads, random access, directory traversal, and concurrent operations. Each pattern stresses different aspects of your implementation.

Memory usage patterns matter as much as speed. Profile your file system implementation to ensure it doesn't leak memory or consume excessive resources:

func TestMemoryUsage(t *testing.T) {
    fsys := NewMemoryFS()

    // Load test data
    for i := 0; i < 1000; i++ {
        filename := fmt.Sprintf("file%d.txt", i)
        fsys.WriteFile(filename, make([]byte, 10240), 0644)
    }

    // Measure baseline memory
    var m1 runtime.MemStats
    runtime.GC()
    runtime.ReadMemStats(&m1)

    // Perform operations
    performRandomAccess(fsys, 10000)

    // Check for memory leaks
    var m2 runtime.MemStats
    runtime.GC()
    runtime.ReadMemStats(&m2)

    memGrowth := m2.Alloc - m1.Alloc
    if memGrowth > 1024*1024 { // 1MB threshold
        t.Errorf("Excessive memory growth: %d bytes", memGrowth)
    }
}

The combination of fstest.TestFS for correctness validation and custom benchmarks for performance creates a comprehensive testing strategy. Your file system implementation gains confidence through systematic verification while maintaining performance characteristics under realistic workloads.

Mock File System Implementations

Mock file systems give you complete control over the testing environment, allowing you to simulate any condition your code might encounter in production. Unlike stubs that return fixed responses, mocks actively track interactions and can change behavior based on the sequence of operations.

In-Memory Test File Systems

In-memory file systems provide the fastest and most controllable testing environment. They eliminate disk I/O completely while maintaining the same interface as real file systems. The fstest.MapFS type serves as an excellent starting point:

func TestFileProcessor(t *testing.T) {
    testFS := fstest.MapFS{
        "input/data.csv": &fstest.MapFile{
            Data:    []byte("name,age\nAlice,30\nBob,25"),
            Mode:    0644,
            ModTime: time.Date(2023, 1, 1, 12, 0, 0, 0, time.UTC),
        },
        "config/settings.json": &fstest.MapFile{
            Data: []byte(`{"delimiter": ",", "headers": true}`),
        },
    }

    processor := NewFileProcessor(testFS)
    result, err := processor.ProcessData("input/data.csv", "config/settings.json")

    assert.NoError(t, err)
    assert.Len(t, result.Records, 2)
    assert.Equal(t, "Alice", result.Records[0]["name"])
}

For more complex scenarios, create custom in-memory implementations that support write operations and state tracking:

type MemoryFS struct {
    files   map[string]*MemoryFile
    dirs    map[string]bool
    mutex   sync.RWMutex
    calls   []string // Track method calls for verification
}

func (m *MemoryFS) Open(name string) (fs.File, error) {
    m.mutex.Lock()
    m.calls = append(m.calls, fmt.Sprintf("Open(%s)", name))
    m.mutex.Unlock()

    m.mutex.RLock()
    defer m.mutex.RUnlock()

    if file, exists := m.files[name]; exists {
        return &MemoryFile{data: file.data, info: file.info}, nil
    }
    return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
}

func (m *MemoryFS) WriteFile(name string, data []byte, perm fs.FileMode) error {
    m.mutex.Lock()
    defer m.mutex.Unlock()

    m.calls = append(m.calls, fmt.Sprintf("WriteFile(%s, %d bytes)", name, len(data)))
    m.files[name] = &MemoryFile{
        data: append([]byte(nil), data...), // Copy to prevent external modification
        info: &MemoryFileInfo{name: name, size: int64(len(data)), mode: perm},
    }
    return nil
}

This approach lets you verify not just the final state but also the sequence of operations your code performs.

Controlled Error Injection

Mock file systems excel at simulating error conditions that are difficult to reproduce with real file systems. You can trigger specific errors at precise moments to test error handling paths:

type ErrorInjectingFS struct {
    base      fs.FS
    failAfter int
    callCount int
    errorType error
}

func (e *ErrorInjectingFS) Open(name string) (fs.File, error) {
    e.callCount++
    if e.callCount > e.failAfter {
        return nil, e.errorType
    }
    return e.base.Open(name)
}

func TestErrorHandling(t *testing.T) {
    baseFS := fstest.MapFS{
        "file1.txt": &fstest.MapFile{Data: []byte("content1")},
        "file2.txt": &fstest.MapFile{Data: []byte("content2")},
        "file3.txt": &fstest.MapFile{Data: []byte("content3")},
    }

    // Simulate disk full error after 2 successful operations
    errorFS := &ErrorInjectingFS{
        base:      baseFS,
        failAfter: 2,
        errorType: errors.New("no space left on device"),
    }

    processor := NewBatchProcessor(errorFS)
    files := []string{"file1.txt", "file2.txt", "file3.txt"}

    results, err := processor.ProcessFiles(files)

    // Should process first 2 files successfully, then fail
    assert.Error(t, err)
    assert.Len(t, results, 2)
    assert.Contains(t, err.Error(), "no space left on device")
}

Error injection patterns help you test partial failure scenarios, retry logic, and cleanup operations. You can simulate network timeouts, permission errors, corrupted files, and resource exhaustion.

State Verification Techniques

Mock file systems enable detailed verification of how your code interacts with the file system. Track not just what files were accessed, but when, in what order, and with what parameters:

type TrackedFS struct {
    base       fs.FS
    operations []Operation
    mutex      sync.Mutex
}

type Operation struct {
    Type      string
    Path      string
    Timestamp time.Time
    Thread    int
}

func (t *TrackedFS) Open(name string) (fs.File, error) {
    t.mutex.Lock()
    t.operations = append(t.operations, Operation{
        Type:      "Open",
        Path:      name,
        Timestamp: time.Now(),
        Thread:    getCurrentThreadID(),
    })
    t.mutex.Unlock()

    return t.base.Open(name)
}

func TestFileAccessPattern(t *testing.T) {
    tracked := &TrackedFS{base: createTestFS()}
    processor := NewConcurrentProcessor(tracked)

    processor.ProcessDirectory("test-dir")

    // Verify operations occurred in expected order
    ops := tracked.GetOperations()
    assert.Equal(t, "Open", ops[0].Type)
    assert.Equal(t, "test-dir", ops[0].Path)

    // Verify no concurrent access to same file
    fileAccess := groupOperationsByFile(ops)
    for file, accesses := range fileAccess {
        ensureNoOverlap(t, file, accesses)
    }
}

State verification extends beyond simple call tracking. You can verify that files are closed properly, that locks are acquired and released correctly, and that temporary files are cleaned up. Mock file systems give you visibility into aspects of your code's behavior that would be invisible with real file systems.

The power of mock file systems lies in their predictability and observability. Every aspect of the file system's behavior is under your control, and every interaction is visible to your tests. This combination enables thorough testing of complex file system operations that would be brittle or impossible to test otherwise.

Test Data Management

Effective test data management determines whether your file system tests remain maintainable as your codebase grows. Poor test data practices lead to flaky tests, unclear failure messages, and maintenance nightmares. Well-organized test data makes tests self-documenting and reliable.

Embedding Test Fixtures

Go's embed directive provides an elegant solution for including test files directly in your binary. This approach ensures test data travels with your code and eliminates path-related issues across different environments:

//go:embed testdata/*
var testDataFS embed.FS

func TestConfigParser(t *testing.T) {
    tests := []struct {
        name     string
        filename string
        expected Config
        wantErr  bool
    }{
        {
            name:     "valid config",
            filename: "testdata/valid-config.json",
            expected: Config{Host: "localhost", Port: 8080},
        },
        {
            name:     "invalid json",
            filename: "testdata/invalid.json",
            wantErr:  true,
        },
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            config, err := ParseConfig(testDataFS, tt.filename)
            if tt.wantErr {
                assert.Error(t, err)
                return
            }
            assert.NoError(t, err)
            assert.Equal(t, tt.expected, *config)
        })
    }
}

Organize embedded test data in a clear directory structure that mirrors your test cases:

testdata/
├── configs/
│   ├── valid-config.json
│   ├── invalid-syntax.json
│   └── missing-required-field.json
├── templates/
│   ├── simple.tmpl
│   └── complex-with-partials.tmpl
└── datasets/
    ├── small-sample.csv
    └── large-sample.csv

This structure makes it obvious which test data belongs to which test scenario. File names should be descriptive enough that you understand their purpose without opening them.

For binary test data or files that change frequently, consider using a hybrid approach:

func loadTestData(t *testing.T, filename string) []byte {
    // Try embedded data first
    if data, err := testDataFS.ReadFile("testdata/" + filename); err == nil {
        return data
    }

    // Fall back to external file for development
    data, err := os.ReadFile(filepath.Join("testdata", filename))
    require.NoError(t, err, "failed to load test data: %s", filename)
    return data
}

Dynamic Test File Generation

Some tests require generated data rather than static fixtures. Dynamic generation is particularly useful for testing edge cases, large datasets, or scenarios with specific characteristics:

func generateTestCSV(rows int, corruptLine int) []byte {
    var buf bytes.Buffer
    buf.WriteString("id,name,email,age\n")

    for i := 1; i <= rows; i++ {
        if i == corruptLine {
            // Intentionally corrupt this line for error testing
            buf.WriteString(fmt.Sprintf("%d,\"Unclosed Quote,user%d@example.com,25\n", i, i))
        } else {
            buf.WriteString(fmt.Sprintf("%d,User%d,user%d@example.com,%d\n", i, i, i, 20+i%50))
        }
    }
    return buf.Bytes()
}

func TestCSVParserWithLargeFile(t *testing.T) {
    testFS := fstest.MapFS{
        "large.csv": &fstest.MapFile{
            Data: generateTestCSV(10000, 5000), // Corrupt line 5000
        },
    }

    parser := NewCSVParser(testFS)
    records, err := parser.ParseFile("large.csv")

    assert.Error(t, err)
    assert.Contains(t, err.Error(), "line 5000")
    assert.Len(t, records, 4999) // Should have parsed up to the error
}

Dynamic generation shines when testing boundary conditions. Generate files that are exactly at size limits, create directory structures with maximum depth, or produce data with specific statistical distributions:

func createDeepDirectoryStructure(depth int) fstest.MapFS {
    fs := make(fstest.MapFS)

    var path strings.Builder
    for i := 0; i < depth; i++ {
        if i > 0 {
            path.WriteString("/")
        }
        path.WriteString(fmt.Sprintf("dir%d", i))
    }

    // Add a file at the deepest level
    filePath := path.String() + "/deep-file.txt"
    fs[filePath] = &fstest.MapFile{
        Data: []byte(fmt.Sprintf("File at depth %d", depth)),
    }

    return fs
}

func TestDeepDirectoryHandling(t *testing.T) {
    maxDepth := 100
    testFS := createDeepDirectoryStructure(maxDepth)

    walker := NewDirectoryWalker(testFS)
    files, err := walker.FindAllFiles(".")

    assert.NoError(t, err)
    assert.Len(t, files, 1)
    assert.Contains(t, files[0], "deep-file.txt")
}

Cleanup Strategies

Test cleanup becomes critical when tests create temporary files or modify shared state. Implement cleanup strategies that work reliably even when tests fail or panic:

func TestFileProcessor(t *testing.T) {
    // Create temporary directory
    tempDir := t.TempDir() // Automatically cleaned up by testing framework

    // For more control, use custom cleanup
    customTempDir, err := os.MkdirTemp("", "test-processor-*")
    require.NoError(t, err)
    t.Cleanup(func() {
        os.RemoveAll(customTempDir)
    })

    // Test operations...
    processor := NewFileProcessor(os.DirFS(tempDir))
    result, err := processor.ProcessFiles([]string{"input.txt"})

    assert.NoError(t, err)
    assert.NotNil(t, result)
}

For tests that modify global state or shared resources, use cleanup functions that restore the original state:

func TestWithModifiedEnvironment(t *testing.T) {
    // Save original environment
    originalHome := os.Getenv("HOME")
    originalConfig := os.Getenv("CONFIG_PATH")

    t.Cleanup(func() {
        os.Setenv("HOME", originalHome)
        os.Setenv("CONFIG_PATH", originalConfig)
    })

    // Modify environment for test
    testHome := t.TempDir()
    os.Setenv("HOME", testHome)
    os.Setenv("CONFIG_PATH", filepath.Join(testHome, "config"))

    // Run test with modified environment
    config, err := LoadUserConfig()
    assert.NoError(t, err)
    assert.Equal(t, testHome, config.HomeDirectory)
}

Well-managed test data eliminates a major source of test brittleness. When tests fail, you want to debug your application logic, not struggle with missing files or corrupted test data. The patterns above ensure your test data remains an asset rather than a liability as your test suite grows.

Integration Testing Approaches

Integration tests verify that your file system code works correctly with real file systems and external dependencies. While unit tests with mocks validate logic in isolation, integration tests catch issues that emerge from the interaction between your code and the actual environment it will run in.

Temporary Directory Usage

Integration tests need isolated environments that don't interfere with the host system or other tests. Go's testing package provides t.TempDir() which creates temporary directories that are automatically cleaned up:

func TestFileBackupIntegration(t *testing.T) {
    // Create isolated test environment
    sourceDir := t.TempDir()
    backupDir := t.TempDir()

    // Set up test files
    testFiles := map[string]string{
        "document.txt":     "Important document content",
        "config/app.json":  `{"version": "1.0", "debug": true}`,
        "data/records.csv": "id,name,value\n1,test,100\n2,prod,200",
    }

    for path, content := range testFiles {
        fullPath := filepath.Join(sourceDir, path)
        err := os.MkdirAll(filepath.Dir(fullPath), 0755)
        require.NoError(t, err)
        err = os.WriteFile(fullPath, []byte(content), 0644)
        require.NoError(t, err)
    }

    // Test the backup operation
    backup := NewFileBackup()
    err := backup.CreateBackup(sourceDir, backupDir)
    require.NoError(t, err)

    // Verify backup contents
    for path, expectedContent := range testFiles {
        backupPath := filepath.Join(backupDir, path)
        actualContent, err := os.ReadFile(backupPath)
        require.NoError(t, err)
        assert.Equal(t, expectedContent, string(actualContent))
    }

    // Verify file permissions and metadata
    sourceInfo, err := os.Stat(filepath.Join(sourceDir, "document.txt"))
    require.NoError(t, err)
    backupInfo, err := os.Stat(filepath.Join(backupDir, "document.txt"))
    require.NoError(t, err)

    assert.Equal(t, sourceInfo.Mode(), backupInfo.Mode())
    assert.Equal(t, sourceInfo.Size(), backupInfo.Size())
}

Temporary directories provide true isolation while testing real file system behavior. This catches issues like path separator handling, permission problems, and filesystem-specific limitations that mocks cannot simulate.

Real File System Testing

Some functionality can only be properly tested against real file systems. File locking, atomic operations, and filesystem-specific features require integration testing:

func TestConcurrentFileAccess(t *testing.T) {
    if testing.Short() {
        t.Skip("Skipping integration test in short mode")
    }

    tempDir := t.TempDir()
    testFile := filepath.Join(tempDir, "shared.txt")

    // Create initial file
    err := os.WriteFile(testFile, []byte("initial content\n"), 0644)
    require.NoError(t, err)

    const numWriters = 5
    const writesPerWriter = 10

    var wg sync.WaitGroup
    errChan := make(chan error, numWriters)

    // Launch concurrent writers
    for i := 0; i < numWriters; i++ {
        wg.Add(1)
        go func(writerID int) {
            defer wg.Done()

            for j := 0; j < writesPerWriter; j++ {
                line := fmt.Sprintf("Writer %d, line %d\n", writerID, j)
                if err := appendToFile(testFile, line); err != nil {
                    errChan <- err
                    return
                }
                time.Sleep(time.Millisecond) // Small delay to increase contention
            }
        }(i)
    }

    wg.Wait()
    close(errChan)

    // Check for errors
    for err := range errChan {
        t.Errorf("Concurrent write error: %v", err)
    }

    // Verify file integrity
    content, err := os.ReadFile(testFile)
    require.NoError(t, err)

    lines := strings.Split(string(content), "\n")
    expectedLines := 1 + (numWriters * writesPerWriter) // Initial + all writes
    assert.Equal(t, expectedLines, len(lines))
}

func appendToFile(filename, content string) error {
    file, err := os.OpenFile(filename, os.O_APPEND|os.O_WRONLY, 0644)
    if err != nil {
        return err
    }
    defer file.Close()

    _, err = file.WriteString(content)
    return err
}

Real file system testing reveals race conditions, locking issues, and platform-specific behaviors that are impossible to catch with mocked implementations.

Cross-Platform Considerations

File system behavior varies significantly across operating systems. Integration tests should account for these differences or explicitly test platform-specific behavior:

func TestPathHandling(t *testing.T) {
    tempDir := t.TempDir()

    tests := []struct {
        name        string
        inputPath   string
        expectError bool
        skipOnOS    string
    }{
        {
            name:      "normal path",
            inputPath: "normal-file.txt",
        },
        {
            name:      "path with spaces",
            inputPath: "file with spaces.txt",
        },
        {
            name:      "long filename",
            inputPath: strings.Repeat("a", 255), // Max filename length on most filesystems
        },
        {
            name:        "invalid characters",
            inputPath:   "file<>|.txt",
            expectError: true,
            skipOnOS:    "linux", // Linux allows these characters
        },
        {
            name:      "case sensitivity test",
            inputPath: "CaseSensitive.TXT",
        },
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            if tt.skipOnOS == runtime.GOOS {
                t.Skipf("Skipping test on %s", runtime.GOOS)
            }

            filePath := filepath.Join(tempDir, tt.inputPath)
            err := os.WriteFile(filePath, []byte("test content"), 0644)

            if tt.expectError {
                assert.Error(t, err)
            } else {
                assert.NoError(t, err)

                // Verify file can be read back
                content, readErr := os.ReadFile(filePath)
                assert.NoError(t, readErr)
                assert.Equal(t, "test content", string(content))
            }
        })
    }
}

func TestSymlinkHandling(t *testing.T) {
    if runtime.GOOS == "windows" {
        t.Skip("Symlink test requires Unix-like system")
    }

    tempDir := t.TempDir()

    // Create target file
    targetFile := filepath.Join(tempDir, "target.txt")
    err := os.WriteFile(targetFile, []byte("target content"), 0644)
    require.NoError(t, err)

    // Create symlink
    linkFile := filepath.Join(tempDir, "link.txt")
    err = os.Symlink(targetFile, linkFile)
    require.NoError(t, err)

    // Test reading through symlink
    walker := NewDirectoryWalker()
    files, err := walker.ListFiles(tempDir)
    require.NoError(t, err)

    // Should find both target and link
    assert.Len(t, files, 2)

    // Verify link behavior
    linkInfo, err := os.Lstat(linkFile)
    require.NoError(t, err)
    assert.True(t, linkInfo.Mode()&os.ModeSymlink != 0)
}

Platform-specific tests help ensure your code works correctly across different environments. Use build tags or runtime checks to handle platform differences gracefully.

Integration tests bridge the gap between unit tests and production deployment. They catch environment-specific issues while maintaining enough control to be reliable and repeatable. The key is finding the right balance between real-world conditions and test stability.

Advanced Testing Patterns

Advanced testing patterns push beyond traditional unit and integration tests to discover edge cases and performance characteristics that conventional testing might miss. These techniques are particularly valuable for file system code, which often handles unpredictable inputs and operates under varying system conditions.

Property-Based Testing

Property-based testing generates random inputs to verify that your code maintains certain invariants regardless of the specific data it processes. For file system operations, this approach excels at finding edge cases in path handling, data processing, and error recovery:

func TestFilePathNormalization(t *testing.T) {
    property := func(pathComponents []string) bool {
        if len(pathComponents) == 0 {
            return true // Skip empty input
        }

        // Filter out empty components and those with null bytes
        validComponents := make([]string, 0, len(pathComponents))
        for _, component := range pathComponents {
            if component != "" && !strings.Contains(component, "\x00") {
                validComponents = append(validComponents, component)
            }
        }

        if len(validComponents) == 0 {
            return true
        }

        // Build path and normalize it
        rawPath := strings.Join(validComponents, "/")
        normalized := NormalizePath(rawPath)

        // Properties that should always hold:
        // 1. Normalized path should not contain double slashes
        if strings.Contains(normalized, "//") {
            t.Logf("Double slash found in normalized path: %s", normalized)
            return false
        }

        // 2. Should not end with slash unless it's root
        if len(normalized) > 1 && strings.HasSuffix(normalized, "/") {
            t.Logf("Unexpected trailing slash: %s", normalized)
            return false
        }

        // 3. Should be idempotent - normalizing twice gives same result
        if NormalizePath(normalized) != normalized {
            t.Logf("Not idempotent: %s != %s", normalized, NormalizePath(normalized))
            return false
        }

        return true
    }

    // Run property test with random inputs
    for i := 0; i < 1000; i++ {
        components := generateRandomPathComponents(rand.Intn(10) + 1)
        if !property(components) {
            t.Fatalf("Property violation with input: %v", components)
        }
    }
}

func generateRandomPathComponents(count int) []string {
    components := make([]string, count)
    chars := "abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789-_."

    for i := 0; i < count; i++ {
        length := rand.Intn(20) + 1
        component := make([]byte, length)
        for j := range component {
            component[j] = chars[rand.Intn(len(chars))]
        }
        components[i] = string(component)
    }
    return components
}

Property-based testing for file operations can verify data integrity across various transformations:

func TestFileCompressionRoundTrip(t *testing.T) {
    compressor := NewFileCompressor()

    for i := 0; i < 100; i++ {
        // Generate random data with various characteristics
        originalData := generateRandomData(rand.Intn(10000) + 1)

        // Compress and decompress
        compressed, err := compressor.Compress(originalData)
        require.NoError(t, err)

        decompressed, err := compressor.Decompress(compressed)
        require.NoError(t, err)

        // Property: round-trip should preserve data exactly
        assert.Equal(t, originalData, decompressed, 
            "Round-trip failed for data of length %d", len(originalData))

        // Property: compressed size should be reasonable
        ratio := float64(len(compressed)) / float64(len(originalData))
        assert.Less(t, ratio, 2.0, 
            "Compression ratio too high: %f for data length %d", ratio, len(originalData))
    }
}

func generateRandomData(size int) []byte {
    data := make([]byte, size)

    // Mix different patterns to test various compression scenarios
    switch rand.Intn(4) {
    case 0: // Random data
        rand.Read(data)
    case 1: // Repeated patterns
        pattern := []byte("ABCDEFGH")
        for i := range data {
            data[i] = pattern[i%len(pattern)]
        }
    case 2: // Mostly zeros with some random
        for i := 0; i < size/10; i++ {
            data[rand.Intn(size)] = byte(rand.Intn(256))
        }
    case 3: // Text-like data
        chars := "abcdefghijklmnopqrstuvwxyz      \n\n"
        for i := range data {
            data[i] = chars[rand.Intn(len(chars))]
        }
    }

    return data
}

Fuzzing File System Operations

Fuzzing automatically generates test inputs to find crashes, panics, and unexpected behaviors. Go's built-in fuzzing support makes it easy to fuzz file system operations:

func FuzzFilePathParser(f *testing.F) {
    // Seed corpus with known interesting cases
    f.Add("/path/to/file.txt")
    f.Add("../../../etc/passwd")
    f.Add("file with spaces.txt")
    f.Add("file\x00with\x00nulls")
    f.Add(strings.Repeat("a", 1000))

    f.Fuzz(func(t *testing.T, input string) {
        // The function should never panic, regardless of input
        defer func() {
            if r := recover(); r != nil {
                t.Errorf("ParseFilePath panicked with input %q: %v", input, r)
            }
        }()

        result, err := ParseFilePath(input)

        if err == nil {
            // If parsing succeeded, result should be valid
            if result.Dir == "" && result.Base == "" {
                t.Errorf("ParseFilePath returned empty result for valid input: %q", input)
            }

            // Reconstructed path should be equivalent to original
            reconstructed := filepath.Join(result.Dir, result.Base)
            if !pathsEquivalent(input, reconstructed) {
                t.Errorf("Path reconstruction mismatch: %q -> %q", input, reconstructed)
            }
        }
    })
}

func FuzzFileReader(f *testing.F) {
    f.Add([]byte("normal file content"))
    f.Add([]byte(""))
    f.Add(make([]byte, 10000)) // Large file
    f.Add([]byte("\x00\x01\x02\xff")) // Binary data

    f.Fuzz(func(t *testing.T, data []byte) {
        // Create temporary file with fuzz data
        tempFile := filepath.Join(t.TempDir(), "fuzz-test.dat")
        err := os.WriteFile(tempFile, data, 0644)
        require.NoError(t, err)

        // Test file reader with arbitrary data
        reader := NewFileReader()

        // Should handle any data without crashing
        content, err := reader.ReadFile(tempFile)
        if err != nil {
            // Error is acceptable, but should not panic
            return
        }

        // If read succeeded, content should match original
        assert.Equal(t, data, content)
    })
}

Performance Benchmarking

Performance benchmarks measure how your file system code behaves under different load conditions. Effective benchmarks test realistic scenarios and measure relevant metrics:

func BenchmarkFileOperations(b *testing.B) {
    tempDir := b.TempDir()

    // Test different file sizes
    fileSizes := []int{1024, 10240, 102400, 1024000} // 1KB to 1MB

    for _, size := range fileSizes {
        b.Run(fmt.Sprintf("size_%d", size), func(b *testing.B) {
            data := make([]byte, size)
            rand.Read(data)

            b.ResetTimer()
            b.ReportAllocs()

            for i := 0; i < b.N; i++ {
                filename := filepath.Join(tempDir, fmt.Sprintf("bench_%d.dat", i))

                // Measure write performance
                err := os.WriteFile(filename, data, 0644)
                if err != nil {
                    b.Fatal(err)
                }

                // Measure read performance
                _, err = os.ReadFile(filename)
                if err != nil {
                    b.Fatal(err)
                }

                // Clean up
                os.Remove(filename)
            }

            // Report throughput
            bytesPerOp := int64(size * 2) // Read + write
            b.SetBytes(bytesPerOp)
        })
    }
}

func BenchmarkConcurrentFileAccess(b *testing.B) {
    tempDir := b.TempDir()
    numWorkers := runtime.NumCPU()

    b.ResetTimer()

    b.RunParallel(func(pb *testing.PB) {
        workerID := atomic.AddInt64(&workerCounter, 1)
        data := make([]byte, 1024)
        rand.Read(data)

        for pb.Next() {
            filename := filepath.Join(tempDir, fmt.Sprintf("worker_%d_%d.dat", workerID, time.Now().UnixNano()))

            // Write file
            err := os.WriteFile(filename, data, 0644)
            if err != nil {
                b.Fatal(err)
            }

            // Read it back
            _, err = os.ReadFile(filename)
            if err != nil {
                b.Fatal(err)
            }

            // Clean up
            os.Remove(filename)
        }
    })
}

var workerCounter int64

func BenchmarkDirectoryTraversal(b *testing.B) {
    // Create directory structure for testing
    tempDir := b.TempDir()
    createTestDirectoryStructure(tempDir, 5, 10) // 5 levels deep, 10 files per level

    walker := NewDirectoryWalker()

    b.ResetTimer()

    for i := 0; i < b.N; i++ {
        files, err := walker.WalkDirectory(tempDir)
        if err != nil {
            b.Fatal(err)
        }

        // Ensure we found expected number of files
        if len(files) < 50 {
            b.Fatalf("Expected at least 50 files, found %d", len(files))
        }
    }
}

Advanced testing patterns reveal problems that traditional testing approaches miss. Property-based testing finds edge cases in your logic, fuzzing discovers inputs that cause crashes, and performance benchmarks ensure your code scales appropriately. These techniques require more investment than basic unit tests, but they provide confidence that your file system code handles the full range of conditions it will encounter in production.

Embedded File Systems: Using embed.FS in Production 8/9

Rez Moss — Sat, 26 Jul 2025 16:06:00 +0000

Understanding embed.FS

Go's embed package, introduced in Go 1.16, fundamentally changed how we handle static assets in Go applications. At its core, embed.FS implements the standard fs.FS interface, creating a read-only file system that exists entirely in memory as part of your compiled binary.

When you use //go:embed directives, the Go compiler reads files from your source tree during the build process and converts them into byte data that gets compiled directly into your executable. This happens at build time, not runtime - meaning the files you're embedding must exist when you compile your program, and they become immutable parts of your binary.

package main

import (
    "embed"
    "io/fs"
)

//go:embed static/*
var staticFiles embed.FS

//go:embed config.yaml
var configFile []byte

//go:embed templates/*.html
var templates embed.FS

The embed.FS type provides a complete file system interface. You can call Open() to get file handles, ReadDir() to list directory contents, and ReadFile() for direct file reading. Unlike traditional file operations that hit the disk, these operations work entirely in memory, accessing pre-loaded data structures built into your binary.

Performance characteristics differ significantly from disk-based file access. Reading embedded files eliminates filesystem latency entirely - there are no system calls, no disk seeks, and no network delays if you're running on distributed storage. However, this comes with memory overhead. Every embedded file consumes RAM for the lifetime of your process, whether you access it or not.

The embedded file system maintains the original directory structure from your source tree. If you embed static/css/main.css, you access it using that same path through the embed.FS interface. The file system is case-sensitive and uses forward slashes regardless of your host operating system.

One crucial aspect to understand is that embed.FS creates a completely separate namespace from your host filesystem. When you call staticFiles.Open("style.css"), you're not opening a file relative to your current working directory - you're opening a file from the embedded filesystem that was captured at build time from your source tree.

Embedding Strategies

The way you structure your embed directives directly impacts both your build process and runtime performance. Go provides several approaches for including files, each with distinct trade-offs you need to consider.

Single file embedding works well for configuration files, small templates, or individual assets. When you embed a single file into a []byte or string, Go stores it as a simple byte array in your binary:

//go:embed version.txt
var version string

//go:embed logo.png
var logoData []byte

This approach minimizes overhead for small files but becomes unwieldy when you need to manage multiple related assets. You'll end up with numerous package-level variables, making your code harder to organize.

Directory tree embedding provides a cleaner solution for related files. When you embed an entire directory, Go creates a hierarchical file system structure:

//go:embed static
var staticAssets embed.FS

//go:embed templates
var templateFiles embed.FS

The embedded file system preserves the complete directory structure, including subdirectories. If your static folder contains css/main.css and js/app.js, both files remain accessible at their original paths within the embedded filesystem.

Glob patterns offer the most flexible approach, letting you selectively include files based on patterns rather than directory boundaries:

//go:embed templates/*.html templates/*.txt
var templates embed.FS

//go:embed static/css/*.css static/js/*.js
var webAssets embed.FS

//go:embed configs
var configFiles embed.FS

The * pattern matches files within the current directory level. Note that Go's embed directive does not support the ** recursive glob pattern, to include files from nested subdirectories, embed the parent directory instead. You can combine multiple patterns in a single directive, and Go will merge all matching files into one embedded filesystem.

Managing embedded asset size becomes critical in production deployments. Every embedded file increases your binary size and memory footprint. Large binaries take longer to download, use more disk space, and consume additional RAM. Consider excluding development-only files, test fixtures, or documentation from your embedded assets.

You can use build tags to conditionally embed different asset sets:

//go:build !dev
//go:embed dist/minified
var webAssets embed.FS

//go:build dev
//go:embed src/unminified  
var webAssets embed.FS

For applications with extensive static assets, consider splitting them across multiple embed directives based on usage patterns. Frequently accessed files might go in one embedded filesystem, while rarely used assets go in another. This doesn't reduce total memory usage but can improve cache locality and make your code more organized.

Remember that embedded files cannot be modified at runtime. If you need to update assets without recompiling, you'll need to implement a fallback mechanism that checks the regular filesystem before falling back to embedded files.

Integration Patterns

Embedded file systems integrate seamlessly with Go's standard HTTP and template packages, but each integration pattern requires specific considerations for optimal performance and maintainability.

HTTP file serving becomes straightforward with embed.FS through the http.FS adapter. The standard library provides http.FileServer that works directly with embedded filesystems:

//go:embed static
var staticFiles embed.FS

func main() {
    // Serve embedded files from /static/ URL path
    staticFS, _ := fs.Sub(staticFiles, "static")
    http.Handle("/static/", http.StripPrefix("/static/", http.FileServer(http.FS(staticFS))))

    // Alternative: serve from root
    http.Handle("/", http.FileServer(http.FS(staticFiles)))
}

The fs.Sub function creates a subtree view of your embedded filesystem, letting you mount specific directories at different URL paths. This approach handles HTTP headers, content types, and caching automatically, but you lose control over fine-grained response customization.

For more control, read embedded files directly and serve them through custom handlers:

func serveAsset(w http.ResponseWriter, r *http.Request) {
    content, err := staticFiles.ReadFile("css/main.css")
    if err != nil {
        http.NotFound(w, r)
        return
    }

    w.Header().Set("Content-Type", "text/css")
    w.Header().Set("Cache-Control", "public, max-age=3600")
    w.Write(content)
}

Template systems work naturally with embedded files, though you need to use template.ParseFS instead of template.ParseFiles:

//go:embed templates/*.html
var templateFiles embed.FS

func loadTemplates() (*template.Template, error) {
    return template.ParseFS(templateFiles, "templates/*.html")
}

// For html/template with embedded files
func executeTemplate(w http.ResponseWriter, name string, data interface{}) {
    tmpl, err := template.ParseFS(templateFiles, "templates/"+name)
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }
    tmpl.Execute(w, data)
}

You can also pre-parse templates at startup and cache them in memory, which eliminates the parsing overhead on each request:

var templates *template.Template

func init() {
    templates = template.Must(template.ParseFS(templateFiles, "templates/*.html"))
}

Configuration file embedding works well for default configurations that applications can override at runtime:

//go:embed config/defaults.yaml
var defaultConfig []byte

type Config struct {
    Database DatabaseConfig `yaml:"database"`
    Server   ServerConfig   `yaml:"server"`
}

func loadConfig(configPath string) (*Config, error) {
    var config Config

    // Start with embedded defaults
    if err := yaml.Unmarshal(defaultConfig, &config); err != nil {
        return nil, err
    }

    // Override with external config if provided
    if configPath != "" {
        external, err := os.ReadFile(configPath)
        if err == nil {
            yaml.Unmarshal(external, &config)
        }
    }

    return &config, nil
}

This pattern gives you sensible defaults while maintaining flexibility for deployment-specific configurations. The embedded config serves as fallback when external configuration files are missing or incomplete.

When integrating embedded assets with middleware, remember that embed.FS operations are safe for concurrent access but consider caching frequently accessed files in memory if you're serving high traffic:

type CachedAssets struct {
    fs    embed.FS
    cache map[string][]byte
    mu    sync.RWMutex
}

func (c *CachedAssets) ReadFile(name string) ([]byte, error) {
    c.mu.RLock()
    if data, exists := c.cache[name]; exists {
        c.mu.RUnlock()
        return data, nil
    }
    c.mu.RUnlock()

    data, err := c.fs.ReadFile(name)
    if err != nil {
        return nil, err
    }

    c.mu.Lock()
    c.cache[name] = data
    c.mu.Unlock()

    return data, nil
}

Development vs Production Workflows

Managing embedded assets across different environments requires careful consideration of build processes, development efficiency, and production optimization. The static nature of embedded files creates unique challenges when you need rapid iteration during development.

Hot reloading becomes problematic with embedded assets since they're compiled into your binary. During development, you want immediate feedback when modifying CSS, JavaScript, or templates. The solution involves conditional logic that serves files from disk in development mode and from embedded assets in production:

//go:embed static
var embeddedAssets embed.FS

var isDevelopment = os.Getenv("GO_ENV") == "development"

func getFileSystem() http.FileSystem {
    if isDevelopment {
        return http.Dir("./static")
    }

    fsys, err := fs.Sub(embeddedAssets, "static")
    if err != nil {
        panic(err)
    }
    return http.FS(fsys)
}

func main() {
    http.Handle("/static/", http.StripPrefix("/static/", 
        http.FileServer(getFileSystem())))
}

This approach lets you modify files during development and see changes immediately without recompiling. However, you must ensure your development directory structure matches your embedded structure exactly, or you'll encounter runtime differences between environments.

Asset versioning becomes critical for cache management in production deployments. Since embedded files can't change without redeployment, you need strategies to handle browser caching and CDN invalidation:

//go:embed static
var staticFiles embed.FS

var assetVersion string

func init() {
    // Generate version hash from embedded content
    hasher := sha256.New()
    fs.WalkDir(staticFiles, "static", func(path string, d fs.DirEntry, err error) error {
        if !d.IsDir() {
            content, _ := staticFiles.ReadFile(path)
            hasher.Write(content)
        }
        return nil
    })
    assetVersion = fmt.Sprintf("%x", hasher.Sum(nil))[:8]
}

func serveWithVersion(w http.ResponseWriter, r *http.Request) {
    // Add version to headers for cache busting
    w.Header().Set("ETag", `"`+assetVersion+`"`)
    w.Header().Set("Cache-Control", "public, max-age=31536000")

    // Serve from embedded FS
    http.FileServer(http.FS(staticFiles)).ServeHTTP(w, r)
}

Alternatively, you can embed a version file that gets updated during your build process:

//go:embed version.txt
var buildVersion string

func init() {
    buildVersion = strings.TrimSpace(buildVersion)
}

Build optimization techniques help manage binary size and compilation time. Use build constraints to exclude development assets from production builds:

//go:build !production
//go:embed static/unminified
var devAssets embed.FS

//go:build production
//go:embed static/minified
var prodAssets embed.FS

//go:build !production
var staticFiles = devAssets

//go:build production  
var staticFiles = prodAssets

Your build pipeline can process assets before embedding them. Minify CSS and JavaScript, optimize images, and remove development-only files:

# Build script example
if [ "$GO_ENV" = "production" ]; then
    # Minify assets
    npm run build:production

    # Build with production tag
    go build -tags production -ldflags "-s -w" ./cmd/server
else
    go build ./cmd/server
fi

Consider using separate binaries for different deployment targets. A development binary might embed source maps and unminified assets, while production builds embed only optimized files. This requires maintaining parallel embed directives but gives you complete control over what gets included in each environment.

For applications with large asset collections, implement lazy loading patterns where you embed only critical assets and load others on demand:

//go:embed critical/*.css critical/*.js
var criticalAssets embed.FS

//go:embed optional
var optionalAssets embed.FS

func serveCritical(w http.ResponseWriter, r *http.Request) {
    // Always serve from embedded assets
    http.FileServer(http.FS(criticalAssets)).ServeHTTP(w, r)
}

func serveOptional(w http.ResponseWriter, r *http.Request) {
    // Check external file system first, fall back to embedded
    if isDevelopment {
        if _, err := os.Stat("./optional/" + r.URL.Path); err == nil {
            http.FileServer(http.Dir("./optional")).ServeHTTP(w, r)
            return
        }
    }
    http.FileServer(http.FS(optionalAssets)).ServeHTTP(w, r)
}

Security Considerations

Embedded file systems introduce unique security vectors that differ from traditional file serving approaches. While embedding eliminates many filesystem-based attacks, it creates new concerns around information disclosure and access control.

Embedded file visibility represents the most significant security consideration. Every file you embed becomes part of your binary and can be extracted by anyone with access to the executable. Standard tools like strings, hexdump, or specialized binary analysis software can reveal embedded content:

strings your-binary | grep -A5 -B5 "password"
hexdump -C your-binary | grep "secret"

This means you should never embed sensitive information like API keys, passwords, database credentials, or private keys. Even configuration files with default credentials become security risks when embedded:

// DANGEROUS - credentials visible in binary
//go:embed config/database.yaml
var dbConfig []byte

// SAFER - embed template, populate at runtime
//go:embed config/database.template.yaml  
var dbTemplate []byte

func loadConfig() {
    template := string(dbTemplate)
    config := strings.ReplaceAll(template, "{{DB_PASSWORD}}", os.Getenv("DB_PASSWORD"))
    // Use populated config
}

Path traversal prevention becomes crucial when serving embedded files through HTTP handlers. Although embed.FS doesn't provide access outside the embedded tree, improper path handling can still cause issues:

// VULNERABLE - doesn't validate paths
func serveFile(w http.ResponseWriter, r *http.Request) {
    path := r.URL.Path[1:] // Remove leading slash
    content, err := staticFiles.ReadFile(path)
    if err != nil {
        http.NotFound(w, r)
        return
    }
    w.Write(content)
}

// SAFER - validate and clean paths
func serveFile(w http.ResponseWriter, r *http.Request) {
    path := filepath.Clean(r.URL.Path[1:])

    // Ensure path doesn't escape embedded directory
    if strings.Contains(path, "..") || strings.HasPrefix(path, "/") {
        http.Error(w, "Invalid path", http.StatusBadRequest)
        return
    }

    content, err := staticFiles.ReadFile(path)
    if err != nil {
        http.NotFound(w, r)
        return
    }
    w.Write(content)
}

Content validation becomes important when embedding user-generated or external content. Files that seem safe at build time might contain malicious content:

//go:embed user-uploads/*.html
var userContent embed.FS

// RISKY - serving user HTML without validation
func serveUserContent(w http.ResponseWriter, r *http.Request) {
    content, err := userContent.ReadFile("user-uploads/" + r.URL.Path)
    if err != nil {
        http.NotFound(w, r)
        return
    }

    w.Header().Set("Content-Type", "text/html")
    w.Write(content) // Could contain XSS attacks
}

// SAFER - validate and sanitize content
func serveUserContent(w http.ResponseWriter, r *http.Request) {
    content, err := userContent.ReadFile("user-uploads/" + r.URL.Path)
    if err != nil {
        http.NotFound(w, r)
        return
    }

    // Sanitize HTML content
    sanitized := bluemonday.UGCPolicy().SanitizeBytes(content)

    w.Header().Set("Content-Type", "text/html")
    w.Header().Set("X-Content-Type-Options", "nosniff")
    w.Write(sanitized)
}

Information disclosure through error messages can reveal internal structure. Avoid exposing embedded filesystem paths in error responses:

// REVEALS INTERNAL STRUCTURE
func badHandler(w http.ResponseWriter, r *http.Request) {
    _, err := staticFiles.ReadFile("internal/config.json")
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError) // Reveals path
        return
    }
}

// GENERIC ERROR RESPONSE  
func goodHandler(w http.ResponseWriter, r *http.Request) {
    _, err := staticFiles.ReadFile("internal/config.json")
    if err != nil {
        http.Error(w, "Resource not found", http.StatusNotFound)
        return
    }
}

Access control for embedded resources requires application-level implementation since embed.FS doesn't provide built-in permissions:

type SecureFileServer struct {
    fs           embed.FS
    allowedPaths map[string]bool
    requiredRole string
}

func (s *SecureFileServer) ServeFile(w http.ResponseWriter, r *http.Request, userRole string) {
    path := filepath.Clean(r.URL.Path[1:])

    // Check path allowlist
    if !s.allowedPaths[path] {
        http.Error(w, "Forbidden", http.StatusForbidden)
        return
    }

    // Check user permissions
    if userRole != s.requiredRole {
        http.Error(w, "Unauthorized", http.StatusUnauthorized)
        return
    }

    content, err := s.fs.ReadFile(path)
    if err != nil {
        http.NotFound(w, r)
        return
    }

    w.Write(content)
}

Consider the attack surface of your embedded content. Large embedded filesystems increase binary analysis complexity but also provide more potential targets. Minimize embedded content to only what's necessary for your application to function, and regularly audit embedded files for sensitive information before deployment.

Advanced Techniques

Complex applications often require sophisticated file system patterns that go beyond basic embedding. These advanced techniques provide flexibility while maintaining the performance benefits of embedded assets.

Layered file systems allow you to combine multiple embedded filesystems with runtime priorities. This pattern proves useful when you need to override specific files while maintaining a base set of assets:

//go:embed base
var baseAssets embed.FS

//go:embed overrides  
var overrideAssets embed.FS

type LayeredFS struct {
    layers []fs.FS
}

func (lfs *LayeredFS) Open(name string) (fs.File, error) {
    for _, layer := range lfs.layers {
        if file, err := layer.Open(name); err == nil {
            return file, nil
        }
    }
    return nil, fs.ErrNotExist
}

func (lfs *LayeredFS) ReadFile(name string) ([]byte, error) {
    for _, layer := range lfs.layers {
        if data, err := fs.ReadFile(layer, name); err == nil {
            return data, nil
        }
    }
    return nil, fs.ErrNotExist
}

func NewLayeredFS() *LayeredFS {
    // First layer (overrides) takes priority
    return &LayeredFS{
        layers: []fs.FS{overrideAssets, baseAssets},
    }
}

This approach lets you ship base templates and configurations while allowing environment-specific customizations to take precedence. The layered filesystem checks each layer in order, returning the first match it finds.

Override mechanisms provide runtime flexibility for embedded content. You can implement fallback logic that checks external files before falling back to embedded versions:

type OverrideFS struct {
    embedded   embed.FS
    overrideDir string
}

func (ofs *OverrideFS) ReadFile(name string) ([]byte, error) {
    // Try external override first
    if ofs.overrideDir != "" {
        externalPath := filepath.Join(ofs.overrideDir, name)
        if data, err := os.ReadFile(externalPath); err == nil {
            return data, nil
        }
    }

    // Fall back to embedded file
    return ofs.embedded.ReadFile(name)
}

func (ofs *OverrideFS) Open(name string) (fs.File, error) {
    if ofs.overrideDir != "" {
        externalPath := filepath.Join(ofs.overrideDir, name)
        if file, err := os.Open(externalPath); err == nil {
            return file, nil
        }
    }

    return ofs.embedded.Open(name)
}

// Usage
var templateFS = &OverrideFS{
    embedded:    templateFiles,
    overrideDir: os.Getenv("TEMPLATE_OVERRIDE_DIR"),
}

This pattern maintains embedded defaults while allowing operators to customize specific files without recompiling the application. It's particularly valuable for templates, configuration files, and localization resources.

Dynamic content mixing combines embedded static assets with runtime-generated content. This technique works well for applications that need to inject dynamic data into static templates:

//go:embed templates
var templateFS embed.FS

type DynamicRenderer struct {
    staticFS    embed.FS
    dynamicData map[string]interface{}
}

func (dr *DynamicRenderer) RenderTemplate(name string, data interface{}) ([]byte, error) {
    // Read base template from embedded FS
    tmplContent, err := dr.staticFS.ReadFile("templates/" + name)
    if err != nil {
        return nil, err
    }

    // Parse and execute with combined data
    tmpl, err := template.New(name).Parse(string(tmplContent))
    if err != nil {
        return nil, err
    }

    // Merge static and dynamic data
    combinedData := make(map[string]interface{})
    for k, v := range dr.dynamicData {
        combinedData[k] = v
    }

    if dataMap, ok := data.(map[string]interface{}); ok {
        for k, v := range dataMap {
            combinedData[k] = v
        }
    }

    var buf bytes.Buffer
    err = tmpl.Execute(&buf, combinedData)
    return buf.Bytes(), err
}

Virtual filesystem composition lets you create complex file hierarchies by combining multiple embedded filesystems into a unified view:

type CompositeFS struct {
    mounts map[string]fs.FS
}

func (cfs *CompositeFS) Open(name string) (fs.File, error) {
    // Find the appropriate mount point
    for prefix, mountedFS := range cfs.mounts {
        if strings.HasPrefix(name, prefix) {
            relativePath := strings.TrimPrefix(name, prefix)
            relativePath = strings.TrimPrefix(relativePath, "/")
            return mountedFS.Open(relativePath)
        }
    }
    return nil, fs.ErrNotExist
}

func NewCompositeFS() *CompositeFS {
    return &CompositeFS{
        mounts: map[string]fs.FS{
            "static":    staticAssets,
            "templates": templateAssets,  
            "config":    configAssets,
        },
    }
}

Content transformation pipelines process embedded files before serving them. This allows you to apply runtime modifications without changing the embedded source:

type TransformFS struct {
    source      embed.FS
    transforms  map[string]func([]byte) []byte
}

func (tfs *TransformFS) ReadFile(name string) ([]byte, error) {
    data, err := tfs.source.ReadFile(name)
    if err != nil {
        return nil, err
    }

    // Apply transformations based on file extension
    ext := filepath.Ext(name)
    if transform, exists := tfs.transforms[ext]; exists {
        data = transform(data)
    }

    return data, nil
}

// Example usage with CSS minification
var cssFS = &TransformFS{
    source: styleAssets,
    transforms: map[string]func([]byte) []byte{
        ".css": func(data []byte) []byte {
            // Apply CSS minification
            return minifyCSS(data)
        },
        ".js": func(data []byte) []byte {
            // Apply JS minification  
            return minifyJS(data)
        },
    },
}

These advanced patterns provide powerful capabilities while maintaining the core benefits of embedded filesystems. They let you build flexible, maintainable applications that can adapt to different deployment scenarios without sacrificing performance or reliability.

Creating Custom File Systems: Implementing the FS Interface 7/9

Rez Moss — Sat, 12 Jul 2025 15:59:00 +0000

Design Principles for Custom File Systems

When building a custom file system that implements Go's fs.FS interface, you need to establish clear design principles that will guide your implementation decisions. These principles ensure your file system behaves predictably and integrates seamlessly with existing Go code.

Interface Compliance Requirements

The foundation of any custom file system is strict adherence to the fs.FS interface contract. Your implementation must return consistent behavior that other Go packages can rely on. The core Open(name string) (fs.File, error) method serves as the entry point for all file operations, and its behavior sets the tone for your entire implementation.

Path handling requires careful attention. Your file system should accept forward slashes as path separators, regardless of the underlying platform. Invalid paths containing elements like ".." or absolute paths starting with "/" should return fs.ErrInvalid. This consistency allows your file system to work correctly with packages like http.FileServer or template.ParseFS.

File names must be validated according to the interface specification. Empty names, names containing null bytes, or names with invalid Unicode sequences should trigger appropriate errors. Your implementation should also handle edge cases like "." for the current directory, which some callers might use to explore the file system root.

Error Handling Conventions

Error handling in file systems requires a delicate balance between providing useful information and maintaining security. The fs package defines several sentinel errors that your implementation should use consistently. fs.ErrNotExist should be returned when files or directories don't exist, while fs.ErrPermission indicates access restrictions.

Custom error types become necessary when you need to provide additional context. However, ensure these errors still satisfy the expected interfaces. A well-designed file system often implements fs.PathError for operations that fail on specific paths, providing both the operation name and the path that caused the failure.

Consider the caller's perspective when crafting error messages. Avoid exposing internal implementation details that could confuse users or create security vulnerabilities. A HTTP-based file system, for instance, shouldn't leak server error details through file system error messages.

Thread Safety Considerations

Modern applications frequently access file systems concurrently, making thread safety a critical design consideration. Your file system implementation must handle multiple goroutines accessing the same resources without corruption or deadlocks.

The fs.FS interface itself doesn't require specific thread safety guarantees, but practical usage patterns demand it. Multiple goroutines might call Open() simultaneously, or different goroutines might read from files opened from the same file system instance. Your implementation should support these patterns safely.

Consider using read-write mutexes when your file system maintains internal state that supports both read and write operations. For read-heavy workloads, this allows multiple concurrent readers while still protecting against writers. However, be cautious about lock granularity - overly broad locks can create performance bottlenecks.

State management becomes particularly important in file systems that cache data or maintain connections to external resources. Design your caching strategies with concurrency in mind, potentially using sync.Map for concurrent access or implementing per-resource locking to reduce contention.

Memory-based file systems face unique challenges since they typically store all data in memory structures. These structures must support concurrent access while maintaining consistency. Consider using atomic operations for simple state changes and proper synchronization primitives for complex operations that span multiple data structures.

Basic FS Implementation

Building your first custom file system starts with understanding how to map abstract file operations to your underlying data storage. A memory-based implementation provides an excellent foundation for learning these concepts before moving to more complex storage backends.

Memory-Based File System Example

A memory-based file system stores all files and directories in RAM using Go's built-in data structures. This approach offers fast access and simplifies the initial implementation while teaching core concepts that apply to any storage backend.

type MemFS struct {
    mu    sync.RWMutex
    files map[string]*memFile
    dirs  map[string]*memDir
}

type memFile struct {
    data    []byte
    modTime time.Time
    mode    fs.FileMode
}

type memDir struct {
    entries []fs.DirEntry
    modTime time.Time
}

func NewMemFS() *MemFS {
    return &MemFS{
        files: make(map[string]*memFile),
        dirs:  make(map[string]*memDir),
    }
}

func (mfs *MemFS) Open(name string) (fs.File, error) {
    if !fs.ValidPath(name) {
        return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrInvalid}
    }

    mfs.mu.RLock()
    defer mfs.mu.RUnlock()

    if file, exists := mfs.files[name]; exists {
        return &memFileReader{file: file, name: name}, nil
    }

    if dir, exists := mfs.dirs[name]; exists {
        return &memDirReader{dir: dir, name: name}, nil
    }

    return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
}

The key insight here is separating the file system structure from the file handles returned to callers. The MemFS struct maintains the storage, while separate reader types handle individual file access. This separation allows multiple callers to open the same file simultaneously without interfering with each other.

File and Directory Simulation

Simulating files and directories requires implementing the various interfaces that callers expect. Files must implement fs.File, which includes methods for reading, seeking, and obtaining file information. Directories implement additional interfaces for listing contents.

type memFileReader struct {
    file   *memFile
    name   string
    offset int64
}

func (r *memFileReader) Read(p []byte) (int, error) {
    if r.offset >= int64(len(r.file.data)) {
        return 0, io.EOF
    }

    n := copy(p, r.file.data[r.offset:])
    r.offset += int64(n)
    return n, nil
}

func (r *memFileReader) Stat() (fs.FileInfo, error) {
    return &memFileInfo{
        name:    filepath.Base(r.name),
        size:    int64(len(r.file.data)),
        mode:    r.file.mode,
        modTime: r.file.modTime,
    }, nil
}

func (r *memFileReader) Close() error {
    return nil
}

Directory simulation requires additional consideration for how you'll represent directory entries. The fs.DirEntry interface provides a lightweight way to describe directory contents without requiring full file information upfront.

type memDirReader struct {
    dir     *memDir
    name    string
    entries []fs.DirEntry
    offset  int
}

func (r *memDirReader) ReadDir(n int) ([]fs.DirEntry, error) {
    if r.offset >= len(r.entries) {
        return nil, io.EOF
    }

    end := r.offset + n
    if n <= 0 || end > len(r.entries) {
        end = len(r.entries)
    }

    entries := r.entries[r.offset:end]
    r.offset = end
    return entries, nil
}

Path Validation and Security

Path validation forms a critical security boundary in your file system implementation. Beyond the basic fs.ValidPath check, you need to consider how paths map to your internal storage and prevent access to unintended resources.

Implement path normalization early in your request processing. Convert all paths to a canonical form that your internal logic can work with consistently. This typically means cleaning paths, resolving any relative components, and ensuring consistent separator usage.

func (mfs *MemFS) normalizePath(path string) (string, error) {
    if !fs.ValidPath(path) {
        return "", fs.ErrInvalid
    }

    // Clean the path to remove redundant separators and resolve . components
    cleaned := filepath.Clean(path)

    // Convert back slashes to forward slashes for consistency
    cleaned = filepath.ToSlash(cleaned)

    // Ensure we don't have any parent directory references
    if strings.Contains(cleaned, "..") {
        return "", fs.ErrInvalid
    }

    return cleaned, nil
}

Security considerations extend beyond path validation. Consider what information your error messages reveal. A file system that exposes whether a path doesn't exist versus lacking permission to access it might leak information about the underlying storage structure. Design your error responses to provide useful feedback without compromising security.

Consider implementing access controls at the file system level rather than relying solely on the underlying storage. This approach gives you fine-grained control over what callers can access and provides a consistent security model regardless of your storage backend.

Advanced Interface Implementations

Beyond the basic fs.FS interface, Go provides several optional interfaces that can significantly improve your file system's performance and functionality. Implementing these interfaces allows your file system to integrate more deeply with the Go ecosystem and provide optimized operations for common use cases.

ReadDirFS for Directory Optimization

The fs.ReadDirFS interface provides a direct way to list directory contents without requiring callers to open a directory file first. This optimization becomes particularly valuable when your underlying storage can list directories more efficiently than the default file-based approach.

func (mfs *MemFS) ReadDir(name string) ([]fs.DirEntry, error) {
    cleanName, err := mfs.normalizePath(name)
    if err != nil {
        return nil, &fs.PathError{Op: "readdir", Path: name, Err: err}
    }

    mfs.mu.RLock()
    defer mfs.mu.RUnlock()

    // Handle root directory case
    if cleanName == "." {
        return mfs.listRootEntries(), nil
    }

    dir, exists := mfs.dirs[cleanName]
    if !exists {
        return nil, &fs.PathError{Op: "readdir", Path: name, Err: fs.ErrNotExist}
    }

    // Return a copy to prevent external modification
    entries := make([]fs.DirEntry, len(dir.entries))
    copy(entries, dir.entries)
    return entries, nil
}

The performance benefit becomes clear when you consider network-based or database-backed file systems. Instead of opening a directory file and making multiple read calls, ReadDir allows a single operation to retrieve all directory contents. This reduces both latency and resource usage.

Implementing ReadDirFS also improves compatibility with Go's standard library functions. Functions like filepath.WalkDir automatically detect and use this interface when available, providing better performance for directory traversal operations.

StatFS for Metadata Efficiency

The fs.StatFS interface enables efficient metadata retrieval without opening files. This becomes crucial for operations that need file information but don't require file contents, such as directory listings that display file sizes or modification times.

func (mfs *MemFS) Stat(name string) (fs.FileInfo, error) {
    cleanName, err := mfs.normalizePath(name)
    if err != nil {
        return nil, &fs.PathError{Op: "stat", Path: name, Err: err}
    }

    mfs.mu.RLock()
    defer mfs.mu.RUnlock()

    if file, exists := mfs.files[cleanName]; exists {
        return &memFileInfo{
            name:    filepath.Base(cleanName),
            size:    int64(len(file.data)),
            mode:    file.mode,
            modTime: file.modTime,
        }, nil
    }

    if dir, exists := mfs.dirs[cleanName]; exists {
        return &memDirInfo{
            name:    filepath.Base(cleanName),
            mode:    fs.ModeDir | 0755,
            modTime: dir.modTime,
        }, nil
    }

    return nil, &fs.PathError{Op: "stat", Path: name, Err: fs.ErrNotExist}
}

For remote file systems, StatFS can dramatically reduce network round trips. Instead of opening a file to get its metadata, your implementation can query the remote system directly for file information, potentially batching multiple stat operations into a single request.

The interface also enables more sophisticated caching strategies. You can cache file metadata separately from file contents, allowing stat operations to return immediately from cache while still requiring network access for file reads.

GlobFS for Pattern Matching

The fs.GlobFS interface provides efficient pattern matching capabilities that many applications require. Rather than forcing callers to traverse entire directory trees manually, this interface allows your file system to optimize pattern matching based on its internal structure.

func (mfs *MemFS) Glob(pattern string) ([]string, error) {
    if !fs.ValidPath(pattern) {
        return nil, &fs.PathError{Op: "glob", Path: pattern, Err: fs.ErrInvalid}
    }

    mfs.mu.RLock()
    defer mfs.mu.RUnlock()

    var matches []string

    // Check all files against the pattern
    for path := range mfs.files {
        if matched, _ := filepath.Match(pattern, path); matched {
            matches = append(matches, path)
        }
    }

    // Check all directories against the pattern
    for path := range mfs.dirs {
        if matched, _ := filepath.Match(pattern, path); matched {
            matches = append(matches, path)
        }
    }

    sort.Strings(matches)
    return matches, nil
}

For database-backed file systems, GlobFS enables pushing pattern matching down to the database level using SQL LIKE operations or similar constructs. This approach can significantly reduce the amount of data transferred and processed in your application.

Tree-structured storage systems can optimize glob operations by pruning search paths early. If a pattern doesn't match a directory prefix, the entire subtree can be skipped, dramatically reducing search time for complex directory hierarchies.

Advanced glob implementations might also support more sophisticated patterns than the basic shell-style matching. Regular expressions, SQL-style patterns, or custom matching logic can be implemented while still maintaining the standard interface contract.

The key to effective GlobFS implementation lies in understanding your storage system's strengths. Memory-based systems can afford to check every path, while remote systems benefit from server-side filtering. Database systems might translate patterns into efficient queries, while cached systems could maintain pattern indexes for frequently used glob operations.

Error Handling Patterns

Robust error handling distinguishes professional file system implementations from simple prototypes. The fs package establishes specific error conventions that your implementation should follow to ensure consistent behavior across the Go ecosystem.

PathError Construction

The fs.PathError type serves as the standard way to report path-specific errors in file system operations. This error type wraps the underlying error while providing context about the operation and path that failed.

type PathError struct {
    Op   string
    Path string
    Err  error
}

func (e *PathError) Error() string {
    return e.Op + " " + e.Path + ": " + e.Err.Error()
}

func (e *PathError) Unwrap() error {
    return e.Err
}

Your file system should construct PathError instances consistently across all operations. The Op field should use lowercase operation names that match the interface method being called. Common operations include "open", "stat", "readdir", and "glob".

func (mfs *MemFS) Open(name string) (fs.File, error) {
    if !fs.ValidPath(name) {
        return nil, &fs.PathError{
            Op:   "open",
            Path: name,
            Err:  fs.ErrInvalid,
        }
    }

    // ... rest of implementation

    if _, exists := mfs.files[name]; !exists {
        return nil, &fs.PathError{
            Op:   "open",
            Path: name,
            Err:  fs.ErrNotExist,
        }
    }
}

The path in PathError should be the original path provided by the caller, not your internal normalized version. This preserves the context that users need to understand and debug their code. However, be cautious about exposing sensitive information through error paths in security-sensitive applications.

Consistent Error Messages

Error message consistency helps users build reliable applications on top of your file system. The fs package defines several sentinel errors that have specific meanings, and your implementation should use these errors appropriately.

fs.ErrNotExist indicates that a requested file or directory doesn't exist. This error should be used consistently whether the path is completely invalid or points to a location that simply doesn't contain a file.

var (
    ErrInvalid    = errInvalid() // "invalid argument"
    ErrPermission = errPermission() // "permission denied"
    ErrExist      = errExist() // "file already exists"
    ErrNotExist   = errNotExist() // "file does not exist"
    ErrClosed     = errClosed() // "file already closed"
)

fs.ErrPermission signals access control violations. Your file system should return this error when operations fail due to insufficient permissions, not when they fail for other reasons like network connectivity issues.

fs.ErrInvalid covers malformed inputs, including invalid paths, null bytes in filenames, or other parameter validation failures. This error helps distinguish between "you asked for something that doesn't exist" and "you asked for something incorrectly."

func validateFileName(name string) error {
    if name == "" {
        return fs.ErrInvalid
    }

    if strings.Contains(name, "\x00") {
        return fs.ErrInvalid
    }

    if !utf8.ValidString(name) {
        return fs.ErrInvalid
    }

    return nil
}

Timeout and Permission Errors

Network-based and remote file systems encounter additional error conditions that require careful handling. Timeout errors should provide enough context for callers to distinguish between different types of failures while maintaining security boundaries.

type TimeoutError struct {
    Op      string
    Path    string
    Timeout time.Duration
}

func (e *TimeoutError) Error() string {
    return fmt.Sprintf("%s %s: operation timed out after %v", e.Op, e.Path, e.Timeout)
}

func (e *TimeoutError) Timeout() bool {
    return true
}

func (e *TimeoutError) Temporary() bool {
    return true
}

Implementing the Timeout() and Temporary() methods allows error-handling code to respond appropriately to different failure modes. Network clients can retry temporary errors but might handle permanent errors differently.

Permission errors require special consideration in multi-user or security-sensitive environments. Your error messages should provide enough information for legitimate debugging without exposing system internals that could aid attackers.

func (mfs *MemFS) checkPermission(path string, operation string) error {
    // Simplified permission checking
    if mfs.isRestricted(path) {
        return &fs.PathError{
            Op:   operation,
            Path: path,
            Err:  fs.ErrPermission,
        }
    }
    return nil
}

Consider implementing custom error types for domain-specific failures. A database-backed file system might define connection errors, while a cached file system could report cache coherency issues. These custom errors should still wrap appropriate standard errors when possible.

type CacheError struct {
    Op   string
    Path string
    Err  error
}

func (e *CacheError) Error() string {
    return fmt.Sprintf("cache %s %s: %v", e.Op, e.Path, e.Err)
}

func (e *CacheError) Unwrap() error {
    return &fs.PathError{
        Op:   e.Op,
        Path: e.Path,
        Err:  e.Err,
    }
}

Error handling also extends to resource cleanup. Ensure that your file system properly handles partial failures and doesn't leak resources when operations are interrupted. This becomes particularly important for file systems that manage external connections, temporary files, or other system resources.

Testing Custom Implementations

Testing custom file systems requires a systematic approach that validates both interface compliance and implementation-specific behavior. The Go standard library provides tools to help verify your implementation meets the expected contracts, while additional testing strategies ensure robust behavior under various conditions.

Using testing/fstest.TestFS

The testing/fstest package provides TestFS, a comprehensive test suite that validates your file system against the standard interface requirements. This test suite checks hundreds of edge cases and interface compliance issues that are easy to miss in manual testing.

func TestMemFS(t *testing.T) {
    // Create a test file system with known content
    mfs := NewMemFS()

    // Add test files and directories
    mfs.WriteFile("hello.txt", []byte("hello world"), 0644)
    mfs.WriteFile("dir/nested.txt", []byte("nested content"), 0644)
    mfs.MkdirAll("empty-dir", 0755)

    // Run the standard compliance tests
    if err := fstest.TestFS(mfs, "hello.txt", "dir/nested.txt", "dir", "empty-dir"); err != nil {
        t.Fatal(err)
    }
}

TestFS expects you to provide a list of paths that should exist in your test file system. The function then performs exhaustive testing of these paths and many invalid paths, ensuring your implementation handles edge cases correctly.

The test suite validates critical behaviors like path normalization, error handling, and interface compliance. It checks that your Open method returns appropriate errors for invalid paths, that directory listing works correctly, and that file reading behaves as expected.

func setupTestFS() fs.FS {
    mfs := NewMemFS()

    // Create a realistic directory structure
    testFiles := map[string]string{
        "README.md":           "# Test File System",
        "src/main.go":         "package main\n\nfunc main() {}",
        "src/util/helper.go":  "package util",
        "docs/api.md":         "# API Documentation",
        "configs/app.yaml":    "version: 1.0",
    }

    for path, content := range testFiles {
        mfs.WriteFile(path, []byte(content), 0644)
    }

    return mfs
}

Running TestFS early and frequently during development helps catch interface violations before they become deeply embedded in your implementation. The test suite is thorough but not exhaustive - it focuses on interface compliance rather than performance or implementation-specific features.

Unit Testing Strategies

Beyond interface compliance, your file system needs unit tests that verify implementation-specific behavior, performance characteristics, and error conditions. These tests should focus on the unique aspects of your file system that TestFS doesn't cover.

func TestConcurrentAccess(t *testing.T) {
    mfs := NewMemFS()
    mfs.WriteFile("test.txt", []byte("initial content"), 0644)

    const numGoroutines = 100
    var wg sync.WaitGroup
    wg.Add(numGoroutines)

    // Test concurrent reads
    for i := 0; i < numGoroutines; i++ {
        go func() {
            defer wg.Done()

            file, err := mfs.Open("test.txt")
            if err != nil {
                t.Errorf("failed to open file: %v", err)
                return
            }
            defer file.Close()

            data, err := io.ReadAll(file)
            if err != nil {
                t.Errorf("failed to read file: %v", err)
                return
            }

            if string(data) != "initial content" {
                t.Errorf("unexpected content: %s", data)
            }
        }()
    }

    wg.Wait()
}

Performance testing becomes crucial for file systems that will handle large numbers of files or frequent operations. Benchmark tests help identify performance regressions and validate optimization efforts.

func BenchmarkFileOpen(b *testing.B) {
    mfs := setupLargeTestFS() // Create FS with many files

    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        file, err := mfs.Open("src/main.go")
        if err != nil {
            b.Fatal(err)
        }
        file.Close()
    }
}

func BenchmarkDirectoryListing(b *testing.B) {
    mfs := setupLargeTestFS()

    b.ResetTimer()
    for i := 0; i < b.N; i++ {
        entries, err := mfs.ReadDir("src")
        if err != nil {
            b.Fatal(err)
        }
        _ = entries // Use the result to prevent optimization
    }
}

Edge Case Coverage

Comprehensive testing requires systematically exploring edge cases that real applications might encounter. These tests often reveal subtle bugs that don't appear in normal usage patterns.

func TestEdgeCases(t *testing.T) {
    mfs := NewMemFS()

    tests := []struct {
        name        string
        operation   func() error
        expectError error
    }{
        {
            name: "empty path",
            operation: func() error {
                _, err := mfs.Open("")
                return err
            },
            expectError: fs.ErrInvalid,
        },
        {
            name: "path with null byte",
            operation: func() error {
                _, err := mfs.Open("file\x00name")
                return err
            },
            expectError: fs.ErrInvalid,
        },
        {
            name: "very long path",
            operation: func() error {
                longPath := strings.Repeat("a", 1000)
                _, err := mfs.Open(longPath)
                return err
            },
            expectError: fs.ErrNotExist,
        },
        {
            name: "unicode path",
            operation: func() error {
                mfs.WriteFile("测试.txt", []byte("test"), 0644)
                _, err := mfs.Open("测试.txt")
                return err
            },
            expectError: nil,
        },
    }

    for _, tt := range tests {
        t.Run(tt.name, func(t *testing.T) {
            err := tt.operation()
            if !errors.Is(err, tt.expectError) {
                t.Errorf("expected error %v, got %v", tt.expectError, err)
            }
        })
    }
}

Resource exhaustion testing helps ensure your file system behaves gracefully under stress. These tests might create thousands of files, attempt to read extremely large files, or simulate network failures for remote file systems.

func TestResourceLimits(t *testing.T) {
    if testing.Short() {
        t.Skip("skipping resource limit test in short mode")
    }

    mfs := NewMemFS()

    // Test creating many files
    const numFiles = 10000
    for i := 0; i < numFiles; i++ {
        filename := fmt.Sprintf("file_%d.txt", i)
        err := mfs.WriteFile(filename, []byte("content"), 0644)
        if err != nil {
            t.Fatalf("failed to create file %d: %v", i, err)
        }
    }

    // Verify all files are accessible
    for i := 0; i < numFiles; i++ {
        filename := fmt.Sprintf("file_%d.txt", i)
        _, err := mfs.Open(filename)
        if err != nil {
            t.Errorf("failed to open file %d: %v", i, err)
        }
    }
}

Integration testing with real Go packages validates that your file system works correctly in practical scenarios. Testing with http.FileServer, template.ParseFS, or embed.FS operations ensures compatibility with common use cases.

func TestHTTPIntegration(t *testing.T) {
    mfs := NewMemFS()
    mfs.WriteFile("index.html", []byte("<html><body>Hello</body></html>"), 0644)
    mfs.WriteFile("style.css", []byte("body { color: blue; }"), 0644)

    handler := http.FileServer(http.FS(mfs))
    server := httptest.NewServer(handler)
    defer server.Close()

    // Test serving files through HTTP
    resp, err := http.Get(server.URL + "/index.html")
    if err != nil {
        t.Fatal(err)
    }
    defer resp.Body.Close()

    if resp.StatusCode != http.StatusOK {
        t.Errorf("expected status 200, got %d", resp.StatusCode)
    }
}

Real-World Examples

Implementing custom file systems becomes more meaningful when you understand how they solve practical problems. These real-world examples demonstrate different approaches to common challenges, each highlighting specific design decisions and implementation techniques.

HTTP-Based File System

An HTTP-based file system provides access to remote files through standard HTTP requests. This approach enables serving files from web servers, CDNs, or REST APIs while maintaining the familiar file system interface.

type HTTPFS struct {
    baseURL string
    client  *http.Client
    cache   map[string]*httpFile
    mu      sync.RWMutex
}

type httpFile struct {
    path     string
    size     int64
    modTime  time.Time
    content  []byte
    fetched  bool
}

func NewHTTPFS(baseURL string) *HTTPFS {
    return &HTTPFS{
        baseURL: strings.TrimSuffix(baseURL, "/"),
        client: &http.Client{
            Timeout: 30 * time.Second,
        },
        cache: make(map[string]*httpFile),
    }
}

func (hfs *HTTPFS) Open(name string) (fs.File, error) {
    if !fs.ValidPath(name) {
        return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrInvalid}
    }

    url := hfs.baseURL + "/" + name

    hfs.mu.RLock()
    if file, exists := hfs.cache[name]; exists && file.fetched {
        hfs.mu.RUnlock()
        return &httpFileReader{file: file, name: name}, nil
    }
    hfs.mu.RUnlock()

    // Fetch file metadata and content
    resp, err := hfs.client.Get(url)
    if err != nil {
        return nil, &fs.PathError{Op: "open", Path: name, Err: err}
    }
    defer resp.Body.Close()

    if resp.StatusCode == http.StatusNotFound {
        return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
    }

    if resp.StatusCode != http.StatusOK {
        return nil, &fs.PathError{Op: "open", Path: name, Err: fmt.Errorf("HTTP %d", resp.StatusCode)}
    }

    content, err := io.ReadAll(resp.Body)
    if err != nil {
        return nil, &fs.PathError{Op: "open", Path: name, Err: err}
    }

    // Parse modification time from headers
    modTime := time.Now()
    if lastModified := resp.Header.Get("Last-Modified"); lastModified != "" {
        if parsed, err := http.ParseTime(lastModified); err == nil {
            modTime = parsed
        }
    }

    file := &httpFile{
        path:    name,
        size:    int64(len(content)),
        modTime: modTime,
        content: content,
        fetched: true,
    }

    hfs.mu.Lock()
    hfs.cache[name] = file
    hfs.mu.Unlock()

    return &httpFileReader{file: file, name: name}, nil
}

The HTTP file system demonstrates several key patterns. Caching reduces repeated network requests, while proper error mapping translates HTTP status codes into appropriate file system errors. The implementation handles network timeouts and connection failures gracefully.

For production use, consider implementing cache expiration based on HTTP cache headers, supporting authentication mechanisms, and handling partial content requests for large files. Range requests can significantly improve performance when clients only need portions of large files.

Cached File System Wrapper

A cached file system wrapper adds caching capabilities to any underlying file system. This pattern proves particularly useful for slow storage backends like network file systems or encrypted storage.

type CachedFS struct {
    underlying fs.FS
    cache      map[string]*cacheEntry
    mu         sync.RWMutex
    maxSize    int64
    currentSize int64
}

type cacheEntry struct {
    content   []byte
    info      fs.FileInfo
    timestamp time.Time
    size      int64
}

func NewCachedFS(underlying fs.FS, maxSize int64) *CachedFS {
    return &CachedFS{
        underlying: underlying,
        cache:      make(map[string]*cacheEntry),
        maxSize:    maxSize,
    }
}

func (cfs *CachedFS) Open(name string) (fs.File, error) {
    cfs.mu.RLock()
    if entry, exists := cfs.cache[name]; exists {
        if time.Since(entry.timestamp) < 5*time.Minute { // Cache TTL
            cfs.mu.RUnlock()
            return &cachedFileReader{
                content: entry.content,
                info:    entry.info,
                name:    name,
            }, nil
        }
    }
    cfs.mu.RUnlock()

    // Cache miss or expired - fetch from underlying FS
    file, err := cfs.underlying.Open(name)
    if err != nil {
        return nil, err
    }
    defer file.Close()

    info, err := file.Stat()
    if err != nil {
        return nil, err
    }

    // Only cache files, not directories
    if info.IsDir() {
        return cfs.underlying.Open(name)
    }

    content, err := io.ReadAll(file)
    if err != nil {
        return nil, err
    }

    // Cache the file if there's space
    cfs.mu.Lock()
    if cfs.currentSize+int64(len(content)) <= cfs.maxSize {
        cfs.evictIfNeeded(int64(len(content)))
        cfs.cache[name] = &cacheEntry{
            content:   content,
            info:      info,
            timestamp: time.Now(),
            size:      int64(len(content)),
        }
        cfs.currentSize += int64(len(content))
    }
    cfs.mu.Unlock()

    return &cachedFileReader{
        content: content,
        info:    info,
        name:    name,
    }, nil
}

func (cfs *CachedFS) evictIfNeeded(newSize int64) {
    for cfs.currentSize+newSize > cfs.maxSize && len(cfs.cache) > 0 {
        // Simple LRU: find oldest entry
        var oldestKey string
        var oldestTime time.Time = time.Now()

        for key, entry := range cfs.cache {
            if entry.timestamp.Before(oldestTime) {
                oldestTime = entry.timestamp
                oldestKey = key
            }
        }

        if oldestKey != "" {
            cfs.currentSize -= cfs.cache[oldestKey].size
            delete(cfs.cache, oldestKey)
        }
    }
}

The cached file system wrapper demonstrates composition patterns where one file system enhances another. Cache eviction policies, TTL management, and memory usage tracking become critical concerns for production implementations.

Union File System Implementation

A union file system combines multiple underlying file systems, presenting them as a single unified interface. This pattern enables layered file systems, overlay mounts, and fallback mechanisms.

type UnionFS struct {
    layers []fs.FS
    mu     sync.RWMutex
}

func NewUnionFS(layers ...fs.FS) *UnionFS {
    return &UnionFS{
        layers: layers,
    }
}

func (ufs *UnionFS) Open(name string) (fs.File, error) {
    if !fs.ValidPath(name) {
        return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrInvalid}
    }

    ufs.mu.RLock()
    defer ufs.mu.RUnlock()

    // Try each layer in order
    for i, layer := range ufs.layers {
        file, err := layer.Open(name)
        if err == nil {
            return &unionFile{
                File:  file,
                layer: i,
                path:  name,
            }, nil
        }

        // Continue to next layer unless it's a fatal error
        if !errors.Is(err, fs.ErrNotExist) {
            return nil, err
        }
    }

    return nil, &fs.PathError{Op: "open", Path: name, Err: fs.ErrNotExist}
}

func (ufs *UnionFS) ReadDir(name string) ([]fs.DirEntry, error) {
    if !fs.ValidPath(name) {
        return nil, &fs.PathError{Op: "readdir", Path: name, Err: fs.ErrInvalid}
    }

    ufs.mu.RLock()
    defer ufs.mu.RUnlock()

    entryMap := make(map[string]fs.DirEntry)

    // Merge entries from all layers
    for _, layer := range ufs.layers {
        if readDirFS, ok := layer.(fs.ReadDirFS); ok {
            entries, err := readDirFS.ReadDir(name)
            if err != nil && !errors.Is(err, fs.ErrNotExist) {
                continue // Skip layers with errors
            }

            for _, entry := range entries {
                // First layer wins for duplicate names
                if _, exists := entryMap[entry.Name()]; !exists {
                    entryMap[entry.Name()] = entry
                }
            }
        }
    }

    // Convert map to sorted slice
    entries := make([]fs.DirEntry, 0, len(entryMap))
    for _, entry := range entryMap {
        entries = append(entries, entry)
    }

    sort.Slice(entries, func(i, j int) bool {
        return entries[i].Name() < entries[j].Name()
    })

    return entries, nil
}

Union file systems require careful consideration of precedence rules, directory merging strategies, and conflict resolution. The implementation above uses a simple "first layer wins" approach, but production systems might need more sophisticated policies.

These real-world examples demonstrate that custom file systems solve diverse problems through different architectural approaches. HTTP file systems bridge network protocols, cached wrappers improve performance, and union systems provide flexible composition. Each pattern can be adapted and combined to meet specific application requirements while maintaining the standard Go file system interface.

Specialized File System Interfaces: ReadFileFS, StatFS, and SubFS 6/9

Rez Moss — Sat, 05 Jul 2025 15:52:00 +0000

File systems in Go aren't just about implementing the basic fs.FS interface. When you're building systems that need to handle files efficiently, you'll often find yourself reaching for more specialized interfaces that can unlock significant performance gains and cleaner code patterns.

The Go standard library provides three key specialized interfaces that extend the basic file system functionality: ReadFileFS, StatFS, and SubFS. Each serves a specific purpose and solves common problems you'll encounter when working with file systems at scale.

ReadFileFS Interface Benefits

The ReadFileFS interface adds a single method to the basic fs.FS:

type ReadFileFS interface {
    FS
    ReadFile(name string) ([]byte, error)
}

Optimized Single-Call File Reading

When your file system implements ReadFileFS, functions like fs.ReadFile() will use this specialized method instead of the standard open-read-close sequence. This seemingly simple optimization can have dramatic performance implications.

Consider a typical file reading operation without ReadFileFS:

// Standard approach: 3 system calls
file, err := fsys.Open("config.json")
if err != nil {
    return nil, err
}
defer file.Close()

data, err := io.ReadAll(file)

With ReadFileFS, this becomes a single optimized call that your file system can handle however it sees fit. An in-memory file system might directly return bytes from its internal storage. A network-based file system could make a single HTTP request instead of establishing a connection, reading, then closing.

Performance vs Convenience Trade-offs

The decision to implement ReadFileFS involves weighing immediate convenience against long-term performance characteristics. If your file system primarily serves small files that are read entirely into memory, implementing this interface is usually worthwhile. The performance gains compound when you're dealing with hundreds or thousands of file reads.

However, there's a trade-off in implementation complexity. Your ReadFile method needs to handle all the edge cases that the standard open-read-close pattern would handle: permissions, file not found errors, and proper error wrapping.

When to Implement vs Use ReadFile Function

You should implement ReadFileFS when your underlying storage can optimize the full-file-read operation. This is common in:

Archive-based file systems (ZIP, TAR) where you're already reading file contents during extraction
Database-backed file systems where a single query can retrieve the complete file
Caching file systems where you might have the entire file already in memory

Don't implement it if your file system is simply wrapping another file system without adding optimization opportunities. In those cases, let the underlying system's ReadFileFS implementation handle the optimization.

StatFS for Efficient Metadata Access

The StatFS interface provides direct access to file metadata without requiring file operations:

type StatFS interface {
    FS
    Stat(name string) (FileInfo, error)
}

This interface addresses a common inefficiency in file system operations where you need file information but not the file contents themselves.

Stat Method vs Open+Stat Pattern

Without StatFS, checking file metadata requires opening the file first:

// Standard approach: open file to get info
file, err := fsys.Open("large-video.mp4")
if err != nil {
    return err
}
defer file.Close()

info, err := file.Stat()

This pattern becomes problematic when dealing with large files or slow storage systems. You're establishing a full file handle just to read a few bytes of metadata. With StatFS, this becomes a direct metadata lookup:

// Direct metadata access
info, err := fsys.Stat("large-video.mp4")

The performance difference is particularly pronounced with network file systems, where opening a file might involve authentication, connection establishment, and resource allocation on the remote server.

Metadata-Only Operations

Many file system operations only need metadata: build systems checking modification times, backup utilities determining file sizes, or directory listing operations that display file details. When your file system implements StatFS, these operations become significantly more efficient.

Consider a directory listing that shows file sizes:

// Without StatFS: potentially opens every file
entries, _ := fs.ReadDir(fsys, "photos")
for _, entry := range entries {
    if !entry.IsDir() {
        // This might open the file internally
        info, _ := entry.Info()
        fmt.Printf("%s: %d bytes\n", entry.Name(), info.Size())
    }
}

With StatFS, the fs.Stat() function can bypass file opening entirely, making directory operations much faster when you're dealing with directories containing many files.

Permission Checking Without File Opening

One of the most practical applications of StatFS is permission checking. Often you need to verify that a file exists and is accessible without actually reading it. This is common in security-conscious applications where you want to validate file paths before processing them.

// Check if file exists and get basic info without opening
info, err := fs.Stat(fsys, userProvidedPath)
if err != nil {
    return fmt.Errorf("file not accessible: %w", err)
}

if info.IsDir() {
    return errors.New("expected file, got directory")
}

// Now safely proceed with file operations

This pattern is essential in web servers, file processors, and any system that needs to validate file access patterns before committing to expensive operations.

SubFS for File System Scoping

The SubFS interface enables creating restricted views of file systems:

type SubFS interface {
    FS
    Sub(dir string) (FS, error)
}

This interface solves a fundamental problem in file system design: how to safely limit access to specific portions of a larger file system without complex path manipulation or security checks scattered throughout your code.

Creating Sandboxed File System Views

When you call Sub() on a file system, you get back a new file system that treats the specified directory as its root. This creates a natural sandbox where code operating on the sub-filesystem cannot access files outside the designated area.

// Create a sandboxed view of the templates directory
templateFS, err := fs.Sub(mainFS, "templates")
if err != nil {
    return err
}

// This file system can only access files within templates/
// Attempts to access "../config/secrets.json" will fail
data, err := fs.ReadFile(templateFS, "user-profile.html")

The key insight is that the sub-filesystem has no knowledge of the parent structure. From its perspective, it is the entire file system. This makes it impossible for code using the sub-filesystem to accidentally or maliciously access parent directories.

Security Implications and Boundaries

SubFS creates hard security boundaries that are enforced at the file system level rather than through application logic. This is crucial for applications that process user-provided templates, serve static files, or operate on untrusted directory structures.

Consider a template processing system:

func processUserTemplate(userID string, templateName string) error {
    // Create user-specific file system view
    userFS, err := fs.Sub(rootFS, fmt.Sprintf("users/%s/templates", userID))
    if err != nil {
        return err
    }

    // Template processor can only access this user's templates
    return templateProcessor.Process(userFS, templateName)
}

Even if the template processor has bugs or the templateName contains path traversal attempts like ../../../etc/passwd, the sub-filesystem prevents access to files outside the user's template directory.

Nested Sub Operations

Sub-filesystems can be further subdivided, creating layered access controls:

// Start with user's directory
userFS, _ := fs.Sub(rootFS, "users/alice")

// Further restrict to just the public directory
publicFS, _ := fs.Sub(userFS, "public")

// Or in one operation
publicFS, _ := fs.Sub(rootFS, "users/alice/public")

This nested approach is particularly useful in content management systems where you might have organization-level access, then project-level access, then feature-specific access controls.

The composition also works well with other specialized interfaces. A sub-filesystem that implements ReadFileFS will still provide optimized file reading within its restricted scope:

// Both interfaces work together
if readFS, ok := publicFS.(fs.ReadFileFS); ok {
    // Fast file reading within the sandboxed area
    content, err := readFS.ReadFile("index.html")
}

Implementation Strategies

Building file systems that effectively use these specialized interfaces requires careful consideration of when and how to implement them. The decision isn't just technical—it affects the maintainability and performance characteristics of your entire system.

When to Implement These Interfaces

The choice to implement specialized interfaces should be driven by concrete performance needs and usage patterns, not abstract optimization goals. Start by profiling your actual file system usage to understand where bottlenecks occur.

Implement ReadFileFS when you have evidence that file reading is a bottleneck and your storage layer can optimize full-file reads. This is common in systems that:

Serve many small files (like static web assets)
Cache entire files in memory
Read from compressed archives where you're already decompressing the full content

Implement StatFS when metadata operations are frequent relative to content operations. This happens in:

Directory browsers that show file information
Build systems that check file modification times
Backup systems that compare file metadata before deciding to copy

Implement SubFS when you need to enforce access boundaries at the file system level rather than through application logic. This is essential for:

Multi-tenant systems where users should only access their own files
Plugin systems where extensions need restricted file access
Template processors that handle untrusted content

Performance Optimization Techniques

When implementing these interfaces, focus on optimizations that align with your storage characteristics. For ReadFileFS, consider these patterns:

func (fs *CacheFS) ReadFile(name string) ([]byte, error) {
    // Check cache first
    if data, found := fs.cache.Get(name); found {
        return data, nil
    }

    // Read from underlying storage
    data, err := fs.underlying.ReadFile(name)
    if err != nil {
        return nil, err
    }

    // Cache for future reads
    fs.cache.Set(name, data)
    return data, nil
}

For StatFS, avoid expensive operations in the stat path:

func (fs *NetworkFS) Stat(name string) (fs.FileInfo, error) {
    // Use lightweight HEAD request instead of full GET
    resp, err := fs.client.Head(fs.urlFor(name))
    if err != nil {
        return nil, err
    }

    return &FileInfo{
        name: path.Base(name),
        size: resp.ContentLength,
        mode: fs.defaultMode,
        modTime: resp.LastModified,
    }, nil
}

Compatibility Considerations

When implementing specialized interfaces, ensure your implementations degrade gracefully. Code that depends on these interfaces should always check for their presence using type assertions and have fallback strategies.

Your file system should maintain consistent behavior whether callers use the specialized interfaces or the basic fs.FS methods:

func (fs *MyFS) ReadFile(name string) ([]byte, error) {
    // Specialized implementation
    return fs.optimizedRead(name)
}

func (fs *MyFS) Open(name string) (fs.File, error) {
    // Must return consistent results with ReadFile
    // when the file is read completely
    return fs.openFile(name)
}

The key principle is that implementing a specialized interface should never change the semantic behavior of your file system—it should only change the performance characteristics.

Combining Specialized Interfaces

The real power of these specialized interfaces emerges when you combine them thoughtfully. A well-designed file system can implement multiple interfaces to provide different optimization paths for different use cases.

Interface Composition Patterns

When implementing multiple interfaces, structure your file system to leverage the strengths of each:

type OptimizedFS struct {
    underlying fs.FS
    cache      map[string][]byte
    statCache  map[string]fs.FileInfo
}

// Implements ReadFileFS for fast full-file access
func (ofs *OptimizedFS) ReadFile(name string) ([]byte, error) {
    if data, exists := ofs.cache[name]; exists {
        return data, nil
    }

    data, err := fs.ReadFile(ofs.underlying, name)
    if err != nil {
        return nil, err
    }

    ofs.cache[name] = data
    return data, nil
}

// Implements StatFS for efficient metadata access
func (ofs *OptimizedFS) Stat(name string) (fs.FileInfo, error) {
    if info, exists := ofs.statCache[name]; exists {
        return info, nil
    }

    info, err := fs.Stat(ofs.underlying, name)
    if err != nil {
        return nil, err
    }

    ofs.statCache[name] = info
    return info, nil
}

// Implements SubFS for secure scoping
func (ofs *OptimizedFS) Sub(dir string) (fs.FS, error) {
    subFS, err := fs.Sub(ofs.underlying, dir)
    if err != nil {
        return nil, err
    }

    return &OptimizedFS{
        underlying: subFS,
        cache:      make(map[string][]byte),
        statCache:  make(map[string]fs.FileInfo),
    }, nil
}

This pattern creates a file system that provides optimized access through multiple paths while maintaining the security boundaries that SubFS provides.

Type Assertion Best Practices

When working with file systems that might implement multiple specialized interfaces, use type assertions strategically to access the most efficient path:

func efficientFileProcessor(fsys fs.FS, filename string) error {
    // Try the most efficient path first
    if readFS, ok := fsys.(fs.ReadFileFS); ok {
        data, err := readFS.ReadFile(filename)
        if err != nil {
            return err
        }
        return processFileData(data)
    }

    // Fall back to standard approach
    file, err := fsys.Open(filename)
    if err != nil {
        return err
    }
    defer file.Close()

    data, err := io.ReadAll(file)
    if err != nil {
        return err
    }

    return processFileData(data)
}

For metadata operations, establish a similar pattern:

func checkFileExists(fsys fs.FS, filename string) (bool, error) {
    // Use StatFS if available for efficiency
    if statFS, ok := fsys.(fs.StatFS); ok {
        _, err := statFS.Stat(filename)
        if err != nil {
            if errors.Is(err, fs.ErrNotExist) {
                return false, nil
            }
            return false, err
        }
        return true, nil
    }

    // Fall back to Open approach
    file, err := fsys.Open(filename)
    if err != nil {
        if errors.Is(err, fs.ErrNotExist) {
            return false, nil
        }
        return false, err
    }
    file.Close()
    return true, nil
}

The pattern here is always the same: check for the specialized interface, use it if available, then fall back to the basic fs.FS operations. This ensures your code works with any file system while taking advantage of optimizations when they're available.

Use Cases and Examples

These specialized interfaces solve real problems in production systems. Understanding when and how to apply them comes from seeing them in action across different domains.

Configuration Management Systems

Configuration management often involves reading many small files and checking their metadata frequently. A configuration system that implements all three specialized interfaces can dramatically improve startup times and runtime performance:

type ConfigFS struct {
    baseDir    string
    configData map[string][]byte
    metadata   map[string]fs.FileInfo
}

func (cfs *ConfigFS) ReadFile(name string) ([]byte, error) {
    // Configuration files are typically small and read frequently
    // Cache them aggressively
    if data, exists := cfs.configData[name]; exists {
        return data, nil
    }

    fullPath := filepath.Join(cfs.baseDir, name)
    data, err := os.ReadFile(fullPath)
    if err != nil {
        return nil, err
    }

    cfs.configData[name] = data
    return data, nil
}

func (cfs *ConfigFS) Stat(name string) (fs.FileInfo, error) {
    // Config systems often check modification times
    // to determine when to reload
    if info, exists := cfs.metadata[name]; exists {
        return info, nil
    }

    fullPath := filepath.Join(cfs.baseDir, name)
    info, err := os.Stat(fullPath)
    if err != nil {
        return nil, err
    }

    cfs.metadata[name] = info
    return info, nil
}

func (cfs *ConfigFS) Sub(dir string) (fs.FS, error) {
    // Allow scoped access to configuration sections
    // Useful for plugin systems or multi-tenant configs
    return &ConfigFS{
        baseDir:    filepath.Join(cfs.baseDir, dir),
        configData: make(map[string][]byte),
        metadata:   make(map[string]fs.FileInfo),
    }, nil
}

This pattern is particularly effective for systems that need to:

Read the same configuration files repeatedly
Check for configuration changes without full reloads
Provide isolated configuration views to different system components

Template File Systems

Template engines benefit significantly from these interfaces. Templates are typically small files that are read completely into memory, and template systems often need to check modification times for cache invalidation:

type TemplateFS struct {
    templates map[string]*template.Template
    sources   map[string][]byte
    modTimes  map[string]time.Time
    baseFS    fs.FS
}

func (tfs *TemplateFS) ReadFile(name string) ([]byte, error) {
    // Templates benefit from caching since they're parsed after reading
    if source, exists := tfs.sources[name]; exists {
        return source, nil
    }

    data, err := fs.ReadFile(tfs.baseFS, name)
    if err != nil {
        return nil, err
    }

    tfs.sources[name] = data
    return data, nil
}

func (tfs *TemplateFS) Stat(name string) (fs.FileInfo, error) {
    // Template systems need modification times for cache invalidation
    return fs.Stat(tfs.baseFS, name)
}

func (tfs *TemplateFS) Sub(dir string) (fs.FS, error) {
    // Create scoped template environments
    // Useful for user-specific or theme-specific templates
    subFS, err := fs.Sub(tfs.baseFS, dir)
    if err != nil {
        return nil, err
    }

    return &TemplateFS{
        templates: make(map[string]*template.Template),
        sources:   make(map[string][]byte),
        modTimes:  make(map[string]time.Time),
        baseFS:    subFS,
    }, nil
}

func (tfs *TemplateFS) GetTemplate(name string) (*template.Template, error) {
    // Check if template needs recompilation
    info, err := tfs.Stat(name)
    if err != nil {
        return nil, err
    }

    if tmpl, exists := tfs.templates[name]; exists {
        if cachedTime, exists := tfs.modTimes[name]; exists {
            if !info.ModTime().After(cachedTime) {
                return tmpl, nil
            }
        }
    }

    // Read and compile template
    source, err := tfs.ReadFile(name)
    if err != nil {
        return nil, err
    }

    tmpl, err := template.New(name).Parse(string(source))
    if err != nil {
        return nil, err
    }

    tfs.templates[name] = tmpl
    tfs.modTimes[name] = info.ModTime()
    return tmpl, nil
}

Secure File Serving

Web servers that serve static files can use these interfaces to create secure, efficient file serving systems:

type SecureFileServer struct {
    allowedExts map[string]bool
    baseFS      fs.FS
}

func (sfs *SecureFileServer) ServeFile(w http.ResponseWriter, r *http.Request, filename string) {
    // Use Stat to check file properties before opening
    if statFS, ok := sfs.baseFS.(fs.StatFS); ok {
        info, err := statFS.Stat(filename)
        if err != nil {
            http.NotFound(w, r)
            return
        }

        // Security check: ensure it's a regular file
        if !info.Mode().IsRegular() {
            http.Error(w, "Forbidden", http.StatusForbidden)
            return
        }

        // Check file extension
        ext := filepath.Ext(filename)
        if !sfs.allowedExts[ext] {
            http.Error(w, "Forbidden", http.StatusForbidden)
            return
        }

        // Set appropriate headers
        w.Header().Set("Content-Length", fmt.Sprintf("%d", info.Size()))
        w.Header().Set("Last-Modified", info.ModTime().UTC().Format(http.TimeFormat))
    }

    // Use ReadFileFS for efficient serving of small files
    if readFS, ok := sfs.baseFS.(fs.ReadFileFS); ok {
        data, err := readFS.ReadFile(filename)
        if err != nil {
            http.NotFound(w, r)
            return
        }
        w.Write(data)
        return
    }

    // Fall back to streaming for large files
    file, err := sfs.baseFS.Open(filename)
    if err != nil {
        http.NotFound(w, r)
        return
    }
    defer file.Close()

    io.Copy(w, file)
}

func NewSecureFileServer(baseDir string, allowedPaths []string) *SecureFileServer {
    // Create sub-filesystems for each allowed path
    // This prevents path traversal attacks at the filesystem level
    var combinedFS fs.FS = os.DirFS(baseDir)

    // In a real implementation, you might combine multiple Sub calls
    // or use a more sophisticated approach to handle multiple allowed paths

    return &SecureFileServer{
        allowedExts: map[string]bool{
            ".html": true, ".css": true, ".js": true,
            ".png": true, ".jpg": true, ".jpeg": true,
        },
        baseFS: combinedFS,
    }
}

These examples demonstrate how the specialized interfaces work together to solve real-world problems. The key insight is that each interface addresses a specific performance or security concern, and combining them creates file systems that are both efficient and safe.

Pattern Matching with Glob: Finding Files by Pattern 5/9

Rez Moss — Sat, 28 Jun 2025 15:44:00 +0000

Glob Function Fundamentals

The filepath.Glob function in Go provides a powerful way to find files and directories using pattern matching. At its core, glob matching allows you to specify file patterns using wildcards and special characters, returning a slice of paths that match your criteria.

package main

import (
    "fmt"
    "path/filepath"
)

func main() {
    matches, err := filepath.Glob("*.txt")
    if err != nil {
        fmt.Printf("Error: %v\n", err)
        return
    }

    for _, match := range matches {
        fmt.Println(match)
    }
}

The glob function operates on the current working directory by default, but you can specify absolute or relative paths in your patterns. It returns two values: a slice of matching file paths and an error. The error handling is straightforward - the only error you'll encounter is filepath.ErrBadPattern, which occurs when your pattern syntax is malformed.

// Valid patterns
matches1, _ := filepath.Glob("/home/user/*.log")
matches2, _ := filepath.Glob("../config/*.json")
matches3, _ := filepath.Glob("data/2023/*/report.csv")

// Invalid pattern that would return ErrBadPattern
matches4, err := filepath.Glob("[invalid")
if err == filepath.ErrBadPattern {
    fmt.Println("Pattern syntax error")
}

Under the hood, filepath.Glob uses the same pattern matching logic as path.Match, but extends it to work with file system hierarchies. While path.Match operates on individual path segments, glob can traverse directory structures and match patterns across multiple levels.

The function doesn't follow symbolic links and only returns paths that actually exist in the file system. This behavior ensures that your matches correspond to real, accessible files and directories, making it reliable for file operations that follow the glob call.

Pattern Syntax Deep Dive

Understanding glob pattern syntax is essential for crafting precise file matching expressions. The pattern language builds on familiar shell globbing conventions but has specific rules and limitations you need to master.

Wildcard Characters and Their Meanings

The asterisk (*) is your most versatile tool, matching any sequence of characters except the path separator. This makes it perfect for matching file names with varying content but predictable structure.

// Match all .go files in current directory
matches, _ := filepath.Glob("*.go")

// Match files starting with "test_"
matches, _ = filepath.Glob("test_*.log")

// Match files with any name but specific extension
matches, _ = filepath.Glob("*.json")

The question mark (?) matches exactly one character, giving you precise control over variable positions in file names.

// Match files like "day1.txt", "day2.txt", but not "day10.txt"
matches, _ := filepath.Glob("day?.txt")

// Match "file_a.dat", "file_b.dat", etc.
matches, _ = filepath.Glob("file_?.dat")

// Combine for more complex patterns
matches, _ = filepath.Glob("backup_????_??.sql")

Character Classes and Ranges

Square brackets define character classes, allowing you to specify sets or ranges of acceptable characters at a position.

// Match files ending with digits
matches, _ := filepath.Glob("report_[0-9].txt")

// Match specific characters
matches, _ = filepath.Glob("config_[abc].yaml")

// Match uppercase letters
matches, _ = filepath.Glob("LOG_[A-Z].txt")

// Combine ranges and specific characters
matches, _ = filepath.Glob("file_[0-9a-f].hex")

Character classes support negation using the caret (^) as the first character, matching anything except the specified set.

// Match files not ending with digits
matches, _ := filepath.Glob("temp_[^0-9].tmp")

// Match non-alphabetic characters
matches, _ = filepath.Glob("data_[^a-zA-Z].csv")

Escaping Special Characters

When you need to match literal wildcard characters, Go's glob implementation uses backslash escaping, though the behavior depends on your operating system.

// On Unix systems, escape with backslash
matches, _ := filepath.Glob("file\\*.txt")  // Matches "file*.txt" literally

// Windows handling varies - test your patterns
matches, _ = filepath.Glob("backup\\[daily\\].zip")

For cross-platform compatibility, consider using path.Match to test individual components when dealing with literal special characters, or restructure your file naming conventions to avoid conflicts with glob metacharacters.

The pattern syntax doesn't support regular expression features like quantifiers or alternation. Each pattern element has a specific, limited scope that keeps glob fast and predictable for file system operations.

GlobFS Interface Optimization

The io/fs package introduced the GlobFS interface in Go 1.16, enabling file systems to provide optimized glob implementations. This interface allows custom file systems to implement native pattern matching, potentially offering significant performance improvements over the default traversal-based approach.

type GlobFS interface {
    fs.FS
    Glob(pattern string) ([]string, error)
}

When you use fs.Glob with a file system that implements GlobFS, Go automatically detects and uses the native implementation instead of falling back to manual directory traversal.

import (
    "io/fs"
    "os"
)

func findConfigs(fsys fs.FS) ([]string, error) {
    // This will use native glob if fsys implements GlobFS
    return fs.Glob(fsys, "config/*.yaml")
}

// Using with os.DirFS
fsys := os.DirFS("/etc")
matches, err := fs.Glob(fsys, "nginx/*.conf")

Performance Implications

Native glob implementations can dramatically outperform manual traversal, especially on large directory structures. Database-backed file systems, network file systems, and compressed archives can leverage their internal indexing or query capabilities to filter matches without examining every file.

// Potentially slow: manual traversal
func slowGlob(root string) []string {
    var matches []string
    filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
        if matched, _ := filepath.Match("*.log", d.Name()); matched {
            matches = append(matches, path)
        }
        return nil
    })
    return matches
}

// Fast: uses native implementation if available
func fastGlob(fsys fs.FS) ([]string, error) {
    return fs.Glob(fsys, "**/*.log")
}

The performance difference becomes more pronounced with complex patterns or when searching deep directory hierarchies. Native implementations can often skip entire directory branches or use metadata indexes to accelerate matching.

Fallback Behavior

When a file system doesn't implement GlobFS, Go automatically falls back to a directory walking implementation. This fallback ensures your code works consistently across different file system types without requiring changes.

func robustGlobSearch(fsys fs.FS, pattern string) ([]string, error) {
    // This works regardless of whether fsys implements GlobFS
    matches, err := fs.Glob(fsys, pattern)
    if err != nil {
        return nil, fmt.Errorf("glob failed: %w", err)
    }

    // Additional filtering can be applied to results
    var filtered []string
    for _, match := range matches {
        if info, err := fs.Stat(fsys, match); err == nil && !info.IsDir() {
            filtered = append(filtered, match)
        }
    }

    return filtered, nil
}

The fallback implementation maintains the same semantic behavior as native implementations, ensuring that switching between file system types doesn't break your application logic. However, you should be aware of potential performance differences when working with large-scale file operations.

Testing with both native and fallback implementations helps ensure your glob patterns work correctly across different deployment scenarios and file system configurations.

Advanced Pattern Techniques

Complex file organization scenarios require sophisticated pattern matching approaches that go beyond basic wildcards. Mastering these advanced techniques enables you to handle intricate directory structures and implement precise filtering logic.

Multi-Level Directory Patterns

While standard glob patterns don't support recursive matching with **, you can achieve multi-level directory traversal by combining glob with directory walking or using strategic pattern construction.

func findNestedConfigs(root string) ([]string, error) {
    var allMatches []string

    // Pattern for immediate subdirectories
    level1, _ := filepath.Glob(filepath.Join(root, "*", "*.conf"))
    allMatches = append(allMatches, level1...)

    // Pattern for second-level subdirectories
    level2, _ := filepath.Glob(filepath.Join(root, "*", "*", "*.conf"))
    allMatches = append(allMatches, level2...)

    // Pattern for third-level subdirectories
    level3, _ := filepath.Glob(filepath.Join(root, "*", "*", "*", "*.conf"))
    allMatches = append(allMatches, level3...)

    return allMatches, nil
}

For truly recursive searching, combine glob with filepath.WalkDir to apply pattern matching at each directory level:

func recursiveGlob(root, pattern string) ([]string, error) {
    var matches []string

    err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return nil // Skip problematic directories
        }

        if !d.IsDir() {
            if matched, _ := filepath.Match(pattern, d.Name()); matched {
                matches = append(matches, path)
            }
        }
        return nil
    })

    return matches, err
}

Complex Filtering Scenarios

Real-world applications often require filtering based on multiple criteria. You can chain glob operations or combine them with additional validation logic:

func findRecentLogs(logDir string, days int) ([]string, error) {
    // First, find all potential log files
    candidates, err := filepath.Glob(filepath.Join(logDir, "*.log"))
    if err != nil {
        return nil, err
    }

    // Filter by modification time
    cutoff := time.Now().AddDate(0, 0, -days)
    var recent []string

    for _, candidate := range candidates {
        info, err := os.Stat(candidate)
        if err != nil {
            continue
        }

        if info.ModTime().After(cutoff) {
            recent = append(recent, candidate)
        }
    }

    return recent, nil
}

Pattern composition allows you to build complex selection criteria by combining multiple glob results:

func findDataFiles(baseDir string) (map[string][]string, error) {
    results := make(map[string][]string)

    // Find CSV files
    csvFiles, _ := filepath.Glob(filepath.Join(baseDir, "*.csv"))
    results["csv"] = csvFiles

    // Find JSON files with specific naming
    jsonFiles, _ := filepath.Glob(filepath.Join(baseDir, "data_*.json"))
    results["json"] = jsonFiles

    // Find backup files (multiple extensions)
    backups := make([]string, 0)
    for _, ext := range []string{"bak", "backup", "old"} {
        pattern := filepath.Join(baseDir, "*."+ext)
        matches, _ := filepath.Glob(pattern)
        backups = append(backups, matches...)
    }
    results["backups"] = backups

    return results, nil
}

Combining Glob with Other Operations

Effective file processing often requires combining glob results with sorting, filtering, or transformation operations:

func processLatestBackups(backupDir string, limit int) error {
    // Find all backup files
    backups, err := filepath.Glob(filepath.Join(backupDir, "backup_*.sql"))
    if err != nil {
        return err
    }

    // Sort by modification time (newest first)
    sort.Slice(backups, func(i, j int) bool {
        info1, _ := os.Stat(backups[i])
        info2, _ := os.Stat(backups[j])
        return info1.ModTime().After(info2.ModTime())
    })

    // Process only the most recent files
    processCount := limit
    if len(backups) < limit {
        processCount = len(backups)
    }

    for i := 0; i < processCount; i++ {
        if err := processBackupFile(backups[i]); err != nil {
            return fmt.Errorf("failed to process %s: %w", backups[i], err)
        }
    }

    return nil
}

These advanced techniques enable you to build robust file discovery systems that can handle complex organizational schemes and varying requirements across different deployment environments.

Error Resilience

Glob operations interact with the file system, making them susceptible to various runtime conditions. Understanding Go's approach to error handling in glob functions helps you build resilient applications that gracefully handle both pattern validation issues and file system anomalies.

I/O Error Ignoring Behavior

The filepath.Glob function follows a pragmatic approach to I/O errors during directory traversal. When it encounters permission denied errors, temporarily unavailable directories, or other transient file system issues, it silently continues processing rather than terminating the entire operation.

func demonstrateErrorHandling() {
    // This will return matches from accessible directories
    // and silently skip directories with permission issues
    matches, err := filepath.Glob("/var/log/*/*.log")

    // err will only be non-nil for pattern syntax errors
    if err != nil {
        fmt.Printf("Pattern error: %v\n", err)
        return
    }

    // matches contains all accessible files, even if some directories were skipped
    fmt.Printf("Found %d log files\n", len(matches))
}

This behavior ensures that partial file system access doesn't prevent your application from processing available data. However, you won't receive notifications about skipped directories, so consider logging or monitoring when complete directory access is critical:

func auditableGlob(pattern string) ([]string, []error) {
    matches, err := filepath.Glob(pattern)
    if err != nil {
        return nil, []error{err}
    }

    // Verify access to parent directories if needed
    var accessErrors []error
    parentDirs := make(map[string]bool)

    for _, match := range matches {
        dir := filepath.Dir(match)
        if !parentDirs[dir] {
            if _, err := os.Stat(dir); err != nil {
                accessErrors = append(accessErrors, fmt.Errorf("directory access issue: %s: %w", dir, err))
            }
            parentDirs[dir] = true
        }
    }

    return matches, accessErrors
}

Pattern Validation

The only error that filepath.Glob explicitly returns is filepath.ErrBadPattern, which occurs when your pattern contains malformed character classes or invalid escape sequences.

func validateAndExecuteGlob(pattern string) ([]string, error) {
    // Test the pattern first if you want explicit validation
    _, err := filepath.Match(pattern, "test")
    if err == filepath.ErrBadPattern {
        return nil, fmt.Errorf("invalid pattern syntax: %s", pattern)
    }

    // Execute the glob operation
    matches, err := filepath.Glob(pattern)
    if err == filepath.ErrBadPattern {
        return nil, fmt.Errorf("pattern validation failed during glob: %w", err)
    }

    return matches, nil
}

Common pattern validation errors include unclosed character classes, invalid ranges, and platform-specific escape sequence issues:

func handlePatternErrors() {
    invalidPatterns := []string{
        "[abc",           // Unclosed character class
        "[z-a]",         // Invalid range (reverse order)
        "file\\",        // Trailing escape (platform-dependent)
    }

    for _, pattern := range invalidPatterns {
        _, err := filepath.Glob(pattern)
        if err == filepath.ErrBadPattern {
            fmt.Printf("Invalid pattern detected: %s\n", pattern)
        }
    }
}

For robust applications, implement pattern validation before executing glob operations, especially when patterns come from user input or configuration files:

func safeGlob(userPattern string) ([]string, error) {
    // Sanitize and validate user input
    if strings.Contains(userPattern, "..") {
        return nil, fmt.Errorf("directory traversal not allowed")
    }

    // Test pattern validity
    if _, err := filepath.Match(userPattern, ""); err != nil {
        return nil, fmt.Errorf("invalid pattern format: %w", err)
    }

    // Execute with confidence
    matches, err := filepath.Glob(userPattern)
    if err != nil {
        return nil, fmt.Errorf("glob execution failed: %w", err)
    }

    return matches, nil
}

This error resilience strategy ensures your applications remain stable while providing meaningful feedback when pattern construction issues occur, maintaining a clear distinction between syntax errors and runtime file system conditions.

Practical Examples

Real-world applications benefit from concrete examples that demonstrate glob patterns in common file management scenarios. These examples showcase patterns you'll frequently encounter in system administration, build processes, and application maintenance tasks.

Log File Collection

System administrators often need to collect log files across multiple applications and time periods. Glob patterns excel at identifying files based on naming conventions and directory structures.

func collectSystemLogs(logRoot string) (map[string][]string, error) {
    logCategories := make(map[string][]string)

    // Collect Apache/Nginx access logs
    webLogs, _ := filepath.Glob(filepath.Join(logRoot, "apache2", "access.log*"))
    nginxLogs, _ := filepath.Glob(filepath.Join(logRoot, "nginx", "access.log*"))
    logCategories["web"] = append(webLogs, nginxLogs...)

    // Collect application logs with date patterns
    appLogs, _ := filepath.Glob(filepath.Join(logRoot, "app", "application-????-??-??.log"))
    logCategories["application"] = appLogs

    // Collect error logs from various sources
    errorPatterns := []string{
        filepath.Join(logRoot, "*", "error.log"),
        filepath.Join(logRoot, "*", "*.err"),
        filepath.Join(logRoot, "errors", "*.log"),
    }

    var allErrors []string
    for _, pattern := range errorPatterns {
        matches, _ := filepath.Glob(pattern)
        allErrors = append(allErrors, matches...)
    }
    logCategories["errors"] = allErrors

    return logCategories, nil
}

For log rotation scenarios, you can target specific file generations or time ranges:

func findRecentRotatedLogs(logDir string, service string) ([]string, error) {
    // Match rotated logs: service.log, service.log.1, service.log.2, etc.
    basePattern := filepath.Join(logDir, service+".log")
    rotatedPattern := filepath.Join(logDir, service+".log.[0-9]")
    compressedPattern := filepath.Join(logDir, service+".log.[0-9].gz")

    var allLogs []string

    // Current log file
    if current, err := filepath.Glob(basePattern); err == nil {
        allLogs = append(allLogs, current...)
    }

    // Recent rotated files
    if rotated, err := filepath.Glob(rotatedPattern); err == nil {
        allLogs = append(allLogs, rotated...)
    }

    // Compressed archives
    if compressed, err := filepath.Glob(compressedPattern); err == nil {
        allLogs = append(allLogs, compressed...)
    }

    return allLogs, nil
}

Configuration File Discovery

Applications often need to locate configuration files across multiple possible locations, following standard directory conventions or deployment-specific layouts.

func discoverConfigFiles(appName string) (map[string]string, error) {
    configs := make(map[string]string)

    // Standard system locations
    systemPaths := []string{
        "/etc/" + appName + "/" + appName + ".conf",
        "/etc/" + appName + ".conf",
        "/usr/local/etc/" + appName + ".conf",
    }

    // User-specific locations
    homeDir, _ := os.UserHomeDir()
    userPaths := []string{
        filepath.Join(homeDir, "."+appName, "config"),
        filepath.Join(homeDir, "."+appName+"rc"),
        filepath.Join(homeDir, ".config", appName, "config.yaml"),
    }

    // Check each location
    allPaths := append(systemPaths, userPaths...)
    for _, path := range allPaths {
        if matches, err := filepath.Glob(path); err == nil && len(matches) > 0 {
            for _, match := range matches {
                if info, err := os.Stat(match); err == nil && !info.IsDir() {
                    configs[filepath.Base(filepath.Dir(match))] = match
                }
            }
        }
    }

    return configs, nil
}

Environment-specific configuration discovery handles different deployment scenarios:

func findEnvironmentConfigs(configDir, environment string) ([]string, error) {
    patterns := []string{
        // Environment-specific files
        filepath.Join(configDir, environment, "*.yaml"),
        filepath.Join(configDir, environment, "*.json"),

        // Files with environment suffix
        filepath.Join(configDir, "*-"+environment+".yaml"),
        filepath.Join(configDir, "*_"+environment+".json"),

        // Override files
        filepath.Join(configDir, "override", environment, "*"),
    }

    var allConfigs []string
    for _, pattern := range patterns {
        if matches, err := filepath.Glob(pattern); err == nil {
            allConfigs = append(allConfigs, matches...)
        }
    }

    return allConfigs, nil
}

Build Artifact Cleanup

Build systems generate numerous temporary files and artifacts that require periodic cleanup. Glob patterns help identify and remove these files safely.

func cleanBuildArtifacts(projectRoot string) error {
    // Define cleanup patterns for different artifact types
    cleanupPatterns := []string{
        // Compiled objects and libraries
        filepath.Join(projectRoot, "**", "*.o"),
        filepath.Join(projectRoot, "**", "*.so"),
        filepath.Join(projectRoot, "**", "*.dylib"),

        // Build directories
        filepath.Join(projectRoot, "build", "*"),
        filepath.Join(projectRoot, "dist", "*"),
        filepath.Join(projectRoot, "target", "*"),

        // Temporary files
        filepath.Join(projectRoot, "**", "*.tmp"),
        filepath.Join(projectRoot, "**", ".DS_Store"),
        filepath.Join(projectRoot, "**", "Thumbs.db"),

        // Cache directories
        filepath.Join(projectRoot, "node_modules", ".cache", "*"),
        filepath.Join(projectRoot, ".pytest_cache", "*"),
    }

    var removedCount int
    for _, pattern := range cleanupPatterns {
        matches, err := filepath.Glob(pattern)
        if err != nil {
            continue
        }

        for _, match := range matches {
            if err := os.RemoveAll(match); err == nil {
                removedCount++
            }
        }
    }

    fmt.Printf("Cleaned %d build artifacts\n", removedCount)
    return nil
}

Age-based cleanup combines glob matching with file modification times:

func cleanOldArtifacts(buildDir string, maxAge time.Duration) error {
    artifacts, err := filepath.Glob(filepath.Join(buildDir, "*"))
    if err != nil {
        return err
    }

    cutoff := time.Now().Add(-maxAge)
    var cleaned int

    for _, artifact := range artifacts {
        info, err := os.Stat(artifact)
        if err != nil {
            continue
        }

        if info.ModTime().Before(cutoff) {
            if err := os.RemoveAll(artifact); err == nil {
                cleaned++
            }
        }
    }

    fmt.Printf("Removed %d old artifacts\n", cleaned)
    return nil
}

These practical examples demonstrate how glob patterns integrate into real-world workflows, providing reliable file discovery and management capabilities across diverse application domains.

File System Walking with WalkDir: Recursive Tree Traversal 4/9

Rez Moss — Sat, 21 Jun 2025 15:36:00 +0000

WalkDir Function Comprehensive Guide

The filepath.WalkDir function represents Go's modern approach to recursive directory traversal, replacing the older filepath.Walk function with improved performance and cleaner interface design. Understanding its mechanics is essential for any developer working with file system operations at scale.

Function Signature and Parameters

func WalkDir(root string, fn WalkDirFunc) error

The function signature deliberately keeps things simple. The root parameter accepts any valid file system path - whether it points to a file or directory. When you pass a file path, WalkDir processes only that single file. Directory paths trigger recursive traversal of the entire subtree.

The fn parameter expects a function matching the WalkDirFunc signature:

type WalkDirFunc func(path string, d DirEntry, err error) error

This callback executes for every file and directory encountered during traversal. The function receives the full path, a DirEntry interface providing file metadata, and any error that occurred while accessing the entry.

Lexical Ordering Guarantees

WalkDir provides deterministic traversal through lexical ordering. Within each directory, entries are processed in sorted order by name. This predictability proves crucial for testing and debugging file system operations.

// Directory structure:
// /project/
//   ├── README.md
//   ├── main.go
//   └── utils/
//       ├── helper.go
//       └── parser.go

// Traversal order:
// 1. /project/README.md
// 2. /project/main.go  
// 3. /project/utils/
// 4. /project/utils/helper.go
// 5. /project/utils/parser.go

The lexical ordering applies only within individual directories. Parent directories are always processed before their children, but sibling directories follow alphabetical order.

Memory Usage Considerations

WalkDir optimizes memory usage through several design decisions. Unlike filepath.Walk, it uses DirEntry instead of FileInfo, avoiding expensive stat system calls until you explicitly request detailed file information.

// Memory-efficient approach
err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
    // Only call Info() when you need detailed metadata
    if strings.HasSuffix(path, ".log") {
        info, err := d.Info()
        if err != nil {
            return err
        }
        // Now you have full FileInfo
        processLogFile(path, info)
    }
    return nil
})

The function processes one directory at a time, reading directory entries incrementally rather than loading entire directory trees into memory. This approach scales well even with deeply nested directory structures containing thousands of files.

For large traversals, be mindful of callback function allocations. Avoid creating unnecessary string concatenations or slice allocations within the callback, as these multiply across thousands of file system entries.

WalkDirFunc Callback Patterns

The WalkDirFunc callback serves as your primary interface for processing file system entries during traversal. Mastering its parameter handling and return value semantics gives you precise control over the walking behavior.

Function Parameters: path, DirEntry, error

Each callback invocation receives three parameters that work together to provide complete context about the current file system entry.

The path parameter contains the full file path from the root. This path uses the operating system's native separator and includes the original root prefix:

filepath.WalkDir("/home/user/projects", func(path string, d fs.DirEntry, err error) error {
    // path examples:
    // "/home/user/projects"
    // "/home/user/projects/main.go"
    // "/home/user/projects/src/utils.go"

    // Extract relative path if needed
    relPath, _ := filepath.Rel("/home/user/projects", path)
    return nil
})

The DirEntry parameter provides efficient access to basic file metadata without requiring expensive system calls. Use its methods to check file types and names:

func processEntry(path string, d fs.DirEntry, err error) error {
    if d.IsDir() {
        fmt.Printf("Directory: %s\n", d.Name())
        return nil
    }

    // Check file type without calling Info()
    if d.Type()&fs.ModeSymlink != 0 {
        fmt.Printf("Symlink: %s\n", path)
        return nil
    }

    // Only call Info() when you need full metadata
    if strings.HasSuffix(d.Name(), ".large") {
        info, err := d.Info()
        if err != nil {
            return err
        }
        if info.Size() > 1024*1024 {
            fmt.Printf("Large file: %s (%d bytes)\n", path, info.Size())
        }
    }

    return nil
}

The error parameter indicates problems accessing the current entry. This error handling happens before your callback logic executes, allowing you to decide whether to continue or abort traversal.

Return Value Meanings and Control Flow

Your callback's return value directly controls traversal behavior. Understanding these return patterns enables sophisticated directory walking logic.

Returning nil continues normal traversal:

func normalProcessing(path string, d fs.DirEntry, err error) error {
    // Process the entry
    processFile(path)

    // Continue traversal
    return nil
}

Returning filepath.SkipDir skips the current directory's contents but continues traversing siblings:

func skipHiddenDirs(path string, d fs.DirEntry, err error) error {
    if d.IsDir() && strings.HasPrefix(d.Name(), ".") {
        return filepath.SkipDir  // Skip .git, .vscode, etc.
    }
    return nil
}

Returning filepath.SkipAll terminates the entire traversal immediately:

func findFirstMatch(target string) filepath.WalkDirFunc {
    return func(path string, d fs.DirEntry, err error) error {
        if d.Name() == target {
            fmt.Printf("Found: %s\n", path)
            return filepath.SkipAll  // Stop searching
        }
        return nil
    }
}

Any other error value stops traversal and propagates up to the WalkDir caller:

func strictProcessing(path string, d fs.DirEntry, err error) error {
    if err != nil {
        return fmt.Errorf("access error for %s: %w", path, err)
    }

    if err := processFile(path); err != nil {
        return fmt.Errorf("processing failed for %s: %w", path, err)
    }

    return nil
}

Error Propagation Strategies

Different applications require different error handling approaches. Consider these common patterns based on your fault tolerance requirements.

The fail-fast approach stops on any error:

func failFast(path string, d fs.DirEntry, err error) error {
    if err != nil {
        return err  // Propagate immediately
    }
    return processEntry(path, d)
}

Advanced Traversal Control

Fine-grained control over directory traversal enables efficient file system operations by avoiding unnecessary work. The key lies in understanding when and how to skip portions of the directory tree based on your specific requirements.

SkipDir vs SkipAll Usage

The distinction between filepath.SkipDir and filepath.SkipAll determines the scope of traversal interruption. Understanding their behavior prevents common mistakes in traversal logic.

SkipDir affects only the current directory when returned for a directory entry:

func skipLargeDirs(path string, d fs.DirEntry, err error) error {
    if d.IsDir() {
        // Check if directory should be skipped
        if d.Name() == "node_modules" || d.Name() == ".git" {
            fmt.Printf("Skipping directory: %s\n", path)
            return filepath.SkipDir
        }
    }

    // Process files normally
    fmt.Printf("Processing: %s\n", path)
    return nil
}

// Directory structure:
// /project/
//   ├── src/file1.go
//   ├── node_modules/  <- Skipped entirely
//   │   └── package/
//   ├── docs/
//   │   └── readme.md  <- Still processed
//   └── main.go

// Output:
// Processing: /project
// Processing: /project/src
// Processing: /project/src/file1.go
// Skipping directory: /project/node_modules
// Processing: /project/docs
// Processing: /project/docs/readme.md
// Processing: /project/main.go

SkipAll terminates the entire traversal regardless of where it's returned:

func findAndStop(target string) filepath.WalkDirFunc {
    return func(path string, d fs.DirEntry, err error) error {
        fmt.Printf("Visiting: %s\n", path)

        if d.Name() == target {
            fmt.Printf("Found target: %s\n", path)
            return filepath.SkipAll  // Stop everything
        }

        return nil
    }
}

// Usage: find first occurrence of "config.json"
err := filepath.WalkDir("/app", findAndStop("config.json"))
// Traversal stops immediately when config.json is found

Important: SkipDir has no effect when returned for file entries. Only directory entries can be skipped.

Conditional Directory Skipping

Complex applications often require dynamic skipping logic based on directory contents, depth, or external conditions. Implement these patterns using closure-captured state.

Depth-based skipping prevents traversal beyond a certain level:

func maxDepthWalker(maxDepth int) filepath.WalkDirFunc {
    rootDepth := -1

    return func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        // Calculate current depth
        if rootDepth == -1 {
            rootDepth = strings.Count(path, string(filepath.Separator))
        }

        currentDepth := strings.Count(path, string(filepath.Separator)) - rootDepth

        if d.IsDir() && currentDepth >= maxDepth {
            return filepath.SkipDir
        }

        fmt.Printf("Depth %d: %s\n", currentDepth, path)
        return nil
    }
}

// Usage: traverse only 2 levels deep
err := filepath.WalkDir("/project", maxDepthWalker(2))

Content-based skipping examines directory properties before entering:

func skipEmptyOrHidden(path string, d fs.DirEntry, err error) error {
    if err != nil {
        return err
    }

    if d.IsDir() {
        // Skip hidden directories
        if strings.HasPrefix(d.Name(), ".") {
            return filepath.SkipDir
        }

        // Skip empty directories
        entries, err := os.ReadDir(path)
        if err != nil {
            return err  // Can't read, don't skip
        }

        if len(entries) == 0 {
            fmt.Printf("Skipping empty directory: %s\n", path)
            return filepath.SkipDir
        }
    }

    return nil
}

Pattern-based skipping uses matching rules for directory names:

type SkipPattern struct {
    patterns []string
}

func (sp *SkipPattern) shouldSkip(name string) bool {
    for _, pattern := range sp.patterns {
        if matched, _ := filepath.Match(pattern, name); matched {
            return true
        }
    }
    return false
}

func (sp *SkipPattern) walkFunc(path string, d fs.DirEntry, err error) error {
    if err != nil {
        return err
    }

    if d.IsDir() && sp.shouldSkip(d.Name()) {
        return filepath.SkipDir
    }

    // Process entry
    return processEntry(path, d)
}

// Usage
skipper := &SkipPattern{
    patterns: []string{"temp*", "*_backup", "node_modules"},
}
err := filepath.WalkDir("/project", skipper.walkFunc)

Early Termination Patterns

Early termination patterns optimize performance by stopping traversal once specific conditions are met. These patterns are essential for search operations and resource-constrained environments.

The first-match pattern stops after finding the first occurrence:

func findFirst(predicate func(string, fs.DirEntry) bool) filepath.WalkDirFunc {
    found := false

    return func(path string, d fs.DirEntry, err error) error {
        if err != nil || found {
            return err
        }

        if predicate(path, d) {
            found = true
            fmt.Printf("First match: %s\n", path)
            return filepath.SkipAll
        }

        return nil
    }
}

// Find first .go file
predicate := func(path string, d fs.DirEntry) bool {
    return !d.IsDir() && strings.HasSuffix(path, ".go")
}
err := filepath.WalkDir("/src", findFirst(predicate))

The quota-based pattern stops after processing a certain number of entries:

func processWithQuota(quota int) filepath.WalkDirFunc {
    processed := 0

    return func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        if processed >= quota {
            return filepath.SkipAll
        }

        if !d.IsDir() {
            processed++
            fmt.Printf("Processing %d/%d: %s\n", processed, quota, path)
        }

        return nil
    }
}

The timeout-based pattern stops after a time limit:

func processWithTimeout(timeout time.Duration) filepath.WalkDirFunc {
    start := time.Now()

    return func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        if time.Since(start) > timeout {
            fmt.Println("Timeout reached, stopping traversal")
            return filepath.SkipAll
        }

        return processEntry(path, d)
    }
}

Error Handling in Tree Walking

File system traversal encounters various error conditions that require different handling strategies. The WalkDir function provides a two-phase error reporting mechanism that gives you fine-grained control over error recovery and propagation.

Two-Phase Error Reporting

WalkDir implements a sophisticated error handling model where errors can occur both during directory reading and individual entry access. Understanding this distinction is crucial for building robust file system tools.

The first phase occurs when WalkDir attempts to read a directory's contents. If this fails, your callback receives the directory path with a non-nil error parameter:

func handleDirectoryErrors(path string, d fs.DirEntry, err error) error {
    if err != nil {
        // This error occurred while reading the directory
        fmt.Printf("Directory read error for %s: %v\n", path, err)

        // Decide whether to continue or abort
        if os.IsPermission(err) {
            fmt.Printf("Skipping due to permissions: %s\n", path)
            return nil  // Continue with siblings
        }

        return err  // Abort on other errors
    }

    // Normal processing for successfully read entries
    return processEntry(path, d)
}

The second phase happens when individual entries within a readable directory have access problems. In this case, the callback receives the entry with its specific error:

func handleEntryErrors(path string, d fs.DirEntry, err error) error {
    if err != nil {
        // This error is specific to this entry
        fmt.Printf("Entry access error for %s: %v\n", path, err)

        // Log and continue with other entries
        logError(path, err)
        return nil
    }

    // Entry is accessible, process normally
    return processEntry(path, d)
}

Here's a comprehensive handler that distinguishes between error types:

type ErrorHandler struct {
    dirErrors   []error
    entryErrors []error
}

func (eh *ErrorHandler) handleBothPhases(path string, d fs.DirEntry, err error) error {
    if err != nil {
        // Determine error phase based on context
        if d == nil {
            // Directory reading failed
            eh.dirErrors = append(eh.dirErrors, 
                fmt.Errorf("directory %s: %w", path, err))

            if os.IsPermission(err) {
                return nil  // Skip but continue
            }
            return err  // Fail on critical errors
        } else {
            // Individual entry error
            eh.entryErrors = append(eh.entryErrors, 
                fmt.Errorf("entry %s: %w", path, err))
            return nil  // Continue with other entries
        }
    }

    return processEntry(path, d)
}

Pre-read vs Post-read Error Handling

The timing of error detection affects your handling strategy. Pre-read errors prevent access to directory contents entirely, while post-read errors affect individual entries after successful directory enumeration.

Pre-read errors typically indicate system-level issues:

func handlePreReadErrors(path string, d fs.DirEntry, err error) error {
    if err != nil && d == nil {
        // Pre-read error: couldn't enumerate directory
        switch {
        case os.IsPermission(err):
            fmt.Printf("Permission denied: %s\n", path)
            auditLog.LogDeniedAccess(path)
            return nil  // Continue traversal

        case os.IsNotExist(err):
            fmt.Printf("Path disappeared: %s\n", path)
            return nil  // Handle race conditions gracefully

        case errors.Is(err, syscall.ELOOP):
            fmt.Printf("Symlink loop detected: %s\n", path)
            return nil  // Skip problematic symlinks

        default:
            return fmt.Errorf("critical directory error: %w", err)
        }
    }

    return processNormally(path, d, err)
}

Post-read errors occur after successful directory reading but indicate problems with specific entries:

func handlePostReadErrors(path string, d fs.DirEntry, err error) error {
    if err != nil && d != nil {
        // Post-read error: entry exists but has problems
        fmt.Printf("Entry problem %s: %v\n", path, err)

        // Try to extract what information we can
        if d.IsDir() {
            fmt.Printf("  Problematic directory, skipping contents\n")
        } else {
            fmt.Printf("  Problematic file, logging for later review\n")
            problemFiles.Add(path, err)
        }

        return nil  // Continue with other entries
    }

    return processNormally(path, d, err)
}

Recovery Strategies

Different applications require different recovery approaches when file system errors occur. Implement recovery strategies based on your application's fault tolerance requirements.

The retry strategy attempts to recover from transient errors:

func withRetry(maxRetries int, backoff time.Duration) filepath.WalkDirFunc {
    return func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            for attempt := 0; attempt < maxRetries; attempt++ {
                fmt.Printf("Retry %d for %s\n", attempt+1, path)
                time.Sleep(backoff * time.Duration(attempt+1))

                // Attempt to re-read the problematic entry
                if d == nil {
                    // Directory read failure, try again
                    entries, retryErr := os.ReadDir(path)
                    if retryErr == nil {
                        fmt.Printf("Retry successful for directory %s\n", path)
                        // Process recovered entries
                        for _, entry := range entries {
                            entryPath := filepath.Join(path, entry.Name())
                            if procErr := processEntry(entryPath, entry); procErr != nil {
                                return procErr
                            }
                        }
                        return filepath.SkipDir  // Already processed
                    }
                } else {
                    // Entry access failure, try to get info again
                    if _, retryErr := d.Info(); retryErr == nil {
                        fmt.Printf("Retry successful for entry %s\n", path)
                        return processEntry(path, d)
                    }
                }
            }

            // All retries failed
            return fmt.Errorf("failed after %d retries: %w", maxRetries, err)
        }

        return processEntry(path, d)
    }
}

The graceful degradation strategy continues operation with reduced functionality:

type GracefulWalker struct {
    processedCount int
    errorCount     int
    maxErrors      int
}

func (gw *GracefulWalker) walkWithDegradation(path string, d fs.DirEntry, err error) error {
    if err != nil {
        gw.errorCount++

        // Log error but continue
        fmt.Printf("Error %d: %s - %v\n", gw.errorCount, path, err)

        // Abort if too many errors
        if gw.errorCount > gw.maxErrors {
            return fmt.Errorf("too many errors (%d), aborting", gw.errorCount)
        }

        return nil  // Continue despite error
    }

    gw.processedCount++
    return processEntry(path, d)
}

The error isolation strategy quarantines problematic areas while continuing elsewhere:

type IsolatingWalker struct {
    quarantinedPaths map[string]error
    mutex           sync.RWMutex
}

func (iw *IsolatingWalker) walkWithIsolation(path string, d fs.DirEntry, err error) error {
    if err != nil {
        iw.mutex.Lock()
        iw.quarantinedPaths[path] = err
        iw.mutex.Unlock()

        fmt.Printf("Quarantined: %s (%v)\n", path, err)

        // Skip this subtree but continue elsewhere
        if d != nil && d.IsDir() {
            return filepath.SkipDir
        }
        return nil
    }

    // Check if we're in a quarantined subtree
    iw.mutex.RLock()
    for quarantined := range iw.quarantinedPaths {
        if strings.HasPrefix(path, quarantined) {
            iw.mutex.RUnlock()
            return filepath.SkipDir
        }
    }
    iw.mutex.RUnlock()

    return processEntry(path, d)
}

The error collection approach gathers all errors for batch reporting:

type ErrorCollector struct {
    errors []error
}

func (ec *ErrorCollector) collectErrors(path string, d fs.DirEntry, err error) error {
    if err != nil {
        ec.errors = append(ec.errors, fmt.Errorf("%s: %w", path, err))
        return nil  // Continue despite error
    }

    if procErr := processEntry(path, d); procErr != nil {
        ec.errors = append(ec.errors, fmt.Errorf("%s: %w", path, procErr))
    }

    return nil
}

The selective error handling approach treats different error types differently:


func selectiveHandling(path string, d fs.DirEntry, err error) error {
    if err != nil {
        // Log permission errors but continue
        if os.IsPermission(err) {
            log.Printf("Permission denied: %s", path)
            return nil
        }

        // Fail on other errors
        return err
    }

    return processEntry(path, d)
}

Symbolic Link Behavior

WalkDir implements specific symbolic link handling policies that differ significantly from traditional file system traversal tools. Understanding these behaviors prevents security vulnerabilities and infinite loops while maintaining predictable traversal characteristics.

Root Symlink Resolution

When the root path passed to WalkDir is itself a symbolic link, the function resolves it before beginning traversal. This resolution applies only to the root path and establishes the actual starting point for the walk operation.

// File system setup:
// /real/project/
//   ├── src/main.go
//   └── docs/readme.md
// /links/current -> /real/project

func demonstrateRootResolution() {
    fmt.Println("Walking symlinked root:")

    err := filepath.WalkDir("/links/current", func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        fmt.Printf("Path: %s\n", path)
        return nil
    })

    if err != nil {
        log.Fatal(err)
    }
}

// Output:
// Path: /links/current          <- Root symlink kept in paths
// Path: /links/current/src
// Path: /links/current/src/main.go
// Path: /links/current/docs  
// Path: /links/current/docs/readme.md

Notice that while WalkDir resolves the root symlink to determine what to traverse, it preserves the original symlink path in the callback parameters. This behavior maintains path consistency for your application logic while ensuring the traversal reaches the intended content.

The resolution only affects traversal scope, not path reporting:

func analyzeRootSymlink() {
    symlinkRoot := "/links/project-v2"  // -> /real/project-v2

    err := filepath.WalkDir(symlinkRoot, func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        // All paths maintain the symlink prefix
        if strings.HasPrefix(path, symlinkRoot) {
            fmt.Printf("Consistent path: %s\n", path)
        }

        // But the actual traversal follows the resolved target
        if path == symlinkRoot {
            realPath, _ := filepath.EvalSymlinks(path)
            fmt.Printf("Root %s resolves to %s\n", path, realPath)
        }

        return nil
    })
}

Root symlink resolution has security implications for applications that perform path-based access control:

func secureWalk(allowedRoot string) filepath.WalkDirFunc {
    // Resolve the allowed root to handle symlinks
    resolvedRoot, err := filepath.EvalSymlinks(allowedRoot)
    if err != nil {
        resolvedRoot = allowedRoot  // Fallback to original
    }

    return func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        // Verify we're still within allowed boundaries
        resolvedPath, _ := filepath.EvalSymlinks(path)
        if !strings.HasPrefix(resolvedPath, resolvedRoot) {
            return fmt.Errorf("path escaped allowed root: %s", path)
        }

        return processEntry(path, d)
    }
}

Directory Symlink Non-Following

Unlike root symlinks, WalkDir does not follow symbolic links to directories encountered during traversal. This policy prevents infinite loops and maintains bounded traversal behavior.

// File system setup:
// /project/
//   ├── src/
//   │   ├── main.go
//   │   └── vendor -> ../../shared/vendor
//   ├── docs/
//   │   └── api -> ../external/api-docs
//   └── backup -> /archive/project-backup

func demonstrateSymlinkNonFollowing() {
    err := filepath.WalkDir("/project", func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        if d.Type()&fs.ModeSymlink != 0 {
            if d.IsDir() {
                fmt.Printf("Directory symlink (not followed): %s\n", path)
            } else {
                fmt.Printf("File symlink: %s\n", path)
            }
        } else {
            fmt.Printf("Regular entry: %s\n", path)
        }

        return nil
    })
}

// Output:
// Regular entry: /project
// Regular entry: /project/src
// Regular entry: /project/src/main.go
// Directory symlink (not followed): /project/src/vendor
// Regular entry: /project/docs
// Directory symlink (not followed): /project/docs/api
// Directory symlink (not followed): /project/backup

The non-following behavior applies only to directory symlinks. File symlinks are reported but not dereferenced:

func handleSymlinks(path string, d fs.DirEntry, err error) error {
    if err != nil {
        return err
    }

    if d.Type()&fs.ModeSymlink != 0 {
        // Get symlink target for informational purposes
        target, linkErr := os.Readlink(path)
        if linkErr != nil {
            fmt.Printf("Broken symlink: %s\n", path)
            return nil
        }

        if d.IsDir() {
            fmt.Printf("Directory symlink %s -> %s (not traversed)\n", path, target)
            // WalkDir will not descend into this directory
        } else {
            fmt.Printf("File symlink %s -> %s\n", path, target)
            // Process as needed, but content comes from target
        }
    } else {
        fmt.Printf("Regular %s: %s\n", 
            map[bool]string{true: "directory", false: "file"}[d.IsDir()], 
            path)
    }

    return nil
}

This behavior protects against common symlink attack patterns:

// Dangerous symlink structure that WalkDir handles safely:
// /tmp/safe/
//   ├── data/
//   │   └── important.txt
//   ├── loop -> loop          <- Self-referencing symlink
//   ├── escape -> /etc        <- Directory escape attempt
//   └── cycle -> ../safe      <- Parent directory cycle

func safeTraversal() {
    count := 0
    err := filepath.WalkDir("/tmp/safe", func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        count++
        if count > 1000 {  // Safety check
            return fmt.Errorf("traversal too deep, possible loop")
        }

        fmt.Printf("Safe traversal: %s\n", path)
        return nil
    })

    fmt.Printf("Completed safely with %d entries\n", count)
}

When you need to follow directory symlinks, implement custom logic with cycle detection:

type SymlinkFollower struct {
    visited map[string]bool
    mutex   sync.RWMutex
}

func (sf *SymlinkFollower) followingWalk(root string) error {
    sf.visited = make(map[string]bool)
    return sf.walkWithFollowing(root)
}

func (sf *SymlinkFollower) walkWithFollowing(root string) error {
    return filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        // Check for directory symlinks
        if d.IsDir() && d.Type()&fs.ModeSymlink != 0 {
            target, err := filepath.EvalSymlinks(path)
            if err != nil {
                return nil  // Skip broken symlinks
            }

            sf.mutex.Lock()
            if sf.visited[target] {
                sf.mutex.Unlock()
                fmt.Printf("Cycle detected, skipping: %s -> %s\n", path, target)
                return nil
            }
            sf.visited[target] = true
            sf.mutex.Unlock()

            // Recursively walk the symlink target
            fmt.Printf("Following directory symlink: %s -> %s\n", path, target)
            return sf.walkWithFollowing(target)
        }

        return processEntry(path, d)
    })
}

Real-World Applications

The practical value of WalkDir emerges through real-world implementations that solve common file system problems. These examples demonstrate how to combine the traversal patterns into production-ready tools.

File Search Implementation

Building efficient file search tools requires combining multiple WalkDir features: pattern matching, early termination, and smart filtering. Here's a comprehensive search implementation:

type FileSearcher struct {
    patterns    []string
    maxResults  int
    maxDepth    int
    skipHidden  bool
    caseSensitive bool
    results     []SearchResult
}

type SearchResult struct {
    Path     string
    Size     int64
    ModTime  time.Time
    IsDir    bool
}

func NewFileSearcher(patterns []string, options ...SearchOption) *FileSearcher {
    fs := &FileSearcher{
        patterns:      patterns,
        maxResults:    100,
        maxDepth:      -1,
        caseSensitive: true,
        results:       make([]SearchResult, 0),
    }

    for _, opt := range options {
        opt(fs)
    }

    return fs
}

func (fs *FileSearcher) Search(root string) ([]SearchResult, error) {
    rootDepth := strings.Count(root, string(filepath.Separator))

    err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            // Log but continue on permission errors
            if os.IsPermission(err) {
                return nil
            }
            return err
        }

        // Check depth limit
        if fs.maxDepth >= 0 {
            currentDepth := strings.Count(path, string(filepath.Separator)) - rootDepth
            if d.IsDir() && currentDepth > fs.maxDepth {
                return filepath.SkipDir
            }
        }

        // Skip hidden files/directories if requested
        if fs.skipHidden && strings.HasPrefix(d.Name(), ".") {
            if d.IsDir() {
                return filepath.SkipDir
            }
            return nil
        }

        // Check if we've hit the result limit
        if len(fs.results) >= fs.maxResults {
            return filepath.SkipAll
        }

        // Apply pattern matching
        if fs.matchesPattern(d.Name()) {
            info, err := d.Info()
            if err != nil {
                return nil  // Skip entries we can't stat
            }

            fs.results = append(fs.results, SearchResult{
                Path:    path,
                Size:    info.Size(),
                ModTime: info.ModTime(),
                IsDir:   d.IsDir(),
            })
        }

        return nil
    })

    return fs.results, err
}

func (fs *FileSearcher) matchesPattern(name string) bool {
    if !fs.caseSensitive {
        name = strings.ToLower(name)
    }

    for _, pattern := range fs.patterns {
        searchPattern := pattern
        if !fs.caseSensitive {
            searchPattern = strings.ToLower(pattern)
        }

        // Support both glob patterns and substring matching
        if matched, _ := filepath.Match(searchPattern, name); matched {
            return true
        }

        if strings.Contains(name, searchPattern) {
            return true
        }
    }

    return false
}

// Usage example
func demonstrateFileSearch() {
    searcher := NewFileSearcher(
        []string{"*.go", "*test*"},
        WithMaxResults(50),
        WithMaxDepth(3),
        WithSkipHidden(true),
    )

    results, err := searcher.Search("/project")
    if err != nil {
        log.Fatal(err)
    }

    for _, result := range results {
        fmt.Printf("Found: %s (%d bytes, %s)\n", 
            result.Path, result.Size, result.ModTime.Format("2006-01-02"))
    }
}

Directory Cleanup Tools

Cleanup tools require careful error handling and confirmation mechanisms to avoid data loss. This implementation provides safe cleanup with rollback capabilities:

type CleanupTool struct {
    dryRun      bool
    rules       []CleanupRule
    deletedSize int64
    deletedCount int
    errors      []error
}

type CleanupRule interface {
    ShouldDelete(path string, d fs.DirEntry, info os.FileInfo) bool
    Description() string
}

type AgeBasedRule struct {
    maxAge time.Duration
    pattern string
}

func (r *AgeBasedRule) ShouldDelete(path string, d fs.DirEntry, info os.FileInfo) bool {
    if d.IsDir() {
        return false  // Don't delete directories by age
    }

    if r.pattern != "" {
        if matched, _ := filepath.Match(r.pattern, d.Name()); !matched {
            return false
        }
    }

    return time.Since(info.ModTime()) > r.maxAge
}

func (r *AgeBasedRule) Description() string {
    return fmt.Sprintf("Files older than %v matching %s", r.maxAge, r.pattern)
}

type SizeBasedRule struct {
    minSize int64
    pattern string
}

func (r *SizeBasedRule) ShouldDelete(path string, d fs.DirEntry, info os.FileInfo) bool {
    if d.IsDir() {
        return false
    }

    if r.pattern != "" {
        if matched, _ := filepath.Match(r.pattern, d.Name()); !matched {
            return false
        }
    }

    return info.Size() > r.minSize
}

func (r *SizeBasedRule) Description() string {
    return fmt.Sprintf("Files larger than %d bytes matching %s", r.minSize, r.pattern)
}

func (ct *CleanupTool) AddRule(rule CleanupRule) {
    ct.rules = append(ct.rules, rule)
}

func (ct *CleanupTool) Clean(root string) error {
    fmt.Printf("Starting cleanup of %s (dry run: %v)\n", root, ct.dryRun)

    for _, rule := range ct.rules {
        fmt.Printf("Rule: %s\n", rule.Description())
    }

    err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            ct.errors = append(ct.errors, fmt.Errorf("access error %s: %w", path, err))
            return nil  // Continue despite errors
        }

        // Skip the root directory
        if path == root {
            return nil
        }

        // Get file info for rule evaluation
        info, err := d.Info()
        if err != nil {
            ct.errors = append(ct.errors, fmt.Errorf("stat error %s: %w", path, err))
            return nil
        }

        // Check if any rule applies
        shouldDelete := false
        var matchedRule CleanupRule

        for _, rule := range ct.rules {
            if rule.ShouldDelete(path, d, info) {
                shouldDelete = true
                matchedRule = rule
                break
            }
        }

        if shouldDelete {
            ct.deletedSize += info.Size()
            ct.deletedCount++

            if ct.dryRun {
                fmt.Printf("Would delete: %s (%d bytes) - %s\n", 
                    path, info.Size(), matchedRule.Description())
            } else {
                if err := os.Remove(path); err != nil {
                    ct.errors = append(ct.errors, fmt.Errorf("delete error %s: %w", path, err))
                } else {
                    fmt.Printf("Deleted: %s (%d bytes)\n", path, info.Size())
                }
            }

            // Skip directory contents if we deleted the directory
            if d.IsDir() {
                return filepath.SkipDir
            }
        }

        return nil
    })

    ct.printSummary()
    return err
}

func (ct *CleanupTool) printSummary() {
    action := "Would delete"
    if !ct.dryRun {
        action = "Deleted"
    }

    fmt.Printf("\nSummary:\n")
    fmt.Printf("%s %d files totaling %d bytes\n", action, ct.deletedCount, ct.deletedSize)

    if len(ct.errors) > 0 {
        fmt.Printf("Encountered %d errors:\n", len(ct.errors))
        for _, err := range ct.errors {
            fmt.Printf("  %v\n", err)
        }
    }
}

// Usage example
func demonstrateCleanup() {
    cleaner := &CleanupTool{dryRun: true}

    // Clean up old temporary files
    cleaner.AddRule(&AgeBasedRule{
        maxAge:  7 * 24 * time.Hour,  // 7 days
        pattern: "*.tmp",
    })

    // Clean up large log files
    cleaner.AddRule(&SizeBasedRule{
        minSize: 100 * 1024 * 1024,  // 100MB
        pattern: "*.log",
    })

    err := cleaner.Clean("/tmp/project")
    if err != nil {
        log.Fatal(err)
    }
}

File System Auditing

Auditing tools analyze file system structure and permissions to identify security issues and compliance violations:

type FileSystemAuditor struct {
    checks       []SecurityCheck
    findings     []SecurityFinding
    totalFiles   int
    totalDirs    int
    totalSize    int64
}

type SecurityCheck interface {
    Check(path string, d fs.DirEntry, info os.FileInfo) *SecurityFinding
    Name() string
}

type SecurityFinding struct {
    Check    string
    Path     string
    Severity string
    Message  string
    Details  map[string]interface{}
}

type PermissionCheck struct {
    maxPermissions os.FileMode
}

func (pc *PermissionCheck) Check(path string, d fs.DirEntry, info os.FileInfo) *SecurityFinding {
    if info.Mode().Perm() > pc.maxPermissions {
        return &SecurityFinding{
            Check:    pc.Name(),
            Path:     path,
            Severity: "Medium",
            Message:  "File has overly permissive permissions",
            Details: map[string]interface{}{
                "current":    info.Mode().Perm().String(),
                "maximum":    pc.maxPermissions.String(),
                "is_dir":     d.IsDir(),
            },
        }
    }
    return nil
}

func (pc *PermissionCheck) Name() string {
    return "Permission Check"
}

type SuidCheck struct{}

func (sc *SuidCheck) Check(path string, d fs.DirEntry, info os.FileInfo) *SecurityFinding {
    if !d.IsDir() && (info.Mode()&os.ModeSetuid != 0 || info.Mode()&os.ModeSetgid != 0) {
        severity := "High"
        if info.Mode()&os.ModeSetuid != 0 {
            severity = "Critical"
        }

        return &SecurityFinding{
            Check:    sc.Name(),
            Path:     path,
            Severity: severity,
            Message:  "File has setuid/setgid permissions",
            Details: map[string]interface{}{
                "setuid": info.Mode()&os.ModeSetuid != 0,
                "setgid": info.Mode()&os.ModeSetgid != 0,
                "mode":   info.Mode().String(),
            },
        }
    }
    return nil
}

func (sc *SuidCheck) Name() string {
    return "SUID/SGID Check"
}

type HiddenFileCheck struct {
    allowedHidden []string
}

func (hfc *HiddenFileCheck) Check(path string, d fs.DirEntry, info os.FileInfo) *SecurityFinding {
    if strings.HasPrefix(d.Name(), ".") {
        // Check if this hidden file is in the allowed list
        for _, allowed := range hfc.allowedHidden {
            if matched, _ := filepath.Match(allowed, d.Name()); matched {
                return nil
            }
        }

        return &SecurityFinding{
            Check:    hfc.Name(),
            Path:     path,
            Severity: "Low",
            Message:  "Unexpected hidden file found",
            Details: map[string]interface{}{
                "name":   d.Name(),
                "is_dir": d.IsDir(),
            },
        }
    }
    return nil
}

func (hfc *HiddenFileCheck) Name() string {
    return "Hidden File Check"
}

func (fsa *FileSystemAuditor) AddCheck(check SecurityCheck) {
    fsa.checks = append(fsa.checks, check)
}

func (fsa *FileSystemAuditor) Audit(root string) error {
    fmt.Printf("Starting security audit of %s\n", root)

    err := filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            finding := &SecurityFinding{
                Check:    "Access Check",
                Path:     path,
                Severity: "Medium",
                Message:  "Unable to access file system entry",
                Details:  map[string]interface{}{"error": err.Error()},
            }
            fsa.findings = append(fsa.findings, *finding)
            return nil  // Continue audit despite access errors
        }

        // Update statistics
        if d.IsDir() {
            fsa.totalDirs++
        } else {
            fsa.totalFiles++
            if info, err := d.Info(); err == nil {
                fsa.totalSize += info.Size()
            }
        }

        // Run security checks
        info, err := d.Info()
        if err != nil {
            return nil  // Skip checks if we can't get file info
        }

        for _, check := range fsa.checks {
            if finding := check.Check(path, d, info); finding != nil {
                fsa.findings = append(fsa.findings, *finding)
            }
        }

        return nil
    })

    fsa.generateReport()
    return err
}

func (fsa *FileSystemAuditor) generateReport() {
    fmt.Printf("\n=== Security Audit Report ===\n")
    fmt.Printf("Files scanned: %d\n", fsa.totalFiles)
    fmt.Printf("Directories scanned: %d\n", fsa.totalDirs)
    fmt.Printf("Total size: %d bytes\n", fsa.totalSize)
    fmt.Printf("Security findings: %d\n\n", len(fsa.findings))

    // Group findings by severity
    severityCount := make(map[string]int)
    for _, finding := range fsa.findings {
        severityCount[finding.Severity]++
    }

    for severity, count := range severityCount {
        fmt.Printf("%s: %d findings\n", severity, count)
    }

    fmt.Printf("\nDetailed findings:\n")
    for _, finding := range fsa.findings {
        fmt.Printf("[%s] %s: %s\n", finding.Severity, finding.Check, finding.Path)
        fmt.Printf("  %s\n", finding.Message)
        for key, value := range finding.Details {
            fmt.Printf("  %s: %v\n", key, value)
        }
        fmt.Println()
    }
}

// Usage example
func demonstrateAudit() {
    auditor := &FileSystemAuditor{}

    // Add security checks
    auditor.AddCheck(&PermissionCheck{maxPermissions: 0644})
    auditor.AddCheck(&SuidCheck{})
    auditor.AddCheck(&HiddenFileCheck{
        allowedHidden: []string{".git", ".gitignore", ".env"},
    })

    err := auditor.Audit("/var/www")
    if err != nil {
        log.Fatal(err)
    }
}

Directory Operations: ReadDir, DirEntry, and Navigation 3/9

Rez Moss — Sat, 14 Jun 2025 15:28:00 +0000

Directory Reading Fundamentals

Working with directories is a cornerstone of file system operations in Go. The standard library provides several approaches to read directory contents, each designed for different use cases and performance requirements.

The most straightforward way to read a directory is using the os.ReadDir function, which returns a slice of DirEntry objects representing the directory's contents. This function automatically sorts entries by filename, providing predictable iteration order:

entries, err := os.ReadDir("/path/to/directory")
if err != nil {
    log.Fatal(err)
}

for _, entry := range entries {
    fmt.Println(entry.Name())
}

The ReadDir function is built on top of the ReadDirFS interface, which defines the contract for any file system that can read directories. This interface enables you to work with different file system implementations uniformly:

type ReadDirFS interface {
    ReadDir(name string) ([]DirEntry, error)
}

Abstract file systems like embed.FS or custom implementations can satisfy this interface, allowing your code to work seamlessly across different storage backends. The fs.ReadDir function accepts any ReadDirFS implementation:

// Works with os.DirFS
fsys := os.DirFS("/")
entries, err := fs.ReadDir(fsys, "home/user")

// Also works with embedded file systems
//go:embed static
var staticFiles embed.FS
entries, err = fs.ReadDir(staticFiles, ".")

The DirEntry interface provides essential information about each directory entry without requiring expensive system calls for detailed file metadata. This design choice optimizes performance by deferring costly operations until explicitly requested:

type DirEntry interface {
    Name() string
    IsDir() bool
    Type() FileMode
    Info() (FileInfo, error)
}

Each DirEntry exposes the filename through Name(), directory status via IsDir(), and file type information through Type(). The Type() method returns file mode bits that indicate whether the entry is a regular file, directory, symbolic link, or other special file type.

Go guarantees that ReadDir returns entries sorted by filename in lexicographical order. This sorting behavior is consistent across all platforms and file systems, eliminating the need for manual sorting in most cases. The sorting uses Go's string comparison, which handles Unicode characters correctly but may not match locale-specific sorting expectations.

entries, _ := os.ReadDir(".")
// entries are guaranteed to be sorted by filename
for i := 1; i < len(entries); i++ {
    // This assertion will always be true
    assert(entries[i-1].Name() <= entries[i].Name())
}

This predictable ordering is particularly valuable when building tools that need consistent output across different environments or when implementing directory synchronization algorithms that rely on deterministic iteration order.

DirEntry vs FileInfo Comparison

Understanding the relationship between DirEntry and FileInfo is crucial for writing efficient directory operations. These interfaces serve different purposes and have distinct performance characteristics that affect how you should structure your code.

DirEntry provides lightweight access to basic file information without requiring additional system calls. The Name(), IsDir(), and Type() methods return data that's typically available from the initial directory read operation:

entry := entries[0]
name := entry.Name()        // No additional syscall
isDir := entry.IsDir()      // No additional syscall  
fileType := entry.Type()    // No additional syscall

In contrast, FileInfo contains comprehensive file metadata including size, modification time, and permissions. This information requires a separate stat system call, which DirEntry defers until you explicitly call the Info() method:

// This triggers a stat() syscall
info, err := entry.Info()
if err != nil {
    return err
}

size := info.Size()
modTime := info.ModTime()
mode := info.Mode()

The performance implications become significant when processing directories with many files. Consider this comparison when iterating through a directory with 1000 files:

// Efficient: Only basic info, ~1 syscall total
entries, _ := os.ReadDir(directory)
for _, entry := range entries {
    if entry.IsDir() {
        fmt.Println("Directory:", entry.Name())
    }
}

// Inefficient: Full info, ~1000 additional syscalls
entries, _ := os.ReadDir(directory)
for _, entry := range entries {
    info, _ := entry.Info()
    if info.IsDir() {
        fmt.Println("Directory:", info.Name())
    }
}

The lazy loading design means you should only call Info() when you actually need the additional metadata. Many common operations can be performed using just the DirEntry methods:

func filterDirectories(entries []fs.DirEntry) []fs.DirEntry {
    var dirs []fs.DirEntry
    for _, entry := range entries {
        // Efficient: uses cached information
        if entry.IsDir() {
            dirs = append(dirs, entry)
        }
    }
    return dirs
}

func calculateTotalSize(entries []fs.DirEntry) (int64, error) {
    var total int64
    for _, entry := range entries {
        if !entry.IsDir() {
            // Only call Info() when size is needed
            info, err := entry.Info()
            if err != nil {
                return 0, err
            }
            total += info.Size()
        }
    }
    return total, nil
}

The Type() method deserves special attention as it provides file type information that's more detailed than IsDir(). It returns FileMode bits that distinguish between regular files, directories, symbolic links, named pipes, and other special file types:

switch entry.Type() {
case fs.ModeDir:
    fmt.Println("Directory")
case fs.ModeSymlink:
    fmt.Println("Symbolic link")
case fs.ModeNamedPipe:
    fmt.Println("Named pipe")
case 0: // Regular file
    fmt.Println("Regular file")
default:
    fmt.Printf("Special file type: %v\n", entry.Type())
}

This type information is available without the performance cost of calling Info(), making it ideal for filtering operations that need to distinguish between different file types while maintaining high performance.

Directory Traversal Patterns

Effective directory traversal requires understanding various iteration patterns and filtering techniques. The approach you choose depends on whether you need shallow directory listing, recursive traversal, or selective processing based on file characteristics.

The most common pattern involves iterating through directory contents with basic filtering. You can filter entries by type, name patterns, or other criteria without triggering expensive system calls:

func listExecutables(dir string) error {
    entries, err := os.ReadDir(dir)
    if err != nil {
        return err
    }

    for _, entry := range entries {
        // Filter by file extension
        if !entry.IsDir() && strings.HasSuffix(entry.Name(), ".exe") {
            fmt.Println(entry.Name())
        }
    }
    return nil
}

func findHiddenFiles(dir string) ([]string, error) {
    entries, err := os.ReadDir(dir)
    if err != nil {
        return nil, err
    }

    var hidden []string
    for _, entry := range entries {
        // Filter by name pattern
        if strings.HasPrefix(entry.Name(), ".") {
            hidden = append(hidden, entry.Name())
        }
    }
    return hidden, nil
}

When you need more sophisticated filtering that requires file metadata, combine DirEntry filtering with selective use of Info():

func findLargeFiles(dir string, minSize int64) error {
    entries, err := os.ReadDir(dir)
    if err != nil {
        return err
    }

    for _, entry := range entries {
        // First filter: skip directories without syscall
        if entry.IsDir() {
            continue
        }

        // Second filter: check size only for regular files
        info, err := entry.Info()
        if err != nil {
            continue // Skip files with stat errors
        }

        if info.Size() > minSize {
            fmt.Printf("%s: %d bytes\n", entry.Name(), info.Size())
        }
    }
    return nil
}

For nested directory structures, you'll often need recursive traversal. The filepath.WalkDir function provides an efficient way to traverse directory trees using DirEntry:

func findAllGoFiles(root string) error {
    return filepath.WalkDir(root, func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        // Skip directories and non-Go files efficiently
        if d.IsDir() || !strings.HasSuffix(d.Name(), ".go") {
            return nil
        }

        fmt.Println(path)
        return nil
    })
}

You can also implement custom recursive traversal when you need more control over the traversal process:

func traverseDirectory(dir string, maxDepth int) error {
    return traverseRecursive(dir, 0, maxDepth)
}

func traverseRecursive(dir string, currentDepth, maxDepth int) error {
    if currentDepth > maxDepth {
        return nil
    }

    entries, err := os.ReadDir(dir)
    if err != nil {
        return err
    }

    for _, entry := range entries {
        path := filepath.Join(dir, entry.Name())

        // Process current entry
        fmt.Printf("%s%s\n", strings.Repeat("  ", currentDepth), entry.Name())

        // Recurse into subdirectories
        if entry.IsDir() {
            if err := traverseRecursive(path, currentDepth+1, maxDepth); err != nil {
                return err
            }
        }
    }
    return nil
}

Pattern matching becomes particularly useful when building file discovery tools. You can combine multiple criteria to create sophisticated filtering logic:

func findSourceFiles(dir string) error {
    sourceExtensions := map[string]bool{
        ".go": true, ".py": true, ".js": true, ".rs": true,
    }

    return filepath.WalkDir(dir, func(path string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        // Skip hidden directories entirely
        if d.IsDir() && strings.HasPrefix(d.Name(), ".") {
            return filepath.SkipDir
        }

        // Check for source file extensions
        if !d.IsDir() {
            ext := filepath.Ext(d.Name())
            if sourceExtensions[ext] {
                fmt.Println(path)
            }
        }

        return nil
    })
}

For performance-critical applications, consider batching operations and minimizing system calls within your traversal loops. This approach is especially important when processing directories with thousands of entries where each additional system call compounds the performance impact.

ReadDirFile Interface

The ReadDirFile interface provides advanced directory reading capabilities for scenarios requiring more control over memory usage and processing of large directories. This interface extends basic file operations with streaming directory reads and pagination support.

ReadDirFile combines regular file operations with directory reading, allowing you to work with directory handles directly:

type ReadDirFile interface {
    File
    ReadDir(n int) ([]DirEntry, error)
}

The key advantage of this interface is the n parameter in ReadDir(n int), which controls how many entries to read in a single operation. This pagination mechanism prevents memory exhaustion when dealing with directories containing thousands or millions of files:

func streamDirectory(dirPath string) error {
    file, err := os.Open(dirPath)
    if err != nil {
        return err
    }
    defer file.Close()

    // Read directory in chunks of 100 entries
    for {
        entries, err := file.ReadDir(100)
        if err != nil {
            if err == io.EOF {
                break // End of directory reached
            }
            return err
        }

        // Process this batch of entries
        for _, entry := range entries {
            fmt.Println(entry.Name())
        }

        // If we got fewer entries than requested, we've reached the end
        if len(entries) < 100 {
            break
        }
    }
    return nil
}

The streaming approach becomes essential when working with extremely large directories where loading all entries into memory would be problematic:

func countFilesByExtension(dirPath string) (map[string]int, error) {
    file, err := os.Open(dirPath)
    if err != nil {
        return nil, err
    }
    defer file.Close()

    counts := make(map[string]int)
    batchSize := 500

    for {
        entries, err := file.ReadDir(batchSize)
        if err != nil && err != io.EOF {
            return nil, err
        }

        // Process current batch
        for _, entry := range entries {
            if !entry.IsDir() {
                ext := filepath.Ext(entry.Name())
                if ext == "" {
                    ext = "<no extension>"
                }
                counts[ext]++
            }
        }

        // Check for end conditions
        if err == io.EOF || len(entries) < batchSize {
            break
        }
    }

    return counts, nil
}

EOF handling requires careful attention because ReadDir can return both entries and an EOF error simultaneously. The final batch may contain entries even when EOF is returned:

func processAllEntries(file *os.File) error {
    for {
        entries, err := file.ReadDir(200)

        // Always process entries first, even if error occurred
        for _, entry := range entries {
            if err := processEntry(entry); err != nil {
                return err
            }
        }

        // Then handle the error
        if err != nil {
            if err == io.EOF {
                return nil // Normal completion
            }
            return err // Actual error
        }

        // Continue if no error and we got a full batch
    }
}

The pagination approach also enables progress tracking and cancellation for long-running directory operations:

func analyzeDirectoryWithProgress(dirPath string, cancel <-chan struct{}) error {
    file, err := os.Open(dirPath)
    if err != nil {
        return err
    }
    defer file.Close()

    var totalProcessed int
    batchSize := 1000

    for {
        select {
        case <-cancel:
            return fmt.Errorf("operation cancelled after processing %d entries", totalProcessed)
        default:
        }

        entries, err := file.ReadDir(batchSize)
        if err != nil && err != io.EOF {
            return err
        }

        // Process batch with progress reporting
        for _, entry := range entries {
            processEntry(entry)
            totalProcessed++

            if totalProcessed%10000 == 0 {
                fmt.Printf("Processed %d entries...\n", totalProcessed)
            }
        }

        if err == io.EOF || len(entries) < batchSize {
            fmt.Printf("Completed processing %d total entries\n", totalProcessed)
            break
        }
    }

    return nil
}

When choosing batch sizes, consider the trade-off between memory usage and system call overhead. Smaller batches use less memory but require more system calls, while larger batches are more efficient but consume more memory. A batch size between 100-1000 entries typically provides good balance for most applications.

The ReadDirFile interface is particularly valuable when building file system crawlers, backup tools, or any application that needs to process large directory structures without overwhelming system resources.

Practical Implementations

Building real-world directory tools requires combining the concepts we've explored into cohesive applications. These implementations demonstrate how to structure code for performance, maintainability, and practical utility.

Building a Directory Lister

A comprehensive directory lister showcases various filtering and display options while maintaining good performance characteristics:

type DirectoryLister struct {
    ShowHidden    bool
    ShowSizes     bool
    SortBySize    bool
    MaxDepth      int
    FileTypes     []string
}

func (dl *DirectoryLister) List(path string) error {
    entries, err := os.ReadDir(path)
    if err != nil {
        return fmt.Errorf("failed to read directory %s: %w", path, err)
    }

    // Filter entries based on configuration
    filtered := dl.filterEntries(entries)

    // Sort if requested
    if dl.SortBySize {
        sort.Slice(filtered, func(i, j int) bool {
            return dl.compareBySize(filtered[i], filtered[j])
        })
    }

    // Display entries
    for _, entry := range filtered {
        if err := dl.displayEntry(entry, path); err != nil {
            return err
        }
    }

    return nil
}

func (dl *DirectoryLister) filterEntries(entries []fs.DirEntry) []fs.DirEntry {
    var filtered []fs.DirEntry

    for _, entry := range entries {
        // Skip hidden files if not requested
        if !dl.ShowHidden && strings.HasPrefix(entry.Name(), ".") {
            continue
        }

        // Filter by file types if specified
        if len(dl.FileTypes) > 0 && !entry.IsDir() {
            ext := strings.ToLower(filepath.Ext(entry.Name()))
            if !dl.containsExtension(ext) {
                continue
            }
        }

        filtered = append(filtered, entry)
    }

    return filtered
}

func (dl *DirectoryLister) displayEntry(entry fs.DirEntry, basePath string) error {
    var size string
    var modTime string

    if dl.ShowSizes || dl.SortBySize {
        info, err := entry.Info()
        if err != nil {
            return err
        }

        if dl.ShowSizes {
            if entry.IsDir() {
                size = "<DIR>"
            } else {
                size = dl.formatSize(info.Size())
            }
            modTime = info.ModTime().Format("2006-01-02 15:04")
        }
    }

    // Format output based on options
    if dl.ShowSizes {
        fmt.Printf("%10s %s %s\n", size, modTime, entry.Name())
    } else {
        fmt.Printf("%s\n", entry.Name())
    }

    return nil
}

File Type Statistics

A file type analyzer demonstrates efficient processing of large directories while gathering statistical information:

type FileTypeStats struct {
    TypeCounts map[string]int
    SizeTotals map[string]int64
    TotalFiles int
    TotalDirs  int
    TotalSize  int64
}

func AnalyzeDirectory(path string) (*FileTypeStats, error) {
    stats := &FileTypeStats{
        TypeCounts: make(map[string]int),
        SizeTotals: make(map[string]int64),
    }

    return stats, filepath.WalkDir(path, func(filePath string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        if d.IsDir() {
            stats.TotalDirs++
            return nil
        }

        stats.TotalFiles++

        // Get file extension
        ext := strings.ToLower(filepath.Ext(d.Name()))
        if ext == "" {
            ext = "<no extension>"
        }

        stats.TypeCounts[ext]++

        // Get size information efficiently
        info, err := d.Info()
        if err != nil {
            return nil // Skip files we can't stat
        }

        size := info.Size()
        stats.SizeTotals[ext] += size
        stats.TotalSize += size

        return nil
    })
}

func (stats *FileTypeStats) PrintReport() {
    fmt.Printf("Directory Analysis Report\n")
    fmt.Printf("========================\n")
    fmt.Printf("Total Files: %d\n", stats.TotalFiles)
    fmt.Printf("Total Directories: %d\n", stats.TotalDirs)
    fmt.Printf("Total Size: %s\n\n", formatBytes(stats.TotalSize))

    // Sort extensions by file count
    type extInfo struct {
        ext   string
        count int
        size  int64
    }

    var extensions []extInfo
    for ext, count := range stats.TypeCounts {
        extensions = append(extensions, extInfo{
            ext:   ext,
            count: count,
            size:  stats.SizeTotals[ext],
        })
    }

    sort.Slice(extensions, func(i, j int) bool {
        return extensions[i].count > extensions[j].count
    })

    fmt.Printf("%-15s %8s %12s %8s\n", "Extension", "Files", "Total Size", "Avg Size")
    fmt.Printf("%-15s %8s %12s %8s\n", "---------", "-----", "----------", "--------")

    for _, ext := range extensions {
        avgSize := ext.size / int64(ext.count)
        fmt.Printf("%-15s %8d %12s %8s\n", 
            ext.ext, ext.count, 
            formatBytes(ext.size), formatBytes(avgSize))
    }
}

Directory Size Calculator

A directory size calculator demonstrates efficient recursive processing with progress reporting and memory-conscious design:

type DirSizeCalculator struct {
    ProgressCallback func(path string, size int64)
    IncludeHidden    bool
    FollowSymlinks   bool
    visitedInodes    map[uint64]bool
}

func NewDirSizeCalculator() *DirSizeCalculator {
    return &DirSizeCalculator{
        visitedInodes: make(map[uint64]bool),
    }
}

func (calc *DirSizeCalculator) Calculate(path string) (int64, error) {
    var totalSize int64

    err := filepath.WalkDir(path, func(filePath string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        // Skip hidden files/directories if requested
        if !calc.IncludeHidden && strings.HasPrefix(d.Name(), ".") {
            if d.IsDir() {
                return filepath.SkipDir
            }
            return nil
        }

        // Only process regular files
        if d.IsDir() {
            return nil
        }

        info, err := d.Info()
        if err != nil {
            return nil // Skip files we can't stat
        }

        // Handle hard links to avoid double-counting
        if sys := info.Sys(); sys != nil {
            if stat, ok := sys.(*syscall.Stat_t); ok {
                inode := stat.Ino
                if calc.visitedInodes[inode] {
                    return nil // Skip, already counted
                }
                calc.visitedInodes[inode] = true
            }
        }

        size := info.Size()
        totalSize += size

        // Report progress if callback is set
        if calc.ProgressCallback != nil {
            calc.ProgressCallback(filePath, size)
        }

        return nil
    })

    return totalSize, err
}

func (calc *DirSizeCalculator) CalculateWithBreakdown(path string) (map[string]int64, error) {
    dirSizes := make(map[string]int64)

    err := filepath.WalkDir(path, func(filePath string, d fs.DirEntry, err error) error {
        if err != nil {
            return err
        }

        if d.IsDir() {
            return nil
        }

        info, err := d.Info()
        if err != nil {
            return nil
        }

        size := info.Size()
        dir := filepath.Dir(filePath)

        // Add size to current directory and all parent directories
        for currentDir := dir; strings.HasPrefix(currentDir, path); currentDir = filepath.Dir(currentDir) {
            dirSizes[currentDir] += size
            if currentDir == path {
                break
            }
        }

        return nil
    })

    return dirSizes, err
}

These implementations demonstrate how to combine DirEntry operations, error handling, and performance optimization techniques to build robust directory processing tools. Each example balances functionality with efficiency, showing practical patterns you can adapt for your specific requirements.

Working with the Core FS Interface: Files, Directories, and Basic Operations (2/9)

Rez Moss — Sat, 07 Jun 2025 15:18:00 +0000

Understanding the FS Interface in Depth

The filesystem (FS) interface in Go provides a standardized way to interact with file systems, whether they're operating system filesystems, embedded filesystems, or custom implementations. At its core, the fs.FS interface defines a contract that any filesystem must fulfill, making your code portable across different storage backends.

The FS interface is deliberately minimal, containing just one method:

type FS interface {
    Open(name string) (File, error)
}

This simplicity is by design. The Open method serves as the entry point to any filesystem, taking a path and returning either a File interface or an error. Every filesystem operation ultimately flows through this single method, creating a consistent abstraction layer.

Method Signatures and Contracts

When implementing or working with filesystem interfaces, understanding the contracts is crucial. The Open method expects a valid path string and returns a File interface that must be closed after use. The path must be a valid path as defined by the ValidPath function, and the returned File must implement the basic File interface methods.

The File interface itself provides the fundamental operations:

type File interface {
    Stat() (FileInfo, error)
    Read([]byte) (int, error)
    Close() error
}

Each method has specific behavioral contracts. Stat() must return file metadata without side effects, Read() follows the standard io.Reader semantics, and Close() must be called to release resources, even if previous operations failed.

ValidPath Function and Security Implications

The ValidPath function determines whether a given path string is valid for use with the FS interface. This function serves as a critical security boundary, preventing path traversal attacks and ensuring consistent behavior across different filesystem implementations.

func ValidPath(name string) bool

A valid path must be UTF-8 encoded, use forward slashes as separators, not begin with a slash, and contain no empty elements or dot-dot (..) elements. These restrictions prevent attackers from using paths like ../../../etc/passwd to escape the intended filesystem boundaries.

When building applications that accept user-provided paths, always validate them using ValidPath before passing them to filesystem operations. This validation should happen at the application boundary, not deep within your filesystem handling code.

Error Types and Handling Strategies

The FS interface defines several standard error types that provide semantic meaning beyond generic error strings. Understanding these error types allows you to handle different failure scenarios appropriately.

The most common error types include:

ErrNotExist: The requested file or directory doesn't exist
ErrPermission: Insufficient permissions to perform the operation
ErrInvalid: The operation is invalid for the file type or state

Rather than checking error strings, use the error checking functions:

if errors.Is(err, fs.ErrNotExist) {
    // Handle file not found
} else if errors.Is(err, fs.ErrPermission) {
    // Handle permission denied
}

These semantic error types enable your code to respond intelligently to different failure conditions. For example, you might retry permission errors after a delay, create missing files when encountering ErrNotExist, or fail fast on invalid operations.

File Interface and Its Methods

Once you've obtained a File through the FS.Open() method, you're working with the File interface that provides the fundamental operations for reading file content and accessing metadata. The File interface builds upon Go's standard io interfaces while adding filesystem-specific capabilities.

Read, Stat, and Close Operations

The Read method follows the standard io.Reader interface, allowing you to read file content into a byte slice. It returns the number of bytes read and an error, following the same semantics as other readers in Go:

file, err := fsys.Open("data.txt")
if err != nil {
    return err
}
defer file.Close()

buffer := make([]byte, 1024)
n, err := file.Read(buffer)
if err != nil && err != io.EOF {
    return err
}
// Process buffer[:n]

The Stat method provides access to file metadata without reading the file content. This operation is typically fast since it only accesses the file's metadata, not its contents:

info, err := file.Stat()
if err != nil {
    return err
}
fmt.Printf("File size: %d bytes\n", info.Size())
fmt.Printf("Modified: %v\n", info.ModTime())

The Close method releases any resources associated with the file. This is critical for preventing resource leaks, especially when working with many files or long-running applications. Always call Close(), preferably using defer to ensure it executes even if other operations fail.

Working with File Metadata via FileInfo

The FileInfo interface provides rich metadata about files and directories. Understanding how to interpret this information is essential for building robust file processing applications:

type FileInfo interface {
    Name() string       // base name of the file
    Size() int64        // length in bytes for regular files
    Mode() FileMode     // file mode bits
    ModTime() time.Time // modification time
    IsDir() bool        // true if directory
    Sys() interface{}   // underlying data source (system-specific)
}

The Name() method returns only the base filename, not the full path. If you need the full path, you must track it separately. The Size() method returns the file size in bytes for regular files, but the value is system-dependent for directories and other special file types.

The Mode() method returns a FileMode value that encodes both the file type and permission bits. You can use this to determine if a file is a regular file, directory, symbolic link, or other special file type:

info, _ := file.Stat()
mode := info.Mode()

if mode.IsRegular() {
    // Regular file
} else if mode.IsDir() {
    // Directory
} else if mode&fs.ModeSymlink != 0 {
    // Symbolic link
}

Resource Management Best Practices

Proper resource management is crucial when working with files. The most common mistake is forgetting to close files, which can lead to resource exhaustion in long-running applications.

Always use defer to ensure files are closed:

func processFile(fsys fs.FS, filename string) error {
    file, err := fsys.Open(filename)
    if err != nil {
        return err
    }
    defer file.Close() // This runs even if later operations fail

    // Process file...
    return nil
}

When working with multiple files, be careful about defer in loops. The defer statement doesn't execute until the function returns, so opening many files in a loop without explicit closure can exhaust file descriptors:

// Problematic - all files stay open until function returns
for _, filename := range filenames {
    file, err := fsys.Open(filename)
    if err != nil {
        continue
    }
    defer file.Close() // Defers accumulate!
    // Process file...
}

// Better approach - explicit closure
for _, filename := range filenames {
    func() {
        file, err := fsys.Open(filename)
        if err != nil {
            return
        }
        defer file.Close()
        // Process file...
    }()
}

Another approach is to explicitly close files within the loop, but you lose the safety that defer provides against early returns and panics.

Basic File Operations

Working with files through the FS interface involves common patterns that you'll use repeatedly. Understanding these patterns and their proper implementation helps you write reliable file processing code.

Opening and Reading Files

The most fundamental operation is opening and reading file content. The FS interface provides a clean abstraction for this, but there are several approaches depending on your needs:

// Reading entire file content (for small files)
func readEntireFile(fsys fs.FS, filename string) ([]byte, error) {
    file, err := fsys.Open(filename)
    if err != nil {
        return nil, fmt.Errorf("opening %s: %w", filename, err)
    }
    defer file.Close()

    return io.ReadAll(file)
}

// Reading file in chunks (for large files or streaming)
func readFileInChunks(fsys fs.FS, filename string, chunkSize int) error {
    file, err := fsys.Open(filename)
    if err != nil {
        return fmt.Errorf("opening %s: %w", filename, err)
    }
    defer file.Close()

    buffer := make([]byte, chunkSize)
    for {
        n, err := file.Read(buffer)
        if n > 0 {
            // Process buffer[:n]
            processChunk(buffer[:n])
        }
        if err == io.EOF {
            break
        }
        if err != nil {
            return fmt.Errorf("reading %s: %w", filename, err)
        }
    }
    return nil
}

For text files, you might want to read line by line using a scanner:

func readLines(fsys fs.FS, filename string) ([]string, error) {
    file, err := fsys.Open(filename)
    if err != nil {
        return nil, err
    }
    defer file.Close()

    var lines []string
    scanner := bufio.NewScanner(file)
    for scanner.Scan() {
        lines = append(lines, scanner.Text())
    }

    if err := scanner.Err(); err != nil {
        return nil, fmt.Errorf("scanning %s: %w", filename, err)
    }

    return lines, nil
}

Checking File Existence and Properties

Before processing files, you often need to check if they exist and examine their properties. The FS interface provides several ways to accomplish this:

// Check if file exists
func fileExists(fsys fs.FS, filename string) bool {
    _, err := fs.Stat(fsys, filename)
    return err == nil
}

// Get file information without opening
func getFileInfo(fsys fs.FS, filename string) (fs.FileInfo, error) {
    return fs.Stat(fsys, filename)
}

// Check if path is a directory
func isDirectory(fsys fs.FS, path string) (bool, error) {
    info, err := fs.Stat(fsys, path)
    if err != nil {
        return false, err
    }
    return info.IsDir(), nil
}

// Check file size before processing
func checkFileSize(fsys fs.FS, filename string, maxSize int64) error {
    info, err := fs.Stat(fsys, filename)
    if err != nil {
        return err
    }

    if info.Size() > maxSize {
        return fmt.Errorf("file %s too large: %d bytes (max %d)", 
            filename, info.Size(), maxSize)
    }

    return nil
}

Using fs.Stat() is more efficient than opening a file just to check its properties, since it doesn't require reading file content or allocating file descriptors.

Proper Error Handling Patterns

Error handling in file operations requires attention to different failure scenarios. The key is to provide meaningful context while preserving the original error information:

func robustFileOperation(fsys fs.FS, filename string) error {
    // Check file exists first
    if _, err := fs.Stat(fsys, filename); err != nil {
        if errors.Is(err, fs.ErrNotExist) {
            return fmt.Errorf("file %s does not exist", filename)
        }
        return fmt.Errorf("cannot access %s: %w", filename, err)
    }

    // Open and process file
    file, err := fsys.Open(filename)
    if err != nil {
        if errors.Is(err, fs.ErrPermission) {
            return fmt.Errorf("permission denied accessing %s", filename)
        }
        return fmt.Errorf("failed to open %s: %w", filename, err)
    }
    defer func() {
        if closeErr := file.Close(); closeErr != nil {
            log.Printf("Warning: failed to close %s: %v", filename, closeErr)
        }
    }()

    // Process file content
    buffer := make([]byte, 4096)
    for {
        n, err := file.Read(buffer)
        if n > 0 {
            if processErr := processContent(buffer[:n]); processErr != nil {
                return fmt.Errorf("processing %s: %w", filename, processErr)
            }
        }

        if err == io.EOF {
            break
        }
        if err != nil {
            return fmt.Errorf("reading %s: %w", filename, err)
        }
    }

    return nil
}

This pattern demonstrates several important practices:

Check file accessibility before opening when appropriate
Distinguish between different error types for better user experience
Wrap errors with context using fmt.Errorf and %w verb
Handle close errors appropriately (usually logging rather than returning)
Provide meaningful error messages that help diagnose issues

For operations that might encounter temporary failures, consider implementing retry logic:

func readFileWithRetry(fsys fs.FS, filename string, maxRetries int) ([]byte, error) {
    var lastErr error

    for attempt := 0; attempt <= maxRetries; attempt++ {
        data, err := readEntireFile(fsys, filename)
        if err == nil {
            return data, nil
        }

        lastErr = err

        // Only retry on certain error types
        if errors.Is(err, fs.ErrPermission) || 
           errors.Is(err, fs.ErrNotExist) {
            break // Don't retry these
        }

        if attempt < maxRetries {
            time.Sleep(time.Millisecond * 100 * time.Duration(attempt+1))
        }
    }

    return nil, fmt.Errorf("failed after %d attempts: %w", maxRetries+1, lastErr)
}

Directory Operations Introduction

Directories are special file system entities that contain references to other files and directories. The FS interface treats directories as a special type of file, but they require different handling patterns and have unique characteristics that affect how you work with them.

Distinguishing Files from Directories

The most reliable way to determine whether a path refers to a file or directory is through the FileInfo interface. Several methods provide this information:

func examineFileType(fsys fs.FS, path string) error {
    info, err := fs.Stat(fsys, path)
    if err != nil {
        return fmt.Errorf("cannot stat %s: %w", path, err)
    }

    if info.IsDir() {
        fmt.Printf("%s is a directory\n", path)
        fmt.Printf("Directory size: %d bytes\n", info.Size()) // Often system-dependent
    } else {
        fmt.Printf("%s is a regular file\n", path)
        fmt.Printf("File size: %d bytes\n", info.Size())
    }

    return nil
}

When you attempt to open a directory with the standard Open method, the behavior depends on the filesystem implementation. Some filesystems allow reading directory contents directly, while others may return an error:

func attemptDirectoryOpen(fsys fs.FS, dirPath string) {
    file, err := fsys.Open(dirPath)
    if err != nil {
        fmt.Printf("Cannot open directory %s: %v\n", dirPath, err)
        return
    }
    defer file.Close()

    info, err := file.Stat()
    if err != nil {
        fmt.Printf("Cannot stat opened directory: %v\n", err)
        return
    }

    if info.IsDir() {
        fmt.Printf("Successfully opened directory %s\n", dirPath)
        // Reading directory contents requires ReadDirFS interface
    }
}

FileMode and Permission Bits

The FileMode type encodes both file type information and permission bits. Understanding how to interpret and work with FileMode is essential for proper file handling:

func analyzeFileMode(info fs.FileInfo) {
    mode := info.Mode()

    // Check file type using mode bits
    switch {
    case mode.IsRegular():
        fmt.Println("Regular file")
    case mode.IsDir():
        fmt.Println("Directory")
    case mode&fs.ModeSymlink != 0:
        fmt.Println("Symbolic link")
    case mode&fs.ModeDevice != 0:
        fmt.Println("Device file")
    case mode&fs.ModeNamedPipe != 0:
        fmt.Println("Named pipe")
    case mode&fs.ModeSocket != 0:
        fmt.Println("Socket")
    case mode&fs.ModeCharDevice != 0:
        fmt.Println("Character device")
    }

    // Extract permission bits (Unix-style)
    perm := mode.Perm()
    fmt.Printf("Permissions: %o\n", perm)

    // Check specific permission bits
    if perm&0400 != 0 {
        fmt.Println("Owner can read")
    }
    if perm&0200 != 0 {
        fmt.Println("Owner can write")
    }
    if perm&0100 != 0 {
        fmt.Println("Owner can execute")
    }
}

The permission bits follow Unix conventions, even on Windows systems, though the actual enforcement may vary by operating system. The Perm() method returns only the permission bits, masking out the file type information:

func checkReadable(info fs.FileInfo) bool {
    mode := info.Mode()

    // For directories, check if we can read directory contents
    if mode.IsDir() {
        return mode.Perm()&0400 != 0 // Owner read permission
    }

    // For regular files, check read permission
    if mode.IsRegular() {
        return mode.Perm()&0400 != 0
    }

    // For other file types, be cautious
    return false
}

func isExecutable(info fs.FileInfo) bool {
    mode := info.Mode()

    // Only regular files can be executable in the traditional sense
    if !mode.IsRegular() {
        return false
    }

    // Check if any execute bit is set (owner, group, or other)
    return mode.Perm()&0111 != 0
}

Understanding file modes becomes particularly important when building tools that need to preserve or check file permissions:

func validateFilePermissions(fsys fs.FS, filename string, requiredPerms fs.FileMode) error {
    info, err := fs.Stat(fsys, filename)
    if err != nil {
        return err
    }

    actualPerms := info.Mode().Perm()

    // Check if file has at least the required permissions
    if actualPerms&requiredPerms != requiredPerms {
        return fmt.Errorf("file %s has permissions %o, requires %o", 
            filename, actualPerms, requiredPerms)
    }

    return nil
}

// Usage example
func main() {
    var fsys fs.FS = os.DirFS(".")

    // Check if file is readable by owner
    if err := validateFilePermissions(fsys, "config.txt", 0400); err != nil {
        fmt.Printf("Permission check failed: %v\n", err)
    }

    // Check if file is executable
    if err := validateFilePermissions(fsys, "script.sh", 0100); err != nil {
        fmt.Printf("File is not executable: %v\n", err)
    }
}

For reading directory contents, you need to check if the filesystem implements the ReadDirFS interface:

func listDirectoryContents(fsys fs.FS, dirPath string) ([]fs.DirEntry, error) {
    if readDirFS, ok := fsys.(fs.ReadDirFS); ok {
        return readDirFS.ReadDir(dirPath)
    }

    return nil, fmt.Errorf("filesystem does not support directory reading")
}

Practical Examples

Understanding the theory behind the FS interface is important, but seeing it applied in real-world scenarios helps solidify the concepts. These examples demonstrate common patterns you'll encounter when building applications that work with files and directories.

Building a Simple File Reader

A practical file reader needs to handle various file types, sizes, and error conditions gracefully. Here's a comprehensive example that incorporates the concepts we've covered:

package main

import (
    "bufio"
    "fmt"
    "io"
    "io/fs"
    "os"
    "path/filepath"
    "strings"
)

type FileReader struct {
    fsys       fs.FS
    maxSize    int64
    bufferSize int
}

func NewFileReader(fsys fs.FS, maxSize int64) *FileReader {
    return &FileReader{
        fsys:       fsys,
        maxSize:    maxSize,
        bufferSize: 4096,
    }
}

func (fr *FileReader) ReadFile(filename string) (*FileContent, error) {
    // Validate path
    if !fs.ValidPath(filename) {
        return nil, fmt.Errorf("invalid path: %s", filename)
    }

    // Check file exists and get info
    info, err := fs.Stat(fr.fsys, filename)
    if err != nil {
        if errors.Is(err, fs.ErrNotExist) {
            return nil, fmt.Errorf("file not found: %s", filename)
        }
        return nil, fmt.Errorf("cannot access file %s: %w", filename, err)
    }

    // Ensure it's a regular file
    if !info.Mode().IsRegular() {
        return nil, fmt.Errorf("%s is not a regular file", filename)
    }

    // Check size limits
    if fr.maxSize > 0 && info.Size() > fr.maxSize {
        return nil, fmt.Errorf("file %s too large: %d bytes (max %d)", 
            filename, info.Size(), fr.maxSize)
    }

    // Open and read file
    file, err := fr.fsys.Open(filename)
    if err != nil {
        return nil, fmt.Errorf("failed to open %s: %w", filename, err)
    }
    defer file.Close()

    content := &FileContent{
        Name:     info.Name(),
        Size:     info.Size(),
        ModTime:  info.ModTime(),
        Mode:     info.Mode(),
    }

    // Read content based on file size
    if info.Size() < int64(fr.bufferSize*2) {
        // Small file - read all at once
        content.Data, err = io.ReadAll(file)
        if err != nil {
            return nil, fmt.Errorf("failed to read %s: %w", filename, err)
        }
    } else {
        // Large file - read in chunks
        content.Data, err = fr.readInChunks(file, info.Size())
        if err != nil {
            return nil, fmt.Errorf("failed to read %s in chunks: %w", filename, err)
        }
    }

    // Detect if content is text
    content.IsText = fr.isTextContent(content.Data)

    return content, nil
}

func (fr *FileReader) readInChunks(file fs.File, size int64) ([]byte, error) {
    data := make([]byte, 0, size)
    buffer := make([]byte, fr.bufferSize)

    for {
        n, err := file.Read(buffer)
        if n > 0 {
            data = append(data, buffer[:n]...)
        }

        if err == io.EOF {
            break
        }
        if err != nil {
            return nil, err
        }
    }

    return data, nil
}

func (fr *FileReader) isTextContent(data []byte) bool {
    // Simple heuristic: check first 512 bytes for null bytes
    checkSize := 512
    if len(data) < checkSize {
        checkSize = len(data)
    }

    for i := 0; i < checkSize; i++ {
        if data[i] == 0 {
            return false
        }
    }

    return true
}

type FileContent struct {
    Name    string
    Size    int64
    ModTime time.Time
    Mode    fs.FileMode
    Data    []byte
    IsText  bool
}

func (fc *FileContent) String() string {
    var sb strings.Builder

    sb.WriteString(fmt.Sprintf("File: %s\n", fc.Name))
    sb.WriteString(fmt.Sprintf("Size: %d bytes\n", fc.Size))
    sb.WriteString(fmt.Sprintf("Modified: %v\n", fc.ModTime))
    sb.WriteString(fmt.Sprintf("Mode: %v\n", fc.Mode))
    sb.WriteString(fmt.Sprintf("Type: "))

    if fc.IsText {
        sb.WriteString("Text")
    } else {
        sb.WriteString("Binary")
    }

    return sb.String()
}

func (fc *FileContent) Lines() []string {
    if !fc.IsText {
        return nil
    }

    return strings.Split(string(fc.Data), "\n")
}

Error Handling Scenarios

Robust error handling requires anticipating various failure modes and responding appropriately:

func demonstrateErrorHandling() {
    fsys := os.DirFS(".")
    reader := NewFileReader(fsys, 1024*1024) // 1MB limit

    testFiles := []string{
        "existing.txt",
        "nonexistent.txt",
        "../outside.txt",  // Invalid path
        "directory",       // Directory instead of file
        "huge.dat",        // File too large
        "readonly.txt",    // Permission denied
    }

    for _, filename := range testFiles {
        fmt.Printf("\nTesting file: %s\n", filename)

        content, err := reader.ReadFile(filename)
        if err != nil {
            // Handle specific error types
            switch {
            case errors.Is(err, fs.ErrNotExist):
                fmt.Printf("  File does not exist\n")
            case errors.Is(err, fs.ErrPermission):
                fmt.Printf("  Permission denied\n")
            case strings.Contains(err.Error(), "invalid path"):
                fmt.Printf("  Path validation failed\n")
            case strings.Contains(err.Error(), "too large"):
                fmt.Printf("  File exceeds size limit\n")
            case strings.Contains(err.Error(), "not a regular file"):
                fmt.Printf("  Not a regular file\n")
            default:
                fmt.Printf("  Unexpected error: %v\n", err)
            }
            continue
        }

        fmt.Printf("  Successfully read: %s\n", content.String())
    }
}

Resource Cleanup Patterns

Proper resource management is critical for long-running applications. Here are patterns for ensuring resources are cleaned up correctly:

// Pattern 1: Simple defer for single file
func processFile(fsys fs.FS, filename string) error {
    file, err := fsys.Open(filename)
    if err != nil {
        return err
    }
    defer func() {
        if closeErr := file.Close(); closeErr != nil {
            // Log close errors but don't override the main error
            log.Printf("Failed to close %s: %v", filename, closeErr)
        }
    }()

    // Process file...
    return nil
}

// Pattern 2: Multiple files with explicit cleanup
func processMultipleFiles(fsys fs.FS, filenames []string) error {
    var openFiles []fs.File

    // Cleanup function
    cleanup := func() {
        for _, file := range openFiles {
            if err := file.Close(); err != nil {
                log.Printf("Failed to close file: %v", err)
            }
        }
    }
    defer cleanup()

    // Open all files first
    for _, filename := range filenames {
        file, err := fsys.Open(filename)
        if err != nil {
            return fmt.Errorf("failed to open %s: %w", filename, err)
        }
        openFiles = append(openFiles, file)
    }

    // Process all files...
    return nil
}

// Pattern 3: Resource pool for high-throughput scenarios
type FileProcessor struct {
    fsys      fs.FS
    semaphore chan struct{}
}

func NewFileProcessor(fsys fs.FS, maxConcurrent int) *FileProcessor {
    return &FileProcessor{
        fsys:      fsys,
        semaphore: make(chan struct{}, maxConcurrent),
    }
}

func (fp *FileProcessor) ProcessFiles(filenames []string) error {
    errCh := make(chan error, len(filenames))

    for _, filename := range filenames {
        go func(fname string) {
            // Acquire semaphore
            fp.semaphore <- struct{}{}
            defer func() { <-fp.semaphore }()

            // Process file with proper cleanup
            if err := fp.processSingleFile(fname); err != nil {
                errCh <- fmt.Errorf("processing %s: %w", fname, err)
                return
            }
            errCh <- nil
        }(filename)
    }

    // Collect results
    var errors []error
    for i := 0; i < len(filenames); i++ {
        if err := <-errCh; err != nil {
            errors = append(errors, err)
        }
    }

    if len(errors) > 0 {
        return fmt.Errorf("processing failed: %v", errors)
    }

    return nil
}

func (fp *FileProcessor) processSingleFile(filename string) error {
    file, err := fp.fsys.Open(filename)
    if err != nil {
        return err
    }
    defer file.Close()

    // Process file content...
    return nil
}

// Usage example
func main() {
    fsys := os.DirFS("./data")
    processor := NewFileProcessor(fsys, 10) // Max 10 concurrent files

    filenames := []string{"file1.txt", "file2.txt", "file3.txt"}
    if err := processor.ProcessFiles(filenames); err != nil {
        log.Printf("Processing failed: %v", err)
    }
}

These examples demonstrate production-ready patterns for working with the FS interface. The file reader handles various edge cases and error conditions, the error handling scenarios show how to respond to different failure modes, and the resource cleanup patterns ensure your applications don't leak file descriptors or other system resources.

Go's fs Package: Modern File System Abstraction 1/9

Rez Moss — Sat, 31 May 2025 15:12:00 +0000

What is the fs package and why it matters

The fs package landed in Go 1.16 and fundamentally changed how we think about file operations in Go programs. Before this package existed, developers relied heavily on the os package for all file system interactions, which created tight coupling between application logic and the underlying operating system.

The fs package introduces an abstract file system interface that sits between your code and the actual file system implementation. Instead of calling os.Open() directly and dealing with concrete file handles, you work with interfaces that can be implemented by various file system backends - whether that's the real OS file system, an embedded file system, a ZIP archive, or even a mock implementation for testing.

This abstraction brings three major benefits to Go developers. First, testability improves dramatically because you can easily swap out file system implementations during testing without touching temporary files or dealing with OS-specific behaviors. Second, security gets better through controlled access - you can restrict what parts of the file system your code can reach by providing a specific fs.FS implementation. Third, portability increases because your code doesn't need to know whether it's reading from disk, memory, or network storage.

The key insight here is that most application code doesn't actually care where files come from. Whether you're reading configuration from a local file or from an embedded resource, the reading logic stays the same. The fs package captures this pattern and makes it explicit in the type system.

Core concepts and philosophy

The fs package builds on Go's interface-based design philosophy. Rather than providing concrete implementations, it defines a set of small, focused interfaces that different file system implementations can satisfy. This approach follows the principle that interfaces should be defined by the consumer, not the provider.

The central concept is the separation between logical and physical file systems. Your application logic operates on the logical level - it knows it needs to read a file called "config.json" but doesn't care if that file lives on disk, in memory, or inside a ZIP archive. The physical implementation handles the actual storage mechanics.

This separation enables powerful patterns like embedded file systems, where you can bundle static assets directly into your binary using embed.FS, and virtual file systems that exist purely in memory or are generated on-demand. The beauty is that switching between these different backends requires minimal code changes because they all implement the same interfaces.

The package also embraces composition over inheritance. Instead of one monolithic file system interface, you get small interfaces like fs.FS for basic file access, fs.ReadDirFS for directory listing, and fs.StatFS for file metadata. Implementations can choose which capabilities to support, and your code can check for specific interfaces using type assertions when you need advanced features.

Basic FS interface walkthrough

The foundation of the entire fs package is the FS interface, which is deliberately minimal:

type FS interface {
    Open(name string) (File, error)
}

This single method contract means any type that can open files by name qualifies as a file system. The name parameter follows specific rules - it must use forward slashes as separators, cannot start with a slash or contain . or .. elements, and should be relative paths only.

The Open method returns a File interface, which provides the basic operations you'd expect:

type File interface {
    Stat() (FileInfo, error)
    Read([]byte) (int, error)
    Close() error
}

Notice that File embeds io.Reader, so you can use any File wherever an io.Reader is expected. This integration with existing Go interfaces means the fs package plays well with the rest of the standard library.

Error handling follows a consistent pattern throughout the package. When operations fail, they return *PathError which wraps the underlying error with context about which path and operation caused the problem. This gives you structured error information that you can inspect programmatically:

if pathErr, ok := err.(*fs.PathError); ok {
    fmt.Printf("Operation %s failed on path %s: %v", 
        pathErr.Op, pathErr.Path, pathErr.Err)
}

The interfaces are designed to fail gracefully. If you try to open a directory as a file, or access something that doesn't exist, you get well-defined error values like fs.ErrNotExist that you can check with errors.Is().

Simple practical example

Let's see how the fs package works in practice by comparing it with the traditional os package approach. Here's how you might read a configuration file the old way:

// Traditional approach with os package
func loadConfigOld() (*Config, error) {
    file, err := os.Open("config/app.json")
    if err != nil {
        return nil, err
    }
    defer file.Close()

    data, err := io.ReadAll(file)
    if err != nil {
        return nil, err
    }

    var config Config
    err = json.Unmarshal(data, &config)
    return &config, err
}

Now here's the same functionality using the fs package with os.DirFS:

// Modern approach with fs package
func loadConfig(fsys fs.FS) (*Config, error) {
    file, err := fsys.Open("app.json")
    if err != nil {
        return nil, err
    }
    defer file.Close()

    data, err := io.ReadAll(file)
    if err != nil {
        return nil, err
    }

    var config Config
    err = json.Unmarshal(data, &config)
    return &config, err
}

// Usage
func main() {
    configFS := os.DirFS("config")
    config, err := loadConfig(configFS)
    // handle error and use config
}

The difference might seem subtle, but it's significant. In the first version, your function is hardcoded to read from the OS file system at a specific path. In the second version, your function accepts any file system implementation and reads from a relative path within that file system.

This small change opens up powerful possibilities. You can easily test loadConfig by passing in a test file system, switch to reading from embedded files by passing embed.FS, or even read from a ZIP archive by using zip.Reader which implements fs.FS. The function itself never needs to change.

When to use fs vs os package

The choice between fs and os packages depends on what you're building and how much flexibility you need. Use the fs package when you want to decouple your code from specific file system implementations. This is particularly valuable for libraries, reusable components, and applications that need to work with different storage backends.

The fs package shines in scenarios like configuration loading, template processing, static asset serving, and any situation where you might want to switch between embedded resources and external files. It's also the clear winner for testing - writing unit tests becomes much simpler when you can provide mock file systems instead of creating temporary files.

However, stick with the os package when you need full file system capabilities that go beyond reading. The fs package is intentionally read-only and doesn't support operations like creating, writing, or deleting files. If you need to modify files, create directories, or work with file permissions and ownership, you'll need the os package.

The os package is also more appropriate for system-level programming, file management utilities, or when you specifically need to interact with the operating system's file system features. There's no abstraction overhead, and you get direct access to all OS capabilities.

A practical approach is to use fs interfaces in your public APIs and internal abstractions, but fall back to os when you need write operations or system-specific features. Many Go standard library packages now accept fs.FS parameters alongside their traditional file path alternatives, giving you the flexibility to choose the right tool for each situation.