DEV Community

Attempting to Learn Go - Listing Files By Extension

Steve Layton on February 23, 2019

Hello World Near the end of the last post, I noted we would put the static site generator project aside for the time being. I decided th...

Read full post

Dmitry Yakimenko • Feb 23 '19 • Edited

Steve, why do you sort inside the loop on every iteration?

for _, file := range dir {
    if !file.IsDir() {
        ...
        sort.Strings(m[ext[len(ext)-1]]) // <-- HERE
    }
}

And here's my take on it. You can sort by predicate. It's not exactly very efficient though, since the extension is recalculated every time. But come on, Go could be really annoying sometimes. Look at this verbosity:

type ByExt []string

func (a ByExt) Len() int           { return len(a) }
func (a ByExt) Swap(i, j int)      { a[i], a[j] = a[j], a[i] }
func (a ByExt) Less(i, j int) bool { return filepath.Ext(a[i]) < filepath.Ext(a[j]) }

...

func main() {
    ...
    files := []string{}
    for _, file := range dir {
        if !file.IsDir() {
            files = append(files, file.Name())
        }
    }
    sort.Sort(ByExt(files))
    ...
}

In Ruby that would be:

filenames.sort_by { |f| File.extname f }

Dirk Olbrich • Feb 23 '19 • Edited

Nah, no need for the boilerplate to define a custom sort. This would do sorting the map:

var m = make(map[string][]string)
for _, file := range dir {
    if !file.IsDir() {
        fileName := file.Name()
        ext := strings.Split(fileName, ".")
        switch {
        case len(ext) > 1:
            m[ext[len(ext)-1]] = append(m[ext[len(ext)-1]], fileName)
        case len(ext) == 1:
            m["no-ext"] = append(m["no-ext"], fileName)
        }
    }
}
for ext := range m { sort.Strings(m[ext]) }

Dirk Olbrich • Feb 23 '19 • Edited

Edit: sorting the ˋ[]stringˋ within the ˋmap[string][]stringˋ. The map itself can't be sorted.

Dmitry Yakimenko • Feb 23 '19

I don't have a map in my version. I sort an array by a predicate.

Dirk Olbrich • Feb 23 '19

Yes, I have seen it. Your approach is different by just sorting a list of filenames. The orginal intent is to sort files by extension into different buckets.

Your use of filepath.Ext() is quit clever. Haven't thought of that.

Dirk Olbrich • Feb 23 '19 • Edited

This would make the example even shorter:

var m = make(map[string][]string)
for _, file := range dir {
    if !file.IsDir() {
        fileName := file.Name()
        ext := filepath.Ext(fileName)
        m[ext] = append(m[ext], fileName)
    }
}
for ext := range m { sort.Strings(m[ext]) }

Benjamin Cable • Feb 24 '19 • Edited

Hey Steve, fantastic start.

You've come up with some great solutions in there, so I thought I'd share my own.

I restricted myself to fitting a subset of what you've solved thus far, that is to say, get all the files organised by category, and print them out as JSON. I've ignored plain / nested output, since that is somewhat trivial/not business logic.

Here's my solution:

package main

import (
    "encoding/json"
    "flag"
    "io/ioutil"
    "log"
    "os"
    "path/filepath"
    "strings"
)

var directory string

func main() {
    flag.StringVar(&directory, "dir", ".", "sorter -dir ./path/to/dir")
    flag.Parse()

    files, err := ioutil.ReadDir(directory)
    if err != nil {
        log.Fatal(err)
    }

    var categories = make(map[string][]string)
    for _, file := range files {
        // skip directories.
        if file.IsDir() {
            continue
        }

        ext := filepath.Ext(file.Name())
        name := strings.TrimSuffix(file.Name(), ext)

        // empty name signified a dotfile, skip that.
        if name == "" {
            continue
        }

        // get the absolute path to the file, or error out
        fpath, err := filepath.Abs(filepath.Join(directory, file.Name()))
        if err != nil {
            log.Fatalf("failed building absolute path: %v", err)
        }

        // trim dots before adding to the map.
        ext = strings.TrimPrefix(ext, ".")
        categories[ext] = append(categories[ext], fpath)
    }

    if err := json.NewEncoder(os.Stdout).Encode(categories); err != nil {
        panic(err)
    }
}

As you can see, I've drastically cut down on the number of operations needed to get there, as well as corrected for a few problems you weren't looking out for yet. These are mainly:

You should skip dotfiles or hidden files, which start with a . character (at least by default), as these are frequently config files or important somehow.
You're expending a lot of effort sorting / printing your data, when really all you need is a map to handle the listing

output from my program (against a sample directory):

usage: sorter -dir ./sample | jq

{
  "jpg": [
    "/Users/bc/code/Personal/sorter/samples/3.jpg"
  ],
  "pdf": [
    "/Users/bc/code/Personal/sorter/samples/2.pdf"
  ],
  "txt": [
    "/Users/bc/code/Personal/sorter/samples/1.txt",
    "/Users/bc/code/Personal/sorter/samples/2.txt",
    "/Users/bc/code/Personal/sorter/samples/3.txt"
  ]
}

If I wanted plain output, with a map I could do something like this:

for key, category := range categories {
    fmt.Println("kind:", key)
    for _, file := range category {
        fmt.Println("\t", file)
    }
}

which would output like so:

kind: txt
     /Users/bc/code/Personal/sorter/samples/1.txt
     /Users/bc/code/Personal/sorter/samples/2.txt
     /Users/bc/code/Personal/sorter/samples/3.txt
kind: pdf
     /Users/bc/code/Personal/sorter/samples/2.pdf
kind: jpg
     /Users/bc/code/Personal/sorter/samples/3.jpg

This is somewhat trite and gross but you get the point, dealing with one map makes this much easier to handle!

Looking forward to seeing what you come up with next!

Steve Layton • Feb 25 '19

I think the core of my issue is I'm also not leaning on the standard library as much as I should. I didn't realize filepath.Ext() was a thing. :/ Yeah, I read "sorting files by ext" as just that sorting alphabetically, had I left that out I would have been done quite a bit quicker. I suppose that made me go off the rails a bit so to speak. The different printing methods were not needed at all but what are you gonna do lol.