DEV Community

loading...
Cover image for The easiest way to embed static files into a binary file in your Golang app (no external dependencies)

The easiest way to embed static files into a binary file in your Golang app (no external dependencies)

Vic Sh贸stak
Hey! 馃憢 I'm a Software Engineer and founder of a friendly outsource company called True web artisans, with excellent UX/UI knowledge and over 12 years of practical experience.
Updated on 銉6 min read

Introduction

Hello, everyone! 馃憢 I have a very interesting find for you. But first, I'm going to tell you a little background story, so...

Earlier, I tried go.rice and go-bindata, then I used packr and switched to pkger from the same author (Hi, Mark Bates, if you're suddenly reading this). Yep, I was satisfied with this packages and quite happy!

But when I saw this method somewhere on the Internet, I immediately herded it into all of my Golang apps (who needed it, of course).

...and what's problem with ready-made packages?

First of all, it's external packages with their dependencies. It's good when you need to quickly make a working prototype OR don't really care about the size of your code base.

Documentation may not be complete or cover actually what you need at this moment. I have been frustrated to have to wait a long time (sometimes, up to several weeks) to get an answer OR a bug fix.

I have nothing against the authors of these great Go packages! I personally support many of Open Source repositories on GitHub and understand they very well! 馃槈

Disclaimer

This article was written before Go 1.16 released, which added awesome features for adding files to the final binary.

Why I need this?

Good question, by the way, because many programmers do not always understand advantages of this approach to packaging a product (the final program, which you sell to your customer).

鈽濓笍 Especially, who came from scripting languages like JavaScript, Python, PHP or else.

I highlight the following:

  • It's safe. Files, which added to a binary, are impossible (or extremely difficult) to extract back.

  • It's convenient. There's no need to always remember to include a folder with static files, when you give production-ready product to your customer.

embed static files

OK! So what am I suggesting? 馃

My suggest will be very simple and elegant: why not write our own packer for static files using only built-in Go tools? Yes, gophers have already figured out what I'm talking about... Go generate function!

猸愶笍 Please, read first this awesome blog post by Rob Pike (22 Dec 2014) for more understand how it works! Unfortunately, I can't explain this principle better and shorter than Rob did...

No time for further reading? 馃殌

I created repository with full code example on my GitHub especially for you:

GitHub logo koddr / example-embed-static-files-go

The easiest way to embed static files into a binary file in your Golang app (no external dependencies).

Just git clone and read instructions from README.

Project structure

Let's go mod init ... and create some files 馃憣

$ tree .
.
鈹溾攢鈹 Makefile
鈹溾攢鈹 go.mod
鈹溾攢鈹 cmd
鈹   鈹斺攢鈹 app
鈹       鈹斺攢鈹 main.go          <-- main file of your Go app
鈹溾攢鈹 internal
鈹   鈹斺攢鈹 box                  <-- template and functions for create blob
鈹       鈹溾攢鈹 box.go
鈹       鈹斺攢鈹 generator.go
鈹斺攢鈹 static                   <-- folder with static files
    鈹斺攢鈹 index.html
Enter fullscreen mode Exit fullscreen mode

Add most used commands to Makefile:

.PHONY: generate

# 鈿狅笍
generate:
    @go generate ./...
    @echo "[OK] Files added to embed box!"

security:
    @gosec ./...
    @echo "[OK] Go security check was completed!"

build: generate security
    @go build -o ./build/server ./cmd/app/*.go
    @echo "[OK] App binary was created!"

run:
    @./build/server
Enter fullscreen mode Exit fullscreen mode

鈿狅笍 I strongly recommend to use Go security checker for your code! Maybe I'll write a separate article about this great package later.

Create methods for use on code (box.go)

//go:generate go run generator.go

package box

type embedBox struct {
    storage map[string][]byte
}

// Create new box for embed files
func newEmbedBox() *embedBox {
    return &embedBox{storage: make(map[string][]byte)}
}

// Add a file to box
func (e *embedBox) Add(file string, content []byte) {
    e.storage[file] = content
}

// Get file's content
func (e *embedBox) Get(file string) []byte {
    if f, ok := e.storage[file]; ok {
        return f
    }
    return nil
}

// Embed box expose
var box = newEmbedBox()

// Add a file content to box
func Add(file string, content []byte) {
    box.Add(file, content)
}

// Get a file from box
func Get(file string) []byte {
    return box.Get(file)
}
Enter fullscreen mode Exit fullscreen mode

All magic included into first line: //go:generate .... This call Go compiler to use methods from this file for code generate by another.

Create template and functions for blob file (generator.go)

//+build ignore

package main

import (
    "bytes"
    "fmt"
    "go/format"
    "io/ioutil"
    "log"
    "os"
    "path/filepath"
    "strings"
    "text/template"
)

const (
    blobFileName string = "blob.go"
    embedFolder  string = "../../static"
)

// Define vars for build template
var conv = map[string]interface{}{"conv": fmtByteSlice}
var tmpl = template.Must(template.New("").Funcs(conv).Parse(`package box

// Code generated by go generate; DO NOT EDIT.

func init() {
    {{- range $name, $file := . }}
        box.Add("{{ $name }}", []byte{ {{ conv $file }} })
    {{- end }}
}`),
)

func fmtByteSlice(s []byte) string {
    builder := strings.Builder{}

    for _, v := range s {
        builder.WriteString(fmt.Sprintf("%d,", int(v)))
    }

    return builder.String()
}

func main() {
    // Checking directory with files
    if _, err := os.Stat(embedFolder); os.IsNotExist(err) {
        log.Fatal("Configs directory does not exists!")
    }

    // Create map for filenames
    configs := make(map[string][]byte)

    // Walking through embed directory
    err := filepath.Walk(embedFolder, func(path string, info os.FileInfo, err error) error {
        relativePath := filepath.ToSlash(strings.TrimPrefix(path, embedFolder))

        if info.IsDir() {
            // Skip directories
            log.Println(path, "is a directory, skipping...")
            return nil
        } else {
            // If element is a simple file, embed
            log.Println(path, "is a file, packing in...")

            b, err := ioutil.ReadFile(path)
            if err != nil {
                // If file not reading
                log.Printf("Error reading %s: %s", path, err)
                return err
            }

            // Add file name to map
            configs[relativePath] = b
        }

        return nil
    })
    if err != nil {
        log.Fatal("Error walking through embed directory:", err)
    }

    // Create blob file
    f, err := os.Create(blobFileName)
    if err != nil {
        log.Fatal("Error creating blob file:", err)
    }
    defer f.Close()

    // Create buffer
    builder := &bytes.Buffer{}

    // Execute template
    if err = tmpl.Execute(builder, configs); err != nil {
        log.Fatal("Error executing template", err)
    }

    // Formatting generated code
    data, err := format.Source(builder.Bytes())
    if err != nil {
        log.Fatal("Error formatting generated code", err)
    }

    // Writing blob file
    if err = ioutil.WriteFile(blobFileName, data, os.ModePerm); err != nil {
        log.Fatal("Error writing blob file", err)
    }
}
Enter fullscreen mode Exit fullscreen mode

At this point, we set //+build ignore on first line, because this code needed only for code generation process and don't be in main build.

generate it

We're almost done! 馃憤

Let's go through the block file generation process itself in more detail.

So, running command:

$ make generate
Enter fullscreen mode Exit fullscreen mode

...it created for us blob.go file (on current project folder) with this code:

package box

// Code generated by go generate; DO NOT EDIT.

func init() {
    box.Add("/index.html", []byte{35, 32, ...}
    // ...
}
Enter fullscreen mode Exit fullscreen mode

Basically, we converted the files in the ./static folder to a slice of bytes and now we can easily add them (in this clean form) to our binary executable! 馃帀

Time to use it all in your code

package main

import (
    "log"
    "net/http"
    "text/template"

    "github.com/koddr/example-embed-static-files-go/internal/box"
)

type PageData struct {
    Title       string
    Heading     string
    Description string
}

func main() {
    // Define embed file (for a short)
    index := string(box.Get("/index.html"))

    // Template
    tmpl := template.Must(template.New("").Parse(index))

    // Handle function
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        data := PageData{
            Title:       "The easiest way to embed static files into a binary file",
            Heading:     "This is easiest way",
            Description: "My life credo is 'If you can not use N, do not use N'.",
        }

        // Execute template with data
        if err := tmpl.Execute(w, data); err != nil {
            log.Fatal(err)
        }
    })

    // Start server
    if err := http.ListenAndServe(":8080", nil); err != nil {
        log.Fatal(err)
    }
}
Enter fullscreen mode Exit fullscreen mode

result

鈽濓笍 Always use / for looking up! For example: /images/logo.jpg is actually ./static/images/logo.jpg.

Exercises 馃

  • Improve box.Get() function for getting all files from embed folder at once call (for example, by using wildcard). Like box.Get("/images/*") or similar.
  • Exclude path to directory of static files to flags and when call it on go:generate.

Final words

I understand, that it would be much better to take a ready-made library and use its API... but then you would never get to know this wonderful tool and write your first code generator!

Invest in yourself and your horizons! 馃巵

Photo by

[Title] Vitor Santos https://unsplash.com/photos/GOQ32dlahDk
[1] Brunno Tozzo https://unsplash.com/photos/t32lrFimPlU
[2] Christin Hume https://unsplash.com/photos/Hcfwew744z4

P.S.

If you want more 鈥 write a comment below & follow me. Thx! 馃槝

Discussion (7)

Collapse
fguisso profile image
Fernando Guisso

Hi, great article, I'll try to implement in my app.
2 questions:

  • Do you have any best form to serve all static file automatic? because a have this one
static/css/*.css
static/image/*.png
static/js/*js
Enter fullscreen mode Exit fullscreen mode

too many files to create one endpoint for each one and point the blob.

  • If I have a lot of files in /static, all files is load when my app start? do you have problems with memory?
Collapse
koddr profile image
Vic Sh贸stak Author

Hello! 馃憢 Thanks for reply.

  1. If you've many files to embed, you can create little helper func for its append to binary. For example, with filepath.Walk.

  2. I'm not sure, but on my some huge React.js app it works perfectly, as usual. But, some benchmarks it's better... 馃

Collapse
fguisso profile image
Fernando Guisso

my solution:

for filename, content := range static.Map() {                               
         r.HandleFunc(fmt.Sprintf("/static%v", filename),                        
             func(w http.ResponseWriter, r *http.Request) {                      
                 http.ServeContent(w, r, "test.txt", time.Now(),                 
                 bytes.NewReader(content))                                                                  
         })                                                                                                 
}
Enter fullscreen mode Exit fullscreen mode

static is my box pkg and the .Map just return the storages map

The problem now is because I use the html/template to generate some pages, and I can't use the funcs of this pkg :(

Collapse
josegonzalez profile image
Jose Diaz-Gonzalez

Do you know of a decent way to use this pattern but also add support for either partials or layouts? I think thats maybe the only thing missing from this, and I don't think I'm quite at the level of coming up with my own solution :D

Collapse
koddr profile image
Vic Sh贸stak Author

Hi! Please write in more detail what is unclear and how you would like to improve or change something in the article. I will try to improve the article :)

Collapse
kylidboy profile image
Kyli

Great job. But placing the "//go:generate go run generator.go" in the generator.go a better choice?

Collapse
koddr profile image
Vic Sh贸stak Author

Thanks. Why not to do this? Please write in more detail what is unclear ;)