loading...
Cover image for The easiest way to embed static files into a binary file in your Golang app (no external dependencies)

The easiest way to embed static files into a binary file in your Golang app (no external dependencies)

koddr profile image Vic ShΓ³stak ・6 min read

How to Golang (3 Part Series)

1) The easiest way to embed static files into a binary file in your Golang app (no external dependencies) 2) Let's write config for your Golang web app on right way β€” YAML πŸ‘Œ 3) β˜„οΈ How to update version's cache of your package in pkg.go.dev?

Introduction

Hello, everyone! πŸ‘‹ I have a very interesting find for you. But first, I'm going to tell you a little background story, so...

Earlier, I tried go.rice and go-bindata, then I used packr and switched to pkger from the same author (Hi, Mark Bates, if you're suddenly reading this). Yep, I was satisfied with this packages and quite happy!

But when I saw this method somewhere on the Internet, I immediately herded it into all of my Golang apps (who needed it, of course).

...and what's problem with ready-made packages?

First of all, it's external packages with their dependencies. It's good when you need to quickly make a working prototype OR don't really care about the size of your code base.

Documentation may not be complete or cover actually what you need at this moment. I have been frustrated to have to wait a long time (sometimes, up to several weeks) to get an answer OR a bug fix.

I have nothing against the authors of these great Go packages! I personally support many of Open Source repositories on GitHub and understand they very well! πŸ˜‰

Why I need this?

Good question, by the way, because many programmers do not always understand advantages of this approach to packaging a product (the final program, which you sell to your customer).

☝️ Especially, who came from scripting languages like JavaScript, Python, PHP or else.

I highlight the following:

  • It's safe. Files, which added to a binary, are impossible (or extremely difficult) to extract back.

  • It's convenient. There's no need to always remember to include a folder with static files, when you give production-ready product to your customer.

embed static files

OK! So what am I suggesting? πŸ€”

My suggest will be very simple and elegant: why not write our own packer for static files using only built-in Go tools? Yes, gophers have already figured out what I'm talking about... Go generate function!

⭐️ Please, read first this awesome blog post by Rob Pike (22 Dec 2014) for more understand how it works! Unfortunately, I can't explain this principle better and shorter than Rob did...

No time for further reading? πŸš€

I created repository with full code example on my GitHub especially for you:

GitHub logo koddr / example-embed-static-files-go

The easiest way to embed static files into a binary file in your Golang app (no external dependencies).

Just git clone and read instructions from README.

Project structure

Let's go mod init ... and create some files πŸ‘Œ

$ tree .
.
β”œβ”€β”€ Makefile
β”œβ”€β”€ go.mod
β”œβ”€β”€ cmd
β”‚   └── app
β”‚       └── main.go          <-- main file of your Go app
β”œβ”€β”€ internal
β”‚   └── box                  <-- template and functions for create blob
β”‚       β”œβ”€β”€ box.go
β”‚       └── generator.go
└── static                   <-- folder with static files
    └── index.html

Add most used commands to Makefile:

.PHONY: generate

# ⚠️
generate:
    @go generate ./...
    @echo "[OK] Files added to embed box!"

security:
    @gosec ./...
    @echo "[OK] Go security check was completed!"

build: generate security
    @go build -o ./build/server ./cmd/app/*.go
    @echo "[OK] App binary was created!"

run:
    @./build/server

⚠️ I strongly recommend to use Go security checker for your code! Maybe I'll write a separate article about this great package later.

Create methods for use on code (box.go)

//go:generate go run generator.go

package box

type embedBox struct {
    storage map[string][]byte
}

// Create new box for embed files
func newEmbedBox() *embedBox {
    return &embedBox{storage: make(map[string][]byte)}
}

// Add a file to box
func (e *embedBox) Add(file string, content []byte) {
    e.storage[file] = content
}

// Get file's content
func (e *embedBox) Get(file string) []byte {
    if f, ok := e.storage[file]; ok {
        return f
    }
    return nil
}

// Embed box expose
var box = newEmbedBox()

// Add a file content to box
func Add(file string, content []byte) {
    box.Add(file, content)
}

// Get a file from box
func Get(file string) []byte {
    return box.Get(file)
}

All magic included into first line: //go:generate .... This call Go compiler to use methods from this file for code generate by another.

Create template and functions for blob file (generator.go)

//+build ignore

package main

import (
    "bytes"
    "fmt"
    "go/format"
    "io/ioutil"
    "log"
    "os"
    "path/filepath"
    "strings"
    "text/template"
)

const (
    blobFileName string = "blob.go"
    embedFolder  string = "../../static"
)

// Define vars for build template
var conv = map[string]interface{}{"conv": fmtByteSlice}
var tmpl = template.Must(template.New("").Funcs(conv).Parse(`package box

// Code generated by go generate; DO NOT EDIT.

func init() {
    {{- range $name, $file := . }}
        box.Add("{{ $name }}", []byte{ {{ conv $file }} })
    {{- end }}
}`),
)

func fmtByteSlice(s []byte) string {
    builder := strings.Builder{}

    for _, v := range s {
        builder.WriteString(fmt.Sprintf("%d,", int(v)))
    }

    return builder.String()
}

func main() {
    // Checking directory with files
    if _, err := os.Stat(embedFolder); os.IsNotExist(err) {
        log.Fatal("Configs directory does not exists!")
    }

    // Create map for filenames
    configs := make(map[string][]byte)

    // Walking through embed directory
    err := filepath.Walk(embedFolder, func(path string, info os.FileInfo, err error) error {
        relativePath := filepath.ToSlash(strings.TrimPrefix(path, embedFolder))

        if info.IsDir() {
            // Skip directories
            log.Println(path, "is a directory, skipping...")
            return nil
        } else {
            // If element is a simple file, embed
            log.Println(path, "is a file, packing in...")

            b, err := ioutil.ReadFile(path)
            if err != nil {
                // If file not reading
                log.Printf("Error reading %s: %s", path, err)
                return err
            }

            // Add file name to map
            configs[relativePath] = b
        }

        return nil
    })
    if err != nil {
        log.Fatal("Error walking through embed directory:", err)
    }

    // Create blob file
    f, err := os.Create(blobFileName)
    if err != nil {
        log.Fatal("Error creating blob file:", err)
    }
    defer f.Close()

    // Create buffer
    builder := &bytes.Buffer{}

    // Execute template
    if err = tmpl.Execute(builder, configs); err != nil {
        log.Fatal("Error executing template", err)
    }

    // Formatting generated code
    data, err := format.Source(builder.Bytes())
    if err != nil {
        log.Fatal("Error formatting generated code", err)
    }

    // Writing blob file
    if err = ioutil.WriteFile(blobFileName, data, os.ModePerm); err != nil {
        log.Fatal("Error writing blob file", err)
    }
}

At this point, we set //+build ignore on first line, because this code needed only for code generation process and don't be in main build.

generate it

We're almost done! πŸ‘

Let's go through the block file generation process itself in more detail.

So, running command:

$ make generate

...it created for us blob.go file (on current project folder) with this code:

package box

// Code generated by go generate; DO NOT EDIT.

func init() {
    box.Add("/index.html", []byte{35, 32, ...}
    // ...
}

Basically, we converted the files in the ./static folder to a slice of bytes and now we can easily add them (in this clean form) to our binary executable! πŸŽ‰

Time to use it all in your code

package main

import (
    "log"
    "net/http"
    "text/template"

    "github.com/koddr/example-embed-static-files-go/internal/box"
)

type PageData struct {
    Title       string
    Heading     string
    Description string
}

func main() {
    // Define embed file (for a short)
    index := string(box.Get("/index.html"))

    // Template
    tmpl := template.Must(template.New("").Parse(index))

    // Handle function
    http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
        data := PageData{
            Title:       "The easiest way to embed static files into a binary file",
            Heading:     "This is easiest way",
            Description: "My life credo is 'If you can not use N, do not use N'.",
        }

        // Execute template with data
        if err := tmpl.Execute(w, data); err != nil {
            log.Fatal(err)
        }
    })

    // Start server
    if err := http.ListenAndServe(":8080", nil); err != nil {
        log.Fatal(err)
    }
}

result

☝️ Always use / for looking up! For example: /images/logo.jpg is actually ./static/images/logo.jpg.

Exercises πŸ€“

  • Improve box.Get() function for getting all files from embed folder at once call (for example, by using wildcard). Like box.Get("/images/*") or similar.
  • Exclude path to directory of static files to flags and when call it on go:generate.

Final words

I understand, that it would be much better to take a ready-made library and use its API... but then you would never get to know this wonderful tool and write your first code generator!

Invest in yourself and your horizons! 🎁

Photo by

[Title] Vitor Santos https://unsplash.com/photos/GOQ32dlahDk
[1] Brunno Tozzo https://unsplash.com/photos/t32lrFimPlU
[2] Christin Hume https://unsplash.com/photos/Hcfwew744z4

P.S.

If you want more β€” write a comment below & follow me. Thx! 😘

How to Golang (3 Part Series)

1) The easiest way to embed static files into a binary file in your Golang app (no external dependencies) 2) Let's write config for your Golang web app on right way β€” YAML πŸ‘Œ 3) β˜„οΈ How to update version's cache of your package in pkg.go.dev?

Posted on by:

koddr profile

Vic ShΓ³stak

@koddr

Hey! πŸ‘‹ I'm founder and full stack web developer (Go, JavaScript, Docker & automation) at True web artisans. Golang lover, UX evangelist, DX philosopher & UI Dreamer with over 12+ years of experience.

Discussion

markdown guide
 

Hi, great article, I'll try to implement in my app.
2 questions:

  • Do you have any best form to serve all static file automatic? because a have this one
static/css/*.css
static/image/*.png
static/js/*js

too many files to create one endpoint for each one and point the blob.

  • If I have a lot of files in /static, all files is load when my app start? do you have problems with memory?
 

Hello! πŸ‘‹ Thanks for reply.

  1. If you've many files to embed, you can create little helper func for its append to binary. For example, with filepath.Walk.

  2. I'm not sure, but on my some huge React.js app it works perfectly, as usual. But, some benchmarks it's better... πŸ€”

 

my solution:

for filename, content := range static.Map() {                               
         r.HandleFunc(fmt.Sprintf("/static%v", filename),                        
             func(w http.ResponseWriter, r *http.Request) {                      
                 http.ServeContent(w, r, "test.txt", time.Now(),                 
                 bytes.NewReader(content))                                                                  
         })                                                                                                 
}

static is my box pkg and the .Map just return the storages map

The problem now is because I use the html/template to generate some pages, and I can't use the funcs of this pkg :(