From Zero to Hero: Building a Key Issuance Server with `verbose` and `figtree`

In August 2024 Andrei Merlescu wrote a package called verbose and released it under the Apache 2.0 license. The idea was novel: when interacting with sensitive information during multi-region deployments, you want to print to STDOUT a censored expression of data while writing the unredacted form to log files — or in this tutorial's case, scrub secrets from the log file entirely. Private keys, passwords, serial numbers — these are sensitive and should not stream through STDOUT in devops contexts, especially in CI pipelines like GitHub Actions. With verbose, your runtime can register new secrets as they become available, concurrently and safely.

go get -u github.com/andreimerlescu/verbose

aSecret := "abc123"
// SecretBytes cast is required — AddSecret takes verbose.SecretBytes, not string
if err := verbose.AddSecret(verbose.SecretBytes(aSecret), "***"); err != nil {
    log.Fatal(err)
}
stdoutLogger := log.New(os.Stdout, "", log.LstdFlags)
// Tof takes a *log.Logger, not an io.Writer
verbose.Tof(stdoutLogger, "this is a secret: %s", aSecret)

Prints to STDOUT:

this is a secret: ***

---

Step 1 — Scaffold and dependencies

You are going to build a running HTTP server that issues, verifies, and revokes API keys. Every secret the server touches — the admin token that protects your endpoints, every key it issues — will be registered with the verbose logging package the moment it is created. By the time you finish, you will be able to grep your own log file for any plaintext secret and find nothing. That is the goal.

verbose is a logging package that scrubs registered secrets from every line it writes. figtree is a configuration package that loads values from a YAML file, environment variables, and CLI flags in that priority order, with validators and mutation tracking built in.

Create the project:

mkdir keyserver && cd keyserver
go mod init github.com/example/keyserver
go get github.com/andreimerlescu/verbose@latest
go get github.com/andreimerlescu/figtree/v2@latest

Create the files you will fill in across the remaining steps:

touch main.go config.yml

Your go.mod should now look like this:

module github.com/example/keyserver

go 1.22

require (
    github.com/andreimerlescu/figtree/v2 v2.0.14
    github.com/andreimerlescu/verbose    v0.2.0
)

Your directory:

keyserver/
├── go.mod
├── go.sum
├── config.yml
└── main.go

Nothing runs yet. That is fine. You have a clean workspace and both packages are available.

---

Step 2 — CLI design with figtree

Now that the workspace is ready, the first thing to design is configuration. Before any HTTP server, any logger, any route — you need to know what the program accepts and where it reads it from. figtree handles all three sources — file, environment, flags — in one tree.

Every config key is a Go constant. This is not optional style — it prevents typos from silently creating a second key that is never read.

Add this to main.go:

// main.go — complete file at this stage

package main

import (
    "fmt"
    "os"
    "time"

    "github.com/andreimerlescu/figtree/v2"
)

// config keys — always constants, never raw strings at call sites
const (
    kHost          = "host"
    kPort          = "port"
    kLogDir        = "log-dir"
    kTruncate      = "truncate"
    kKeyPrefix     = "key-prefix"
    kAdminToken    = "keyserver-admin-token" // env: KEYSERVER_ADMIN_TOKEN
    kOnCall        = "on-call-engineer"      // env: ON_CALL_ENGINEER
    kShiftDuration = "shift-duration"        // env: SHIFT_DURATION
)

func main() {
    // Grow a tracked tree — Tracking enables the Mutations() channel,
    // Pollinate re-checks env vars on every Getter call (needed for Step 7),
    // ConfigFile makes the intent explicit: this app is config-file-first.
    figs := figtree.With(figtree.Options{
        Tracking:   true,
        Germinate:  true,           // ignore -test.* flags during go test
        Pollinate:  true,           // re-read env vars on every Getter call
        ConfigFile: "./config.yml",
    })

    // -- server --
    figs.NewString(kHost, "127.0.0.1", "host address to listen on")
    figs.NewInt(kPort, 8080, "port to listen on")

    // -- logging --
    figs.NewString(kLogDir, "./logs", "directory to write log files into")
    figs.NewBool(kTruncate, false, "truncate log file on each run")

    // -- keys --
    figs.NewString(kKeyPrefix, "ks_", "prefix for issued API keys")

    // RuleNoFlags means figtree will never register a CLI flag for this key.
    // The only way to supply it is via KEYSERVER_ADMIN_TOKEN or config.yml.
    // The key name is deliberately long — verbose by design.
    figs.NewString(kAdminToken, "", "admin token — use KEYSERVER_ADMIN_TOKEN env var only")
    figs.WithRule(kAdminToken, figtree.RuleNoFlags)

    // -- on-call rotation --
    figs.NewString(kOnCall, "Darron", "name of the engineer currently on call")

    // shift-duration drives the rotation goroutine added in Step 3.
    // Default is 7 hours. SHIFT_DURATION=1 overrides it for local testing.
    figs.NewUnitDuration(kShiftDuration, 7, time.Hour, "length of one on-call shift in hours")

    // Problems() catches developer mistakes — duplicate keys, bad validator
    // combos — before Load() runs. These are your bugs, not user input errors.
    if problems := figs.Problems(); len(problems) > 0 {
        for _, p := range problems {
            fmt.Fprintf(os.Stderr, "figtree problem: %v\n", p)
        }
        os.Exit(1)
    }

    // Load resolves: config.yml → env vars → CLI flags (in that priority order)
    if err := figs.Load(); err != nil {
        fmt.Fprintf(os.Stderr, "figtree.Load: %v\n", err)
        os.Exit(1)
    }

    // Print everything so you can see what loaded — fmt.Println for now,
    // verbose replaces this in Step 4.
    fmt.Printf("host=%s port=%d log-dir=%s truncate=%v\n",
        *figs.String(kHost),
        *figs.Int(kPort),
        *figs.String(kLogDir),
        *figs.Bool(kTruncate),
    )
    fmt.Printf("key-prefix=%s on-call=%s shift=%s\n",
        *figs.String(kKeyPrefix),
        *figs.String(kOnCall),
        *figs.Duration(kShiftDuration),
    )
    fmt.Printf("admin-token=%s\n", *figs.String(kAdminToken))
}

Create config.yml:

host:             "127.0.0.1"
port:             8080
log-dir:          "./logs"
truncate:         false
key-prefix:       "ks_"
on-call-engineer: "Darron"
shift-duration:   7
# keyserver-admin-token is intentionally absent from source control.
# Set it via: export KEYSERVER_ADMIN_TOKEN=your-secret

Run it:

KEYSERVER_ADMIN_TOKEN=mysecret go run .

You should see all values printed. Notice admin-token=mysecret is in your terminal output right now. That is expected — verbose is not wired yet. You will fix that in Step 4 and never see it in plaintext again.

Mistake trap: Try running go run . -keyserver-admin-token=mysecret. figtree will not register that flag because of RuleNoFlags — the flag is unknown to the flag package and your program will exit with an error. The env var is the only valid path. This is intentional.

---

Step 3 — Validators, Problems(), and mutations

Now that the tree is declared, you lock it down. Validators reject bad input before your program does anything with it. The Problems() check you already have catches declaration mistakes. The mutations goroutine watches for runtime changes — which is where the on-call rotation lives.

Add these diffs to main.go:

After figs.NewString(kHost, ...) — add validator:

    // 3 lines above
    figs.NewString(kHost, "127.0.0.1", "host address to listen on")
    // add this
    figs.WithValidator(kHost, figtree.AssureStringNotEmpty)
    // 3 lines below
    figs.NewInt(kPort, 8080, "port to listen on")

After figs.NewInt(kPort, ...) — add validator:

    // 3 lines above
    figs.NewInt(kPort, 8080, "port to listen on")
    // add this
    figs.WithValidator(kPort, figtree.AssureIntInRange(1024, 65535))
    // 3 lines below
    figs.NewString(kLogDir, "./logs", "directory to write log files into")

After figs.NewString(kLogDir, ...) — add validator:

    // 3 lines above
    figs.NewString(kLogDir, "./logs", "directory to write log files into")
    // add this
    figs.WithValidator(kLogDir, figtree.AssureStringNotEmpty)
    // 3 lines below
    figs.NewBool(kTruncate, false, "truncate log file on each run")

After figs.NewString(kKeyPrefix, ...) — add validator:

    // 3 lines above
    figs.NewString(kKeyPrefix, "ks_", "prefix for issued API keys")
    // add this
    figs.WithValidator(kKeyPrefix, figtree.AssureStringNotEmpty)
    // 3 lines below
    figs.NewString(kAdminToken, "", "admin token — use KEYSERVER_ADMIN_TOKEN env var only")

After figs.WithRule(kAdminToken, ...) — add validator:

    // 3 lines above
    figs.WithRule(kAdminToken, figtree.RuleNoFlags)
    // add this — admin token must be present at startup
    figs.WithValidator(kAdminToken, figtree.AssureStringNotEmpty)
    // 3 lines below
    figs.NewString(kOnCall, "Darron", "name of the engineer currently on call")

After figs.NewString(kOnCall, ...) — add validator:

    // 3 lines above
    figs.NewString(kOnCall, "Darron", "name of the engineer currently on call")
    // add this
    figs.WithValidator(kOnCall, figtree.AssureStringNotEmpty)
    // 3 lines below
    figs.NewUnitDuration(kShiftDuration, 7, time.Hour, "length of one on-call shift in hours")

After figs.NewUnitDuration(kShiftDuration, ...) — add validators:

    // 3 lines above
    figs.NewUnitDuration(kShiftDuration, 7, time.Hour, "length of one on-call shift in hours")
    // add these — shifts must be at least 1 hour, at most 12
    figs.WithValidator(kShiftDuration, figtree.AssureDurationMin(1*time.Hour))
    figs.WithValidator(kShiftDuration, figtree.AssureDurationMax(12*time.Hour))
    // 3 lines below
    if problems := figs.Problems(); len(problems) > 0 {

Now add the mutations goroutine and the on-call shift rotation. Both go after figs.Load() — the tree must be live before you watch it:

    // 3 lines above
    if err := figs.Load(); err != nil {
        fmt.Fprintf(os.Stderr, "figtree.Load: %v\n", err)
        os.Exit(1)
    }
    // add this block
    // watch for any config value changing at runtime and log it
    go func() {
        for m := range figs.Mutations() {
            // verbose.Printf replaces this in Step 4
            fmt.Printf("config mutation: %s changed from %v to %v at %s\n",
                m.Property, m.Old, m.New, m.When)
        }
    }()

    // on-call rotation — fires every minute, updates kOnCall based on
    // the current hour. We check every minute so the change is never
    // more than 60 seconds late.
    go func() {
        for range time.Tick(time.Minute) {
            figs.StoreString(kOnCall, onCallEngineer())
        }
    }()
    // 3 lines below
    fmt.Printf("host=%s port=%d log-dir=%s truncate=%v\n",

Add the onCallEngineer function at the bottom of main.go, outside main:

// onCallEngineer returns the name of the engineer on call right now.
//
//   04:00–07:59  → TEAM     (everyone expected online)
//   08:00–14:59  → Darron
//   15:00–21:59  → Bradley
//   22:00–03:59  → Kevin    (crosses midnight)
func onCallEngineer() string {
    h := time.Now().Hour()
    switch {
    case h >= 4 && h < 8:
        return "TEAM"
    case h >= 8 && h < 15:
        return "Darron"
    case h >= 15 && h < 22:
        return "Bradley"
    default: // 22:00–03:59
        return "Kevin"
    }
}

Run it again:

KEYSERVER_ADMIN_TOKEN=mysecret go run .

To prove the validator works, temporarily set port: 99999 in config.yml and run again. You will see figtree refuse to load with a clear error. Set it back to 8080.

Mistake trap: Move the mutations goroutine to before figs.Load() and run it. The channel exists but the tree has not resolved its values yet — mutations fired during load will not be seen because the goroutine may not be scheduled before load completes. Always start the mutations goroutine after Load() returns.

---

Step 4 — Wire verbose, replace fmt, register the admin token

Now that figtree is loading and validating configuration correctly, you can wire up the logger. verbose needs to be initialised before any log call — and the admin token needs to be registered as a secret immediately after, before anything else in the program touches it.

Add verbose to your imports:

import (
    "fmt"
    "os"
    "time"
    // add this
    "github.com/andreimerlescu/verbose"

    "github.com/andreimerlescu/figtree/v2"
)

Add the verbose initialisation block right after the goroutines:

    // 3 lines above
    go func() {
        for range time.Tick(time.Minute) {
            figs.StoreString(kOnCall, onCallEngineer())
        }
    }()
    // add this block
    // initialise verbose before any log call — guard() will stderr and no-op
    // if you call verbose functions before this point
    if err := verbose.NewLogger(verbose.Options{
        Dir:      *figs.String(kLogDir),
        Name:     "keyserver",
        Truncate: *figs.Bool(kTruncate),
        DirMode:  0o755,
        FileMode: 0o640,
    }); err != nil {
        fmt.Fprintf(os.Stderr, "verbose.NewLogger: %v\n", err)
        os.Exit(1)
    }

    // read the admin token once — this is the value we protect at startup
    adminToken := *figs.String(kAdminToken)

    // register the admin token as a verbose secret immediately.
    // verbose stores only the SHA-512 digest — the plaintext is never retained.
    // every log line written after this point that contains the raw token value
    // will have it replaced with [ADMIN_TOKEN] automatically.
    if err := verbose.AddSecret(verbose.SecretBytes(adminToken), "[ADMIN_TOKEN]"); err != nil {
        fmt.Fprintf(os.Stderr, "verbose.AddSecret: %v\n", err)
        os.Exit(1)
    }

    // proof that scrubbing is active — this line contains the raw token.
    // open logs/keyserver.log after running and you will see [ADMIN_TOKEN].
    verbose.Printf("admin token registered — value in log: %s", adminToken)
    // Version() is a function call, not a variable
    verbose.Printf("keyserver starting (verbose v%s / figtree v%s)",
        verbose.VERSION, figtree.Version())
    // 3 lines below
    fmt.Printf("host=%s port=%d log-dir=%s truncate=%v\n",

Replace the fmt.Printf startup summary:

    // remove this
    fmt.Printf("host=%s port=%d log-dir=%s truncate=%v\n",
        *figs.String(kHost),
        *figs.Int(kPort),
        *figs.String(kLogDir),
        *figs.Bool(kTruncate),
    )
    fmt.Printf("key-prefix=%s on-call=%s shift=%s\n",
        *figs.String(kKeyPrefix),
        *figs.String(kOnCall),
        *figs.Duration(kShiftDuration),
    )
    fmt.Printf("admin-token=%s\n", *figs.String(kAdminToken))
    // replace with this
    verbose.Printf("host=%s port=%d log-dir=%s truncate=%v key-prefix=%s on-call=%s shift=%s",
        *figs.String(kHost),
        *figs.Int(kPort),
        *figs.String(kLogDir),
        *figs.Bool(kTruncate),
        *figs.String(kKeyPrefix),
        *figs.String(kOnCall),
        *figs.Duration(kShiftDuration),
    )

Replace the fmt.Printf in the mutations goroutine:

    // 3 lines above
    go func() {
        for m := range figs.Mutations() {
            // remove this
            fmt.Printf("config mutation: %s changed from %v to %v at %s\n",
                m.Property, m.Old, m.New, m.When)
            // replace with this
            verbose.Printf("config mutation: %s changed from %v to %v at %s",
                m.Property, m.Old, m.New, m.When)
        }
    }()

Run it:

KEYSERVER_ADMIN_TOKEN=mysecret go run .

Open logs/keyserver.log. Find the line containing "admin token registered". The value mysecret does not appear — you will see [ADMIN_TOKEN] in its place. From this point forward, mysecret cannot appear in any log line this process writes.

Mistake trap: Call verbose.Printf("test") before verbose.NewLogger — move it above the NewLogger block and run. You will see the message printed to stderr with a "NewLogger or SetLogger has not been called" prefix. verbose does not panic — it fails open to stderr. Move the call back below NewLogger.

---

Step 5 — HTTP server skeleton

With logging and configuration solid, the server can be wired up. The three routes are stubs for now — each returns 200 OK with a placeholder body. The requireAdmin middleware is the important piece here: it logs every auth attempt, which means the admin token will be scrubbed from those log lines automatically.

Add net/http and encoding/json to your imports:

import (
    "fmt"
    "os"
    "time"
    // add these
    "encoding/json"
    "net/http"

    "github.com/andreimerlescu/verbose"
    "github.com/andreimerlescu/figtree/v2"
)

Add the server block at the end of main:

    // 3 lines above
    verbose.Printf("host=%s port=%d log-dir=%s truncate=%v key-prefix=%s on-call=%s shift=%s",
        *figs.String(kHost),
        *figs.Int(kPort),
        *figs.String(kLogDir),
        *figs.Bool(kTruncate),
        *figs.String(kKeyPrefix),
        *figs.String(kOnCall),
        *figs.Duration(kShiftDuration),
    )
    // add this block
    // capture prefix once — used in handlers added in Step 6
    prefix := *figs.String(kKeyPrefix)
    _ = prefix

    // POST /keys — issue a new API key (admin only)
    http.HandleFunc("POST /keys", requireAdmin(adminToken, func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(map[string]string{"status": "not implemented"})
    }))

    // GET /keys/{id}/verify — verify a presented key
    http.HandleFunc("GET /keys/{id}/verify", func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(map[string]string{"status": "not implemented"})
    })

    // DELETE /keys/{id} — revoke and remove a key (admin only)
    http.HandleFunc("DELETE /keys/{id}", requireAdmin(adminToken, func(w http.ResponseWriter, r *http.Request) {
        w.WriteHeader(http.StatusNoContent)
    }))

    addr := fmt.Sprintf("%s:%d", *figs.String(kHost), *figs.Int(kPort))
    verbose.Printf("keyserver listening on http://%s", addr)

    if err := http.ListenAndServe(addr, nil); err != nil {
        // ListenAndServe only returns on error — log it with a trace and exit
        fmt.Fprintf(os.Stderr, "http.ListenAndServe: %v\n",
            verbose.TracefReturn("ListenAndServe failed: %v", err))
        os.Exit(1)
    }
    // 3 lines below — end of main

Add requireAdmin and bearerToken below onCallEngineer:

// requireAdmin rejects requests whose Bearer token does not match adminToken.
// The token is already a verbose secret — the log line is safe.
func requireAdmin(adminToken string, next http.HandlerFunc) http.HandlerFunc {
    return func(w http.ResponseWriter, r *http.Request) {
        presented := bearerToken(r)
        verbose.Printf("%s %s — auth attempt from %s token=%s",
            r.Method, r.URL.Path, r.RemoteAddr, presented)
        if presented != adminToken {
            verbose.Printf("%s %s — FORBIDDEN", r.Method, r.URL.Path)
            http.Error(w, "forbidden", http.StatusForbidden)
            return
        }
        next(w, r)
    }
}

// bearerToken extracts the token from Authorization: Bearer <token>
func bearerToken(r *http.Request) string {
    auth := r.Header.Get("Authorization")
    if len(auth) > 7 && auth[:7] == "Bearer " {
        return auth[7:]
    }
    return ""
}

Run the server and curl the stubs:

KEYSERVER_ADMIN_TOKEN=mysecret go run . &

curl -s -X POST http://localhost:8080/keys \
  -H "Authorization: Bearer mysecret"

curl -s http://localhost:8080/keys/test/verify

curl -s -X DELETE http://localhost:8080/keys/test \
  -H "Authorization: Bearer mysecret"

kill %1

Check logs/keyserver.log. The auth lines show token=[ADMIN_TOKEN] — never mysecret.

Mistake trap: If you are on Go below 1.22, http.HandleFunc("POST /keys", ...) will not compile — the method prefix in the pattern is a 1.22 feature. Run go version to check. If needed, update your toolchain: go get toolchain@go1.22.0.

---

Step 6 — Issuance, verification, revocation, and runtime secret registration

This is the core of the application and the core demonstration of verbose. The moment a key is generated, verbose.AddSecret is called before anything else — before the key is stored, before it is logged, before it is returned to the caller. The window between generation and registration is zero.

When a key is deleted, verbose.RemoveSecret is called. This is safe because a deleted key is dead — no future request will present it, so there is no future log line to protect. Removing it keeps the verbose runtime lean: the secrets registry does not grow forever, and scanning each log line is O(n) in the number of registered secrets. In a long-running server issuing thousands of keys, this matters. The same pattern applies to user sessions: register the session token on login, remove it on logout.

Add the key store above main:

// 3 lines above — imports
// add this block after imports, before main

// keyRecord holds metadata for one issued API key.
// The token value is the map key — never stored inside the struct.
type keyRecord struct {
    ID       string    `json:"id"`
    IssuedAt time.Time `json:"issued_at"`
    IssuedTo string    `json:"issued_to"`
    Revoked  bool      `json:"revoked"`
}

var (
    keyStoreMu sync.RWMutex
    keyStore   = make(map[string]*keyRecord)
)
// 3 lines below — func main() {

Add sync, crypto/rand, and encoding/hex to imports:

    // add to import block
    "crypto/rand"
    "encoding/hex"
    "sync"

Replace the POST /keys stub:

    // 3 lines above
    http.HandleFunc("POST /keys", requireAdmin(adminToken, func(w http.ResponseWriter, r *http.Request) {
    // replace body
        var body struct {
            IssuedTo string `json:"issued_to"`
        }
        if err := json.NewDecoder(r.Body).Decode(&body); err != nil || body.IssuedTo == "" {
            verbose.Printf("POST /keys — bad request from %s: %v", r.RemoteAddr, err)
            http.Error(w, "issued_to is required", http.StatusBadRequest)
            return
        }

        token, err := generateToken(prefix)
        if err != nil {
            // TracefReturn logs with a stack trace and returns the error —
            // capture the return value, do not discard it
            verbose.Printf("POST /keys — token generation failed: %v",
                verbose.TracefReturn("generateToken: %v", err))
            http.Error(w, "internal error", http.StatusInternalServerError)
            return
        }

        // register with verbose FIRST — before store, before log, before response.
        // after this line the token cannot appear in plaintext in any log output.
        if err := verbose.AddSecret(verbose.SecretBytes(token), "[ISSUED_KEY]"); err != nil {
            verbose.Printf("POST /keys — failed to protect token: %v", err)
            http.Error(w, "internal error", http.StatusInternalServerError)
            return
        }

        rec := &keyRecord{
            ID:       token,
            IssuedAt: time.Now().UTC(),
            IssuedTo: body.IssuedTo,
        }

        keyStoreMu.Lock()
        keyStore[token] = rec
        keyStoreMu.Unlock()

        // safe to log — token is already scrubbed to [ISSUED_KEY]
        verbose.Printf("POST /keys — issued key=%s to=%s at=%s",
            token, body.IssuedTo, rec.IssuedAt.Format(time.RFC3339))

        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(http.StatusCreated)
        json.NewEncoder(w).Encode(map[string]string{
            "token":     token,
            "issued_to": body.IssuedTo,
            "issued_at": rec.IssuedAt.Format(time.RFC3339),
        })
    }))
    // 3 lines below

Replace the GET /keys/{id}/verify stub:

    // 3 lines above
    http.HandleFunc("GET /keys/{id}/verify", func(w http.ResponseWriter, r *http.Request) {
    // replace body
        presented := bearerToken(r)
        id := r.PathValue("id")

        // log the attempt — if the caller sent a valid token it is already
        // a verbose secret and will be scrubbed automatically
        verbose.Printf("GET /keys/%s/verify — attempt from %s token=%s",
            id, r.RemoteAddr, presented)

        keyStoreMu.RLock()
        rec, exists := keyStore[presented]
        keyStoreMu.RUnlock()

        if !exists || rec.Revoked {
            verbose.Printf("GET /keys/%s/verify — INVALID", id)
            http.Error(w, "invalid or revoked key", http.StatusUnauthorized)
            return
        }

        verbose.Printf("GET /keys/%s/verify — VALID issued_to=%s", id, rec.IssuedTo)
        w.Header().Set("Content-Type", "application/json")
        json.NewEncoder(w).Encode(map[string]any{
            "valid":     true,
            "issued_to": rec.IssuedTo,
            "issued_at": rec.IssuedAt.Format(time.RFC3339),
        })
    })
    // 3 lines below

Replace the DELETE /keys/{id} stub:

    // 3 lines above
    http.HandleFunc("DELETE /keys/{id}", requireAdmin(adminToken, func(w http.ResponseWriter, r *http.Request) {
    // replace body
        id := r.PathValue("id")

        keyStoreMu.Lock()
        rec, exists := keyStore[id]
        if exists {
            rec.Revoked = true
        }
        keyStoreMu.Unlock()

        if !exists {
            verbose.Printf("DELETE /keys/%s — not found", id)
            http.Error(w, "key not found", http.StatusNotFound)
            return
        }

        // the key is dead — safe to remove from verbose.
        // removing it keeps the secrets registry lean: verbose scans every
        // log line against every registered secret, so a smaller registry
        // means faster logging. do not leave dead secrets registered forever.
        if err := verbose.RemoveSecret(verbose.SecretBytes(id)); err != nil {
            verbose.Printf("DELETE /keys/%s — RemoveSecret error: %v", id, err)
        }

        verbose.Printf("DELETE /keys/%s — revoked and removed from verbose (was issued to %s)",
            id, rec.IssuedTo)

        w.WriteHeader(http.StatusNoContent)
    }))
    // 3 lines below

Add generateToken below bearerToken:

// generateToken returns a cryptographically random token with the given prefix.
func generateToken(prefix string) (string, error) {
    b := make([]byte, 24)
    if _, err := rand.Read(b); err != nil {
        return "", err
    }
    return prefix + hex.EncodeToString(b), nil
}

Mistake trap: Move verbose.AddSecret to after keyStoreMu.Lock() — just two lines later — and run the server. Issue a key. Look at the log line for POST /keys — issued key=. You will see the raw token value in plaintext because it was logged by requireAdmin in the auth check before it was registered. Order is not a style preference here. It is the security guarantee. Move AddSecret back to first.

---

Step 7 — The on-call engineer shift in action

The rotation goroutine has been running since Step 3, quietly updating a value. Now that verbose is wired, the mutation log entries are meaningful — and with Pollinate: true, an engineer can override the on-call value live from a second terminal without restarting the server.

The shift goroutine is already in place. The mutations goroutine is already using verbose.Printf from Step 4. Run the server and open a second terminal:

# terminal 1
KEYSERVER_ADMIN_TOKEN=mysecret go run .

# terminal 2 — override the on-call engineer without restarting
export ON_CALL_ENGINEER=Kevin

Watch logs/keyserver.log — within one minute you will see:

[VERBOSE] keyserver.go:XX: config mutation: on-call-engineer changed from Darron to Kevin at 2026-04-20...

The mutation fires because Pollinate: true causes figtree to re-read ON_CALL_ENGINEER every time figs.String(kOnCall) is called — which the rotation goroutine does every minute.

If the on-call engineer name were a sensitive internal codename — a security handle, a contractor alias, anything you would not want in a log shipped to an aggregator — one verbose.AddSecret(verbose.SecretBytes(name), "[ON_CALL]") call in the rotation goroutine would scrub it from every future log line. The pattern is identical to what you did with the admin token.

Mistake trap: Remove Pollinate: true from figtree.Options, restart the server, and set export ON_CALL_ENGINEER=Kevin again. No mutation fires. The env var changed but figtree did not re-read it because Pollinate was off. Put Pollinate: true back.

---

Step 8 — The bash test script

Create test.sh in the project root. This script is the pass/fail gate for the tutorial. If your implementation is correct it exits 0. Each assertion failure prints which step to revisit and exits immediately.

#!/usr/bin/env bash
set -euo pipefail

# ── dependency check ──────────────────────────────────────────────────────────
if ! command -v curl &>/dev/null; then
    echo "ERROR: curl is required. Install it and rerun."
    exit 1
fi
if ! command -v jq &>/dev/null; then
    echo "ERROR: jq is required."
    echo "  macOS:  brew install jq"
    echo "  Linux:  sudo apt install jq  OR  sudo yum install jq"
    exit 1
fi

# ── config ────────────────────────────────────────────────────────────────────
ADMIN_TOKEN="test-admin-secret-1234"
BASE="http://127.0.0.1:8080"
LOG_FILE="./logs/keyserver.log"

# ── helpers ───────────────────────────────────────────────────────────────────
pass() { echo "  PASS: $1"; }
fail() { echo "  FAIL: $1 — $2"; exit 1; }

assert_status() {
    local label="$1" expected="$2" actual="$3" step="$4"
    if [ "$actual" -ne "$expected" ]; then
        fail "$label" "expected HTTP $expected, got $actual (see step $step)"
    fi
    pass "$label"
}

assert_no_plaintext() {
    local label="$1" secret="$2" step="$3"
    if grep -qF "$secret" "$LOG_FILE" 2>/dev/null; then
        fail "$label" "plaintext secret found in log — verbose.AddSecret not called before logging (see step $step)"
    fi
    pass "$label"
}

assert_log_contains() {
    local label="$1" pattern="$2" step="$3"
    if ! grep -qF "$pattern" "$LOG_FILE" 2>/dev/null; then
        fail "$label" "expected '$pattern' in log — not found (see step $step)"
    fi
    pass "$label"
}

# ── build and start server ────────────────────────────────────────────────────
echo "=> building..."
go build -o keyserver_bin . || { echo "Build failed — fix compilation errors first."; exit 1; }

echo "=> starting server..."
KEYSERVER_ADMIN_TOKEN="$ADMIN_TOKEN" ./keyserver_bin &
SERVER_PID=$!
trap 'kill $SERVER_PID 2>/dev/null; rm -f keyserver_bin' EXIT

# wait for server to be ready — retry up to 10 times
for i in $(seq 1 10); do
    if curl -sf "$BASE/keys/test/verify" &>/dev/null; then
        break
    fi
    sleep 0.5
done

echo ""
echo "── assertion 1: admin token is scrubbed from logs (Step 4) ─────────────"
assert_no_plaintext "admin token not in log" "$ADMIN_TOKEN" "4"

echo ""
echo "── assertion 2: issue key for alice (Step 6) ────────────────────────────"
ALICE_RESP=$(curl -s -w "\n%{http_code}" -X POST "$BASE/keys" \
    -H "Authorization: Bearer $ADMIN_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"issued_to":"[e-mail]"}')
ALICE_STATUS=$(echo "$ALICE_RESP" | tail -1)
ALICE_BODY=$(echo "$ALICE_RESP" | head -1)
assert_status "POST /keys alice" 201 "$ALICE_STATUS" "6"
ALICE_TOKEN=$(echo "$ALICE_BODY" | jq -r .token)

echo ""
echo "── assertion 3: issue key for bob (Step 6) ──────────────────────────────"
BOB_RESP=$(curl -s -w "\n%{http_code}" -X POST "$BASE/keys" \
    -H "Authorization: Bearer $ADMIN_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"issued_to":"[e-mail]"}')
BOB_STATUS=$(echo "$BOB_RESP" | tail -1)
BOB_BODY=$(echo "$BOB_RESP" | head -1)
assert_status "POST /keys bob" 201 "$BOB_STATUS" "6"
BOB_TOKEN=$(echo "$BOB_BODY" | jq -r .token)

echo ""
echo "── assertion 4: issue key for carol (Step 6) ────────────────────────────"
CAROL_RESP=$(curl -s -w "\n%{http_code}" -X POST "$BASE/keys" \
    -H "Authorization: Bearer $ADMIN_TOKEN" \
    -H "Content-Type: application/json" \
    -d '{"issued_to":"[e-mail]"}')
CAROL_STATUS=$(echo "$CAROL_RESP" | tail -1)
CAROL_BODY=$(echo "$CAROL_RESP" | head -1)
assert_status "POST /keys carol" 201 "$CAROL_STATUS" "6"
CAROL_TOKEN=$(echo "$CAROL_BODY" | jq -r .token)

echo ""
echo "── assertion 5: verify alice (Step 6) ───────────────────────────────────"
STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/keys/${ALICE_TOKEN}/verify" \
    -H "Authorization: Bearer $ALICE_TOKEN")
assert_status "GET /keys alice/verify" 200 "$STATUS" "6"

echo ""
echo "── assertion 6: verify bob (Step 6) ─────────────────────────────────────"
STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/keys/${BOB_TOKEN}/verify" \
    -H "Authorization: Bearer $BOB_TOKEN")
assert_status "GET /keys bob/verify" 200 "$STATUS" "6"

echo ""
echo "── assertion 7: verify carol (Step 6) ───────────────────────────────────"
STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/keys/${CAROL_TOKEN}/verify" \
    -H "Authorization: Bearer $CAROL_TOKEN")
assert_status "GET /keys carol/verify" 200 "$STATUS" "6"

echo ""
echo "── assertion 8: revoke bob (Step 6) ─────────────────────────────────────"
STATUS=$(curl -s -o /dev/null -w "%{http_code}" -X DELETE "$BASE/keys/${BOB_TOKEN}" \
    -H "Authorization: Bearer $ADMIN_TOKEN")
assert_status "DELETE /keys bob" 204 "$STATUS" "6"

echo ""
echo "── assertion 9: verify bob after revocation — expect 401 (Step 6) ───────"
STATUS=$(curl -s -o /dev/null -w "%{http_code}" "$BASE/keys/${BOB_TOKEN}/verify" \
    -H "Authorization: Bearer $BOB_TOKEN")
assert_status "GET /keys bob/verify after revoke" 401 "$STATUS" "6"

echo ""
echo "── assertion 10: alice and carol tokens not in log (Step 6) ─────────────"
assert_no_plaintext "alice token not in log" "$ALICE_TOKEN" "6"
assert_no_plaintext "carol token not in log" "$CAROL_TOKEN" "6"

echo ""
echo "── assertion 11: bob token not in log (Step 6) ──────────────────────────"
# bob's token was removed via RemoveSecret after revocation.
# if the implementation is correct the token never appeared in plaintext
# because AddSecret was called before any logging.
assert_no_plaintext "bob token not in log" "$BOB_TOKEN" "6"

echo ""
echo "── assertion 12: scrub markers present in log (Step 4 + 6) ─────────────"
assert_log_contains "[ADMIN_TOKEN] in log" "[ADMIN_TOKEN]" "4"
assert_log_contains "[ISSUED_KEY] in log"  "[ISSUED_KEY]"  "6"

echo ""
echo "── assertion 13: on-call mutation visible in log (Step 7) ───────────────"
export ON_CALL_ENGINEER="TEAM"
echo "   waiting up to 90s for on-call mutation to appear in log..."
for i in $(seq 1 18); do
    if grep -qF "on-call-engineer" "$LOG_FILE" 2>/dev/null; then
        pass "on-call mutation in log"
        break
    fi
    sleep 5
    if [ "$i" -eq 18 ]; then
        fail "on-call mutation in log" "mutation not seen after 90s — check Pollinate:true and mutations goroutine (see step 7)"
    fi
done

echo ""
echo "════════════════════════════════════════════════════════════════════════"
echo "  All assertions passed. Your implementation is correct."
echo "════════════════════════════════════════════════════════════════════════"

Make it executable and run it:

chmod +x test.sh
./test.sh

If every assertion passes you see the final banner. If any assertion fails the script exits immediately with the step number to revisit. Fix that step and rerun — the script always starts fresh.

---

Step 9 — Closing: What You Built and What Comes Next

You started this tutorial with an empty directory and two go get commands. You finished with a running HTTP server that issues, verifies, and revokes API keys — and whose log file provably contains no plaintext secrets. Run ./test.sh one final time and watch every assertion pass. That exit 0 is not a formality. It is a guarantee backed by SHA-512.

Take a moment to understand what you actually built and why each piece was chosen.

verbose is not just a logger. Every other logger in the Go ecosystem — log, slog, zap, zerolog — will faithfully write whatever string you hand it. verbose is the only one that knows what your secrets are and refuses to write them. The moment you call verbose.AddSecret(verbose.SecretBytes(token), "[ISSUED_KEY]"), that token's SHA-512 digest enters a registry and every subsequent log line is scanned against it before it touches disk. The plaintext never persists inside verbose — not in memory, not on disk. When you called verbose.RemoveSecret on revocation, you kept that registry lean deliberately: a server that issues thousands of keys over its lifetime should not accumulate thousands of dead secrets in a scanner it runs on every single log call.

figtree is not just a flag parser. It is a priority-ordered configuration resolver — file, then environment, then CLI — with validators, rules, and a mutation channel that watches values change at runtime. RuleNoFlags on kAdminToken is not decoration. It is an enforcement boundary. The key name "keyserver-admin-token" being deliberately long is not an accident — it is a design statement: secrets should be inconvenient to pass on a command line because command lines end up in shell history, process lists, and CI logs.

The on-call rotation is a small thing, but it demonstrates something real: configuration is not static. Values change while a server runs. Pollinate: true and the mutations channel give you an observable, logged record of every change. In a real system that might be a database password rotating, a feature flag toggling, a rate limit updating. The pattern is identical regardless of what the value is.

---

The packages used in this tutorial are part of a larger body of open source work

Both verbose and figtree were written by Andrei Merlescu — @andreimerlescu on GitHub. His profile carries 99 public repositories built across 17 years of professional software engineering spanning Cisco, Oracle, Warner Bros. Games, and SurgePays, written primarily in Go, Ruby, PHP, and Bash.

Some repositories worth exploring alongside the two you just used:

• checkfs — filesystem existence and permission checks for File and Directory. Pairs naturally with figtree when you want to validate that a config-supplied path actually exists before your server starts.
• sema — semaphore primitives for Go. Useful when the keyserver you just built needs to limit concurrent issuance requests.
• go-passwd — safe password auditing. If your keyserver ever needs to validate that an incoming credential meets a strength policy before issuing a key, this is the package.
• summarize + aigcm — AI-assisted project summarization and git commit message generation using local Ollama models. If you are building on an M-series Mac with local models, these are worth looking at.
• lemmings — load testing built around the concept of NPCs moving through your infrastructure as simulated users across geographic terrains and pack sizes. If you want to know what your keyserver does under real traffic before you ship it, lemmings is how you find out. It was built specifically because existing tools like ab did not answer the question Andrei needed answered when he launched Trakify in 2015 and got a surge of traffic from a Barron's Magazine feature. The tool he wished he had then exists now and it is free.

---

The code is yours. The packages are free. The log file is clean.