Why Does Your Testing Framework Need 17 Functions?

#go #testing #opensource

I counted Ginkgo's top-level API once. Describe, Context, When, It, Specify, By, BeforeEach, AfterEach, BeforeAll, AfterAll, JustBeforeEach, JustAfterEach, BeforeSuite, AfterSuite, SynchronizedBeforeSuite, SynchronizedAfterSuite, DeferCleanup. That's 17, and I stopped counting. There are more if you include the F and P prefixed variants for focus and pending.

GoConvey is leaner but still has Convey, So, ShouldEqual, SkipConvey, FocusConvey, Reset, and its own assertion DSL.

I kept asking myself: what's the minimum API a scoped testing framework actually needs? Not "what's nice to have" -- what's the floor?

Turns out it's one method.

The entire API

s.Test("name", fn)           // leaf test
s.Test("name", fn, builder)  // parent with children
s.Skip()                     // skip this scope

That's samurai. The full public surface fits on a napkin:

func Run(t *testing.T, builder func(*Scope), opts ...Option)
func RunWith[V Context](t, factory, builder, opts...)

type TestScope[V Context]  // one method: Test()
type Scope = TestScope[W]  // alias for the common case
type W = *BaseContext       // has Testing() and Cleanup()

Sequential()  // option
Parallel()    // option (default)

I'm not going to pretend that smaller is automatically better. Ginkgo's BeforeAll exists because people need it. But I do think the Go testing ecosystem has a complexity problem, and most of that complexity exists to manage shared mutable state between tests. If you remove the shared state, the API surface collapses.

The tradeoff

Here's what samurai does that might bother you at first: the builder function runs more than once.

samurai.Run(t, func(s *samurai.Scope) {
    var db *sql.DB  // this gets allocated multiple times

    s.Test("with database", func(ctx context.Context, w samurai.W) {
        db = openTestDB(ctx)
        w.Cleanup(func() { db.Close() })
    }, func(s *samurai.Scope) {
        s.Test("can ping", func(ctx context.Context, w samurai.W) {
            assert.NoError(w.Testing(), db.PingContext(ctx))
        })
        s.Test("can query", func(ctx context.Context, w samurai.W) {
            _, err := db.QueryContext(ctx, "SELECT 1")
            assert.NoError(w.Testing(), err)
        })
    })
})

Two leaf tests, so the builder runs twice. Each run gets its own db. The "with database" callback opens a fresh connection each time. can ping and can query never share a database handle.

This is the point. If you run can ping and can query in parallel (which samurai does by default), they're operating on completely different databases. There is no race. There is no mutex. There is no "who closes the connection first" problem. The isolation comes from the execution model, not from discipline.

Compare this with the BeforeEach pattern where setup runs once and two siblings share the result. That works until someone adds t.Parallel() and discovers the hard way that their *sql.DB pointer is getting reassigned mid-query by the other goroutine.

Side by side

Ginkgo:

var db *sql.DB

BeforeEach(func() {
    db = openTestDB()
    DeferCleanup(func() { db.Close() })
})

It("can ping", func() {
    Expect(db.Ping()).To(Succeed())
})

It("can query", func() {
    _, err := db.Query("SELECT 1")
    Expect(err).NotTo(HaveOccurred())
})

Samurai:

var db *sql.DB

s.Test("with database", func(ctx context.Context, w samurai.W) {
    db = openTestDB(ctx)
    w.Cleanup(func() { db.Close() })
}, func(s *samurai.Scope) {
    s.Test("can ping", func(ctx context.Context, w samurai.W) {
        assert.NoError(w.Testing(), db.PingContext(ctx))
    })
    s.Test("can query", func(ctx context.Context, w samurai.W) {
        _, err := db.QueryContext(ctx, "SELECT 1")
        assert.NoError(w.Testing(), err)
    })
})

Roughly the same line count. The difference is what happens at runtime. In the Ginkgo version, db is shared. In the samurai version, each leaf gets its own db because the whole closure re-runs.

What samurai doesn't have

No BeforeAll. If you need shared infrastructure across tests (a container, a server), set it up in TestMain or at the top of your test function. The framework won't help you share state between paths because that's the thing it's designed to prevent.

No built-in assertions. Use testify, use is, use t.Errorf. The RunWith generic variant lets you embed an assertion library into the test context so you don't have to pass t everywhere, but that's optional.

No Focus or Pending variants of Test. You skip with s.Skip() on the scope. You focus with go test -run. Standard Go.