DEV Community: Farhan

How I increased the performance of my database ~2x using an Adaptive Radix Trie

Farhan — Sat, 19 Mar 2022 22:30:25 +0000

بسملة

TLDR, ART increased performance of my database 2x when compared to a skiplist. The code for an Adaptive Radix Trie (ART) implemented by me is below.

The Need

I was in search of a data structure to support prefix searches for keys in a Redis like in-memory store that I'd made called FlashDB. Currently, string key-value pairs were stored on the backend as a skip list. The design choice was not the best, but to support fast query speed and prefix searches, I assumed ordering the keys in lexical order would be helpful. It wasn't efficient and the benchmarks showed the performance of simple key-value searches would be very slow.

The algorithms behind search engines are nearly countless. The most common one used is a prefix tree, also known as a Trie. Tries are used for building key-value stores and indexes, after search trees (like B-trees and red-black trees) and hash tables.

The Search

There are 3 kinds of data structures to maintain indexes in memory.

Binary tree: T tree is old and not optimized for modern CPU architecture. B+ tree is more cache-friendly but tempers with the predictive comparison from modern CPU. FAST is great for modern CPU architectures, but it does not support incremental updates.
Hash table: Efficient in memory indexes but only support single-point queries. Updates and deletion can be expensive when they require reorganising the hash table.
Tries: Have a number of very appealing properties - for example, the height of a trie is independent of the number of keys it contains, and a trie requires no rebalancing when updated. The complexity of tries grows with the length of the key, not the number of elements in the structure. They suffer bad space performance when the keys are sparse.

Paper Trail had published this interesting article long back on optimised Radix Tries. Radix Tries have good properties like:

Complexity depends on key length, not the number of elements
There no need for rebalancing: two different insert orders result in the same tree
Keys are store in lexicographical order
Keys are implicitly stored in the path

What’s wrong with ordinary tries?

Let’s imagine how we might implement a trie, as simply as we can. A single node in a trie corresponds to a key prefix; a sequence of characters that is a prefix of one or more keys in the tree. But we know what its prefix is by the path we took to get to it, so we don’t need to store it. All we need is a way to find its children.

To find the children, we need to know how to get to all the nodes that represent key prefixes one character longer than the current one, and that’s what the children array is for. Here we’re assuming an 8-bit alphabet, with 256 possible characters per letter, or a fan-out of 256.

The paper calls the width of a single character the span of the trie, and it’s a critical parameter as it determines a trade-off between the fan-out of each node, and the height of the trie, since if you can pack more of a value into a single node by having a larger span, you need fewer nodes to describe the whole string.

The problem, as you can no-doubt see, is that there’s a possibility of a lot of wasted space in each node. Imagine a radix trie containing foo, fox and fat. The root node would have one valid pointer, to f, which would have two valid pointers, to a and o. a would have a pointer to t, and o would have pointers to o and x.

So our trie would have 6 nodes, but a total of 6 * 26 = 156 pointers, of which 150 / 156 = 96% are empty and therefore wasted space! At 8-bytes per pointer, that’s already over 1K wasted space to store just 9 bytes of information.

The paper introduces ART, an adaptive radix tree (trie) for efficient indexing in main memory. It quotes the lookup performance surpasses highly tuned, read-only search trees, while supporting very efficient insertions and deletions as well. At the same time, ART is very space efficient and solves the problem of excessive worst-case space consumption, which plagues most radix trees, by adaptively choosing compact and efficient data structures for internal nodes. It maintains the data in sorted order, which enables additional operations like range scan and prefix lookup.

Perfect! Now time to implement it.

The Implementation

ART uses 3 techniques to make radix tree a viable option:

Dynamic node size
Path compression
Lazy expansion

Dynamic node size:

Inner nodes of the radix tree use a "span" s, they store 2^s pointers. So for a k-bit key, the height of the tree is k/s. Radix trees are smaller than binary trees if the number of elements n is n > 2^(k/s). For pure lookups, a higher span is better. But space usage is bad because most of the pointers are nulls.

ART uses a large span with different fanouts. When inserting, if the node is too small, it is replaced by a larger one. If underfull, the node is replaced with a smaller one.

They use a span of 8 bits to ease the implementation (the radix tree used in Linux uses a 6 bits span). Because changing the fanout of the node for every update operation would be counterproductive, They choose 4 different node sizes:

Node4 is the smaller inner node, it contains a maximum of 4 children nodes stored as arrays
Node16 is a larger Node4. For efficient lookup, it can use SIMD operations
Node48 uses an indirection layer to store children node
Node256 uses a single array of 2^span = 256 length

func newNode4() *Node {
    in := &innerNode{
        nodeType: Node4,
        keys:     make([]byte, Node4Max),
        children: make([]*Node, Node4Max),
        meta: meta{
            prefix: make([]byte, MaxPrefixLen),
        },
    }
    return &Node{innerNode: in}
}

func newNode16() *Node {
    in := &innerNode{
        nodeType: Node16,
        keys:     make([]byte, Node16Max),
        children: make([]*Node, Node16Max),
        meta: meta{
            prefix: make([]byte, MaxPrefixLen),
        },
    }

    return &Node{innerNode: in}
}

func newNode48() *Node {
    in := &innerNode{
        nodeType: Node48,
        keys:     make([]byte, Node256Max),
        children: make([]*Node, Node48Max),
        meta: meta{
            prefix: make([]byte, MaxPrefixLen),
        },
    }

    return &Node{innerNode: in}
}

func newNode256() *Node {
    in := &innerNode{
        nodeType: Node256,
        children: make([]*Node, Node256Max),
        meta: meta{
            prefix: make([]byte, MaxPrefixLen),
        },
    }

    return &Node{innerNode: in}
}

Path compression:

If the sequence of single-child nodes does not end in a leaf, ART uses path compression. This collapses the sequence of nodes, but into the node at the end of the sequence that has more than one child. That is, consider inserting CAST and CASH into an empty trie; path compression would create a node at the root that has the CAS prefix, and two children, one for T and one for H. That way the trie doesn’t need individual nodes for the C->A->S sequence.

Lazy expansion:

Two techniques are described that allows the trie to shrink its height if there are nodes that only have one child. They reduce the impact of having either strings with a unique suffix (lazy expansion) or a common prefix (path compression). Figure 6 in the paper illustrates both techniques clearly.

func (n *Node) shrink() {
    in := n.innerNode

    switch n.Type() {
    case Node4:
        c := in.children[0]
        if !c.IsLeaf() {
            child := c.innerNode
            currentPrefixLen := in.prefixLen

            if currentPrefixLen < MaxPrefixLen {
                in.prefix[currentPrefixLen] = in.keys[0]
                currentPrefixLen++
            }

            if currentPrefixLen < MaxPrefixLen {
                childPrefixLen := min(child.prefixLen, MaxPrefixLen-currentPrefixLen)
                copyBytes(in.prefix[currentPrefixLen:], child.prefix, childPrefixLen)
                currentPrefixLen += childPrefixLen
            }

            copyBytes(child.prefix, in.prefix, min(currentPrefixLen, MaxPrefixLen))
            child.prefixLen += in.prefixLen + 1
        }

        replaceNode(n, c)

    case Node16:
        n4 := newNode4()
        n4in := n4.innerNode
        n4in.copyMeta(n.innerNode)
        n4in.size = 0

        for i := 0; i < len(n4in.keys); i++ {
            n4in.keys[i] = in.keys[i]
            n4in.children[i] = in.children[i]
            n4in.size++
        }

        replaceNode(n, n4)

    case Node48:
        n16 := newNode16()
        n16in := n16.innerNode
        n16in.copyMeta(n.innerNode)
        n16in.size = 0

        for i := 0; i < len(in.keys); i++ {
            idx := in.keys[byte(i)]
            if idx > 0 {
                child := in.children[idx-1]
                if child != nil {
                    n16in.children[n16in.size] = child
                    n16in.keys[n16in.size] = byte(i)
                    n16in.size++
                }
            }
        }

        replaceNode(n, n16)

    case Node256:
        n48 := newNode48()
        n48in := n48.innerNode
        n48in.copyMeta(n.innerNode)
        n48in.size = 0

        for i := 0; i < len(in.children); i++ {
            child := in.children[byte(i)]
            if child != nil {
                n48in.children[n48in.size] = child
                n48in.keys[byte(i)] = byte(n48in.size + 1)
                n48in.size++
            }
        }

        replaceNode(n, n48)
    }
}

Alternatively, the node can also grow as required when the children are more than the maximum keys a node can hold.

func (n *innerNode) grow() {
    switch n.nodeType {
    case Node4:
        n16 := newNode16().innerNode
        n16.copyMeta(n)
        for i := 0; i < n.size; i++ {
            n16.keys[i] = n.keys[i]
            n16.children[i] = n.children[i]
        }
        replace(n, n16)

    case Node16:
        n48 := newNode48().innerNode
        n48.copyMeta(n)

        index := 0
        for i := 0; i < n.size; i++ {
            child := n.children[i]
            if child != nil {
                n48.keys[n.keys[i]] = byte(index + 1)
                n48.children[index] = child
                index++
            }
        }

        replace(n, n48)

    case Node48:
        n256 := newNode256().innerNode
        n256.copyMeta(n)

        for i := 0; i < len(n.keys); i++ {
            child := (n.findChild(byte(i)))
            if child != nil {
                n256.children[byte(i)] = *child
            }
        }

        replace(n, n256)

    case Node256:
    }
}

The Prefix Search Method

// Prefix Scan
tree.Insert([]byte("api"), "bar")
tree.Insert([]byte("api.com"), "bar")
tree.Insert([]byte("api.com.xyz"), "bar")
leafFilter := func(n *art.Node) {
    if n.IsLeaf() {
        fmt.Println("value=", string(n.Key()))
    }
}
tree.Scan([]byte("api"), leafFilter)

The Benchmarks
I tested loading a dictionary of 235886 words into each tree. Here are the results.

// ART tree
BenchmarkWordsArtTreeInsert      79622476 ns/op  46379858 B/op   1604123 allocs/op
BenchmarkWordsArtTreeSearch      28123512 ns/op         0 B/op         0 allocs/op

// Radix tree
BenchmarkWordsRadixInsert        96886770 ns/op  50057340 B/op   1856741 allocs/op
BenchmarkWordsRadixSearch        40109553 ns/op         0 B/op         0 allocs/op

// Skiplist
BenchmarkWordsSkiplistInsert     271771239 ns/op  32366958 B/op  1494019 allocs/op
BenchmarkWordsSkiplistSearch     135836216 ns/op         0 B/op        0 allocs/op

How did it improve flashdb? Well, twice as fast! Here are the benchmarks for the string store:

Skip list

BenchmarkFlashDBPutValue64B-16            159992          7208 ns/op        1461 B/op         19 allocs/op
BenchmarkFlashDBPutValue128B-16           175634          9499 ns/op        2003 B/op         19 allocs/op
BenchmarkFlashDBPutValue256B-16           148362          9278 ns/op        3322 B/op         19 allocs/op
BenchmarkFlashDBPutValue512B-16           120865         16542 ns/op        6037 B/op         19 allocs/op
BenchmarkFlashDBGet-16                   1881042           643.9 ns/op        32 B/op          2 allocs/op

ART Trie

BenchmarkFlashDBPutValue64B-16        204318          5129 ns/op        1385 B/op         19 allocs/op
BenchmarkFlashDBPutValue128B-16       231177          5318 ns/op        1976 B/op         16 allocs/op
BenchmarkFlashDBPutValue256B-16       189516          6202 ns/op        3263 B/op         15 allocs/op
BenchmarkFlashDBPutValue512B-16       165580          8110 ns/op        5866 B/op         16 allocs/op
BenchmarkFlashDBGet-16               4053836           294.9 ns/op        32 B/op          2 allocs/op

References
https://aly.arriqaaq.com/art-building-a-prefix-search-trie-in-go/

Writing a simple in-memory key-value Database in Go

Farhan — Thu, 10 Mar 2022 04:13:12 +0000

Ever wondered how to write a database? I wrote a simple in-memory key-value store in Go, with Redis like commands. Hope this serves as a starting point for anyone interested in building a toy db.

I call mine, FlashDB.

TLDR:

arriqaaq / flashdb

FlashDB is an embeddable, in-memory key/value database in Go (with Redis like commands and super easy to read)

flashdb

FlashDB is a simple, in-memory, key/value store in pure Go It persists to disk, is ACID compliant, and uses locking for multiple readers and a single writer. It supports redis like operations for data structures like SET, SORTED SET, HASH and STRING.

Features

In-memory database for fast reads and writes
Embeddable with a simple API
Supports Redis like operations for SET, SORTED SET, HASH and STRING
Durable append-only file format for persistence
Option to evict old items with an expiration TTL
ACID semantics with locking transactions that support rollbacks

Architecture

FlashDB is made of composable libraries that can be used independently and are easy to understand. The idea is to bridge the learning for anyone new on how to build a simple ACID database.

Getting Started

Installing

To start using FlashDB, install Go and run go get:

$ go get -u

…

View on GitHub

FlashDB is a simple, in-memory, key/value store in pure Go.
It persists to disk, is ACID compliant, and uses locking for multiple
readers and a single writer. It supports redis like operations for
data structures like SET, SORTED SET, HASH and STRING.

Features

In-memory database for fast reads and writes
Embeddable with a simple API
Supports Redis like operations for SET, SORTED SET, HASH and STRING
Durable append-only file format for persistence
Option to evict old items with an expiration TTL
ACID semantics with locking transactions that support rollbacks

Architecture

FlashDB is made of composable libraries that can be used independently and are easy to understand. The idea is to bridge the
learning for anyone new on how to build a simple ACID database.

Getting Started

Installing

To start using FlashDB, install Go and run go get:

$ go get -u github.com/arriqaaq/flashdb

This will retrieve the library.

Opening a database

The primary object in FlashDB is a DB. To open or create your
database, use the flashdb.New() function:

package main

import (
    "log"

    "github.com/arriqaaq/flashdb"
)

func main() {
    config := &flashdb.Config{Path:"/tmp", EvictionInterval: 10}
    db, err := flashdb.New(config)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    ...
}

It's also possible to open a database that does not persist to disk by keeping the path in the config empty.

config := &flashdb.Config{Path:"", EvictionInterval: 10}
flashdb.New(config)

Transactions

All reads and writes must be performed from inside a transaction. FlashDB can have one write transaction opened at a time, but can have many concurrent read transactions. Each transaction maintains a stable view of the database. In other words, once a transaction has begun, the data for that transaction cannot be changed by other transactions.

When a transaction fails, it will roll back, and revert all changes that occurred to the database during that transaction. When a read/write transaction succeeds all changes are persisted to disk.

Read-only Transactions

A read-only transaction should be used when you don't need to make changes to the data. The advantage of a read-only transaction is that there can be many running concurrently.

err := db.View(func(tx *flashdb.Tx) error {
    ...
    return nil
})

Read/write Transactions

A read/write transaction is used when you need to make changes to your data. There can only be one read/write transaction running at a time. So make sure you close it as soon as you are done with it.

err := db.Update(func(tx *flashdb.Tx) error {
    ...
    return nil
})

Setting and getting key/values

To set a value you must open a read/write transaction:

err := db.Update(func(tx *flashdb.Tx) error {
    _, _, err := tx.Set("mykey", "myvalue")
    return err
})

To get the value:

err := db.View(func(tx *flashdb.Tx) error {
    val, err := tx.Get("mykey")
    if err != nil{
        return err
    }
    fmt.Printf("value is %s\n", val)
    return nil
})

Commands

String	Hash	Set	ZSet
SET	HSET	SADD	ZADD
GET	HGET	SISMEMBER	ZSCORE
DELETE	HGETALL	SRANDMEMBER	ZCARD
EXPIRE	HDEL	SREM	ZRANK
TTL	HEXISTS	SMOVE	ZREVRANK
	HLEN	SCARD	ZRANGE
	HKEYS	SMEMBERS	ZREVRANGE
	HVALS	SUNION	ZREM
	HCLEAR	SDIFF	ZGETBYRANK
		SCLEAR	ZREVGETBYRANK
			ZSCORERANGE
			ZREVSCORERANGE
			ZCLEAR

Benchmarks

Go Version : go1.11.4 darwin/amd64
OS: Mac OS X 10.13.6
Architecture: x86_64
16 GB 2133 MHz LPDDR3
CPU: 3.1 GHz Intel Core i7

badger 2022/03/09 14:04:44 INFO: All 0 tables opened in 0s
goos: darwin
goarch: amd64
pkg: github.com/arriqaaq/flashbench
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz

BenchmarkBadgerDBPutValue64B-16             9940        141844 ns/op        2208 B/op         68 allocs/op
BenchmarkBadgerDBPutValue128B-16            7701        192942 ns/op        2337 B/op         68 allocs/op
BenchmarkBadgerDBPutValue256B-16            7368        142600 ns/op        2637 B/op         69 allocs/op
BenchmarkBadgerDBPutValue512B-16            6980        148056 ns/op        3149 B/op         69 allocs/op
BenchmarkBadgerDBGet-16                  1000000          1388 ns/op         408 B/op          9 allocs/op

BenchmarkFlashDBPutValue64B-16            159992          7208 ns/op        1461 B/op         19 allocs/op
BenchmarkFlashDBPutValue128B-16           175634          9499 ns/op        2003 B/op         19 allocs/op
BenchmarkFlashDBPutValue256B-16           148362          9278 ns/op        3322 B/op         19 allocs/op
BenchmarkFlashDBPutValue512B-16           120865         16542 ns/op        6037 B/op         19 allocs/op
BenchmarkFlashDBGet-16                   1881042           643.9 ns/op        32 B/op          2 allocs/op

PASS
ok      github.com/arriqaaq/flashbench  28.947s

With fsync enabled for every update transaction

badger 2022/03/09 14:04:44 INFO: All 0 tables opened in 0s
goos: darwin
goarch: amd64
pkg: github.com/arriqaaq/flashbench
cpu: Intel(R) Core(TM) i9-9980HK CPU @ 2.40GHz

BenchmarkNutsDBPutValue64B-16             52      20301019 ns/op        1315 B/op         17 allocs/op
BenchmarkNutsDBPutValue128B-16            63      23496536 ns/op        1059 B/op         15 allocs/op
BenchmarkNutsDBPutValue256B-16            62      20037952 ns/op        1343 B/op         15 allocs/op
BenchmarkNutsDBPutValue512B-16            62      20090731 ns/op        1754 B/op         15 allocs/op

BenchmarkFlashDBPutValue64B-16           105      16154174 ns/op         770 B/op         18 allocs/op
BenchmarkFlashDBPutValue128B-16           62      21666543 ns/op        1119 B/op         18 allocs/op
BenchmarkFlashDBPutValue256B-16           61      18631201 ns/op        1956 B/op         18 allocs/op
BenchmarkFlashDBPutValue512B-16           62      20118103 ns/op        3071 B/op         18 allocs/op

PASS
ok      github.com/arriqaaq/flashbench  28.947s

The benchmark code can be found here flashdb-bench.

Design Patterns in Kubernetes Codebase

Farhan — Tue, 23 Nov 2021 02:48:57 +0000

Published originally on my blog post here

I love reading open source code, you get to learn different design patterns in action and learn about code architecture too. I have been looking into the Kubernetes codebase, and thought of penning down my research on Golang Design Patterns that have been used in different Kubernetes codebases, in hope that someone could also benefit from it.

Design patterns are typical solutions to common problems in software design. The most universal and high-level patterns are architectural patterns. All patterns can be categorized by their intent, or purpose. We covers two main groups of patterns:

Creational patterns provide object creation mechanisms that increase flexibility and reuse of existing code.
Behavioral patterns take care of effective communication and the assignment of responsibilities between objects.

Creational Patterns

Singleton

Singleton is a creational design pattern that lets you ensure that a class has only one instance, while providing a global access point to this instance.


package singleton

import (
    "sync"
)

type singleton struct {
}

var instance *singleton
var once sync.Once

func GetInstance() *singleton {
    once.Do(func() {
        instance = &singleton{}
    })
    return instance
}

kubernetes/golang has very little use of global variables except for configuration management.

// https://github.com/kubernetes-sigs/controller-runtime/blob/master/alias.go

var (
    // GetConfigOrDie creates a *rest.Config for talking to a Kubernetes apiserver.
    // If --kubeconfig is set, will use the kubeconfig file at that location.  Otherwise will assume running
    // in cluster and use the cluster provided kubeconfig.
    //
    // Will log an error and exit if there is an error creating the rest.Config.
    GetConfigOrDie = config.GetConfigOrDie

    GetConfig = config.GetConfig

    // NewControllerManagedBy returns a new controller builder that will be started by the provided Manager.
    NewControllerManagedBy = builder.ControllerManagedBy

    // NewWebhookManagedBy returns a new webhook builder that will be started by the provided Manager.
    NewWebhookManagedBy = builder.WebhookManagedBy

    // NewManager returns a new Manager for creating Controllers.
    NewManager = manager.New


    // Log is the base logger used by controller-runtime.  It delegates
    // to another logr.Logger.  You *must* call SetLogger to
    // get any actual logging.
    Log = log.Log
)

https://github.com/kubernetes-sigs/controller-runtime/blob/master/pkg/client/config/config.go

// GetConfigOrDie creates a *rest.Config for talking to a Kubernetes apiserver.
// If --kubeconfig is set, will use the kubeconfig file at that location.  Otherwise will assume running
// in cluster and use the cluster provided kubeconfig.
//
// Will log an error and exit if there is an error creating the rest.Config.
func GetConfigOrDie() *rest.Config {
    config, err := GetConfig()
    if err != nil {
        log.Error(err, "unable to get kubeconfig")
        os.Exit(1)
    }
    return config
}

Factory Method

Factory Method is a creational design pattern that provides an interface for creating objects in a superclass, but allows subclasses to alter the type of objects that will be created.

Golang:

package simplefactory

import "fmt"

//API is interface
type API interface {
    Say(name string) string
}

//NewAPI return Api instance by type
func NewAPI(t int) API {
    if t == 1 {
        return &hiAPI{}
    } else if t == 2 {
        return &helloAPI{}
    }
    return nil
}

//hiAPI is one of API implement
type hiAPI struct{}

//Say hi to name
func (*hiAPI) Say(name string) string {
    return fmt.Sprintf("Hi, %s", name)
}

//HelloAPI is another API implement
type helloAPI struct{}

//Say hello to name
func (*helloAPI) Say(name string) string {
    return fmt.Sprintf("Hello, %s", name)
}

Kubernetes:

func NewStore(keyFunc KeyFunc) Store {
    return &cache{
        cacheStorage: NewThreadSafeStore(Indexers{}, Indices{}),
        keyFunc:      keyFunc,
    }
}

type cache struct {
    //cacheStorage bears the burden of thread safety for the cache
    cacheStorage ThreadSafeStore
    //keyFunc is used to make the key for objects stored in and retrieved from items, and
    //should be deterministic.
    keyFunc KeyFunc
}

type Store interface {
    Add(obj interface{}) error
    Update(obj interface{}) error
    Delete(obj interface{}) error
    List() []interface{}
    ListKeys() []string
    Get(obj interface{}) (item interface{}, exists bool, err error)
    GetByKey(key string) (item interface{}, exists bool, err error)

    //Replace will delete the contents of the store, using instead the
    //given list. Store takes ownership of the list, you should not reference
    //it after calling this function.
    Replace([]interface{}, string) error
    Resync() error
}

https://github.com/kubernetes/apimachinery/blob/master/pkg/runtime/serializer/codec_factory.go

func NewCodecFactory(scheme *runtime.Scheme) CodecFactory {
    serializers := newSerializersForScheme(scheme, json.DefaultMetaFactory)
    return newCodecFactory(scheme, serializers)
}

func (f CodecFactory) LegacyCodec(version ...schema.GroupVersion) runtime.Codec {
    return versioning.NewDefaultingCodecForScheme(f.scheme, f.legacySerializer, f.universal, schema.GroupVersions(version), runtime.InternalGroupVersioner)
}

func (f CodecFactory) CodecForVersions(encoder runtime.Encoder, decoder runtime.Decoder, encode runtime.GroupVersioner, decode runtime.GroupVersioner) runtime.Codec {
    // TODO: these are for backcompat, remove them in the future
    if encode == nil {
        encode = runtime.DisabledGroupVersioner
    }
    if decode == nil {
        decode = runtime.InternalGroupVersioner
    }
    return versioning.NewDefaultingCodecForScheme(f.scheme, encoder, decoder, encode, decode)
}

Abstract factory

Abstract Factory is a creational design pattern that lets you produce families of related objects without specifying their concrete classes. Abstract Factory defines an interface for creating all distinct products but leaves the actual product creation to concrete factory classes. Each factory type corresponds to a certain product variety.

Golang

Say, you need to buy a sports kit, a set of two different products: a pair of shoes and a shirt. You would want to buy a full sports kit of the same brand to match all the items.

If we try to turn this into the code, the abstract factory will help us create sets of products so that they would always match each other.

iSportsFactory.go: Abstract factory interface

package main

import "fmt"

type iSportsFactory interface {
    makeShoe() iShoe
    makeShirt() iShirt
}

func getSportsFactory(brand string) (iSportsFactory, error) {
    if brand == "adidas" {
        return &adidas{}, nil
    }

    if brand == "nike" {
        return &nike{}, nil
    }

    return nil, fmt.Errorf("Wrong brand type passed")
}

adidas.go: Concrete factory

package main

type adidas struct {
}

func (a *adidas) makeShoe() iShoe {
    return &adidasShoe{
        shoe: shoe{
            logo: "adidas",
            size: 14,
        },
    }
}

func (a *adidas) makeShirt() iShirt {
    return &adidasShirt{
        shirt: shirt{
            logo: "adidas",
            size: 14,
        },
    }
}

nike.go: Concrete factory

package main

type nike struct {
}

func (n *nike) makeShoe() iShoe {
    return &nikeShoe{
        shoe: shoe{
            logo: "nike",
            size: 14,
        },
    }
}

func (n *nike) makeShirt() iShirt {
    return &nikeShirt{
        shirt: shirt{
            logo: "nike",
            size: 14,
        },
    }
}

iShoe.go: Abstract product

package main

type iShoe interface {
    setLogo(logo string)
    setSize(size int)
    getLogo() string
    getSize() int
}

type shoe struct {
    logo string
    size int
}

func (s *shoe) setLogo(logo string) {
    s.logo = logo
}

func (s *shoe) getLogo() string {
    return s.logo
}

func (s *shoe) setSize(size int) {
    s.size = size
}

func (s *shoe) getSize() int {
    return s.size
}

adidasShoe.go: Concrete product

package main

type adidasShoe struct {
    shoe
}

nikeShoe.go: Concrete product

package main

type nikeShoe struct {
    shoe
}

iShirt.go: Abstract product

package main

type iShirt interface {
    setLogo(logo string)
    setSize(size int)
    getLogo() string
    getSize() int
}

type shirt struct {
    logo string
    size int
}

func (s *shirt) setLogo(logo string) {
    s.logo = logo
}

func (s *shirt) getLogo() string {
    return s.logo
}

func (s *shirt) setSize(size int) {
    s.size = size
}

func (s *shirt) getSize() int {
    return s.size
}

shirt.go: Concrete product

package main

type adidasShirt struct {
    shirt
}

type nikeShirt struct {
    shirt
}

main.go: Client code

package main

import "fmt"

func main() {
    adidasFactory, _ := getSportsFactory("adidas")
    nikeFactory, _ := getSportsFactory("nike")

    nikeShoe := nikeFactory.makeShoe()
    nikeShirt := nikeFactory.makeShirt()

    adidasShoe := adidasFactory.makeShoe()
    adidasShirt := adidasFactory.makeShirt()

    printShoeDetails(nikeShoe)
    printShirtDetails(nikeShirt)

    printShoeDetails(adidasShoe)
    printShirtDetails(adidasShirt)
}

func printShoeDetails(s iShoe) {
    fmt.Printf("Logo: %s", s.getLogo())
    fmt.Println()
    fmt.Printf("Size: %d", s.getSize())
    fmt.Println()
}

func printShirtDetails(s iShirt) {
    fmt.Printf("Logo: %s", s.getLogo())
    fmt.Println()
    fmt.Printf("Size: %d", s.getSize())
    fmt.Println()
}

Kubernetes

https://github.com/kubernetes/client-go/blob/v0.22.3/informers/factory.go

func NewSharedInformerFactory (client internalclientset.Interface, defaultResync time.Duration) SharedInformerFactory {
    return & sharedInformerFactory {
        client: client,
        defaultResync: defaultResync,
        informers: make (map [reflect.Type] cache.SharedIndexInformer),
        startedInformers: make (map [reflect.Type] bool),
    }
}

//SharedInformerFactory provides shared informers for resources in all known
//API group versions.
type SharedInformerFactory interface {
    internalinterfaces.SharedInformerFactory
    ForResource (resource schema.GroupVersionResource) (GenericInformer, error)
    WaitForCacheSync (stopCh <-chan struct {}) map [reflect.Type] bool

    Admissionregistration () admissionregistration.Interface
    Apps () apps.Interface
    Autoscaling () autoscaling.Interface
    Batch () batch.Interface
    Certificates () certificates.Interface
    Core () core.Interface
    Extensions () extensions.Interface
    Networking () networking.Interface
    Policy () policy.Interface
    Rbac () rbac.Interface
    Scheduling () scheduling.Interface
    Settings () settings.Interface
    Storage () storage.Interface
}

//sharedInformerFactory is specific stuct
func (f * sharedInformerFactory) Apps () apps.Interface {
    return apps.New (f)
}

Builder

Builder is a creational design pattern that lets you construct complex objects step by step. The pattern allows you to produce different types and representations of an object using the same construction code. The Builder pattern is used when the desired product is complex and requires multiple steps to complete. In this case, several construction methods would be simpler than a single monstrous constructor. The potential problem with the multistage building process is that a partially built and unstable product may be exposed to the client. The Builder pattern keeps the product private until it’s fully built.

Golang

package builder

type Builder interface {
    Part1()
    Part2()
    Part3()
}

type Director struct {
    builder Builder
}

// NewDirector ...
func NewDirector(builder Builder) *Director {
    return &Director{
        builder: builder,
    }
}

//Construct Product
func (d *Director) Construct() {
    d.builder.Part1()
    d.builder.Part2()
    d.builder.Part3()
}

type Builder1 struct {
    result string
}

func (b *Builder1) Part1() {
    b.result += "1"
}

func (b *Builder1) Part2() {
    b.result += "2"
}

func (b *Builder1) Part3() {
    b.result += "3"
}

func (b *Builder1) GetResult() string {
    return b.result
}

type Builder2 struct {
    result int
}

func (b *Builder2) Part1() {
    b.result += 1
}

func (b *Builder2) Part2() {
    b.result += 2
}

func (b *Builder2) Part3() {
    b.result += 3
}

func (b *Builder2) GetResult() int {
    return b.result
}

Kubernetes

https://github.com/kubernetes/client-go/blob/v0.22.3/kubernetes/clientset.go#L586

func NewForConfigOrDie(c *rest.Config) *Clientset {
    var cs Clientset
    cs.admissionregistrationV1alpha1 = admissionregistrationv1alpha1.NewForConfigOrDie(c)
    cs.appsV1beta1 = appsv1beta1.NewForConfigOrDie(c)
    cs.appsV1beta2 = appsv1beta2.NewForConfigOrDie(c)
    cs.appsV1 = appsv1.NewForConfigOrDie(c)
    cs.authenticationV1 = authenticationv1.NewForConfigOrDie(c)
    cs.authenticationV1beta1 = authenticationv1beta1.NewForConfigOrDie(c)
    cs.authorizationV1 = authorizationv1.NewForConfigOrDie(c)
    cs.authorizationV1beta1 = authorizationv1beta1.NewForConfigOrDie(c)
    cs.autoscalingV1 = autoscalingv1.NewForConfigOrDie(c)
    cs.autoscalingV2beta1 = autoscalingv2beta1.NewForConfigOrDie(c)
    cs.batchV1 = batchv1.NewForConfigOrDie(c)
    cs.batchV1beta1 = batchv1beta1.NewForConfigOrDie(c)
    cs.batchV2alpha1 = batchv2alpha1.NewForConfigOrDie(c)
    cs.certificatesV1beta1 = certificatesv1beta1.NewForConfigOrDie(c)
    cs.coreV1 = corev1.NewForConfigOrDie(c)
    cs.extensionsV1beta1 = extensionsv1beta1.NewForConfigOrDie(c)
    cs.networkingV1 = networkingv1.NewForConfigOrDie(c)
    cs.policyV1beta1 = policyv1beta1.NewForConfigOrDie(c)
    cs.rbacV1 = rbacv1.NewForConfigOrDie(c)
    cs.rbacV1beta1 = rbacv1beta1.NewForConfigOrDie(c)
    cs.rbacV1alpha1 = rbacv1alpha1.NewForConfigOrDie(c)
    cs.schedulingV1alpha1 = schedulingv1alpha1.NewForConfigOrDie(c)
    cs.settingsV1alpha1 = settingsv1alpha1.NewForConfigOrDie(c)
    cs.storageV1beta1 = storagev1beta1.NewForConfigOrDie(c)
    cs.storageV1 = storagev1.NewForConfigOrDie(c)

    cs.DiscoveryClient = discovery.NewDiscoveryClientForConfigOrDie(c)
    return &cs
}

https://github.com/kubernetes-sigs/controller-runtime/tree/master/pkg/builder

// Builder builds a Controller.
type Builder struct {
    forInput         ForInput
    ownsInput        []OwnsInput
    watchesInput     []WatchesInput
    mgr              manager.Manager
    globalPredicates []predicate.Predicate
    ctrl             controller.Controller
    ctrlOptions      controller.Options
    name             string
}

func (blder *Builder) For(object client.Object, opts ...ForOption) *Builder {
    if blder.forInput.object != nil {
        blder.forInput.err = fmt.Errorf("For(...) should only be called once, could not assign multiple objects for reconciliation")
        return blder
    }
    input := ForInput{object: object}
    for _, opt := range opts {
        opt.ApplyToFor(&input)
    }

    blder.forInput = input
    return blder
}

// Watches exposes the lower-level ControllerManagedBy Watches functions through the builder.  Consider using
// Owns or For instead of Watches directly.
// Specified predicates are registered only for given source.
func (blder *Builder) Watches(src source.Source, eventhandler handler.EventHandler, opts ...WatchesOption) *Builder {
    input := WatchesInput{src: src, eventhandler: eventhandler}
    for _, opt := range opts {
        opt.ApplyToWatches(&input)
    }

    blder.watchesInput = append(blder.watchesInput, input)
    return blder
}

// WithEventFilter sets the event filters, to filter which create/update/delete/generic events eventually
// trigger reconciliations.  For example, filtering on whether the resource version has changed.
// Given predicate is added for all watched objects.
// Defaults to the empty list.
func (blder *Builder) WithEventFilter(p predicate.Predicate) *Builder {
    blder.globalPredicates = append(blder.globalPredicates, p)
    return blder
}

// WithOptions overrides the controller options use in doController. Defaults to empty.
func (blder *Builder) WithOptions(options controller.Options) *Builder {
    blder.ctrlOptions = options
    return blder
}

// WithLogger overrides the controller options's logger used.
func (blder *Builder) WithLogger(log logr.Logger) *Builder {
    blder.ctrlOptions.Log = log
    return blder
}

// Build builds the Application Controller and returns the Controller it created.
func (blder *Builder) Build(r reconcile.Reconciler) (controller.Controller, error) {
    if r == nil {
        return nil, fmt.Errorf("must provide a non-nil Reconciler")
    }
    if blder.mgr == nil {
        return nil, fmt.Errorf("must provide a non-nil Manager")
    }
    if blder.forInput.err != nil {
        return nil, blder.forInput.err
    }
    // Checking the reconcile type exist or not
    if blder.forInput.object == nil {
        return nil, fmt.Errorf("must provide an object for reconciliation")
    }

    // Set the ControllerManagedBy
    if err := blder.doController(r); err != nil {
        return nil, err
    }

    // Set the Watch
    if err := blder.doWatch(); err != nil {
        return nil, err
    }

    return blder.ctrl, nil
}

Prototype

Prototype is a creational design pattern that allows cloning objects, even complex ones, without coupling to their specific classes.

Conceptual Example

Let’s try to figure out the Prototype pattern using an example based on the operating system’s file system. The OS file system is recursive: the folders contain files and folders, which may also include files and folders, and so on.

Each file and folder can be represented by an inode interface. inode interface also has the clone function.

Both file and folder structs implement the print and clone functions since they are of the inode type. Also, notice the clone function in both file and folder. The clone function in both of them returns a copy of the respective file or folder. During the cloning, we append the keyword “_clone” for the name field.

inode.go: Prototype interface

package main

type inode interface {
    print(string)
    clone() inode
}

file.go: Concrete prototype

package main

import "fmt"

type file struct {
    name string
}

func (f *file) print(indentation string) {
    fmt.Println(indentation + f.name)
}

func (f *file) clone() inode {
    return &file{name: f.name + "_clone"}
}

folder.go: Concrete prototype

package main

import "fmt"

type folder struct {
    children []inode
    name      string
}

func (f *folder) print(indentation string) {
    fmt.Println(indentation + f.name)
    for _, i := range f.children {
        i.print(indentation + indentation)
    }
}

func (f *folder) clone() inode {
    cloneFolder := &folder{name: f.name + "_clone"}
    var tempChildren []inode
    for _, i := range f.children {
        copy := i.clone()
        tempChildren = append(tempChildren, copy)
    }
    cloneFolder.children = tempChildren
    return cloneFolder
}

main.go: Client code

package main

import "fmt"

func main() {
    file1 := &file{name: "File1"}
    file2 := &file{name: "File2"}
    file3 := &file{name: "File3"}

    folder1 := &folder{
        children: []inode{file1},
        name:      "Folder1",
    }

    folder2 := &folder{
        children: []inode{folder1, file2, file3},
        name:      "Folder2",
    }
    fmt.Println("\nPrinting hierarchy for Folder2")
    folder2.print("  ")

    cloneFolder := folder2.clone()
    fmt.Println("\nPrinting hierarchy for clone Folder")
    cloneFolder.print("  ")
}

output.txt: Execution result

Printing hierarchy for Folder2
  Folder2
    Folder1
        File1
    File2
    File3

Printing hierarchy for clone Folder
  Folder2_clone
    Folder1_clone
        File1_clone
    File2_clone
    File3_clone

https://github.com/kubernetes/apiserver/blob/master/pkg/apis/apiserver/v1/zz_generated.deepcopy.go

// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
func (in *AdmissionConfiguration) DeepCopyInto(out *AdmissionConfiguration) {
    *out = *in
    out.TypeMeta = in.TypeMeta
    if in.Plugins != nil {
        in, out := &in.Plugins, &out.Plugins
        *out = make([]AdmissionPluginConfiguration, len(*in))
        for i := range *in {
            (*in)[i].DeepCopyInto(&(*out)[i])
        }
    }
    return
}

// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new AdmissionConfiguration.
func (in *AdmissionConfiguration) DeepCopy() *AdmissionConfiguration {
    if in == nil {
        return nil
    }
    out := new(AdmissionConfiguration)
    in.DeepCopyInto(out)
    return out
}

Behavioral Patterns

Observer

Observer is a behavioral design pattern that allows some objects to notify other objects about changes in their state. The Observer pattern provides a way to subscribe and unsubscribe to and from these events for any object that implements a subscriber interface.

Golang

In the e-commerce website, items go out of stock from time to time. There can be customers who are interested in a particular item that went out of stock. The customer subscribes only to the particular item he is interested in and gets notified if the item is available. Also, multiple customers can subscribe to the same product.

The major components of the observer pattern are:

Subject, the instance which publishes an event when anything happens.
Observer, which subscribes to the subject events and gets notified when they happen.

package observer

type subject interface {
    register(Observer observer)
    deregister(Observer observer)
    notifyAll()
}

type observer interface {
    update(string)
    getID() string
}

type item struct {
    observerList []observer
    name         string
    inStock      bool
}

func newItem(name string) *item {
    return &item{
        name: name,
    }
}
func (i *item) updateAvailability() {
    fmt.Printf("Item %s is now in stock\n", i.name)
    i.inStock = true
    i.notifyAll()
}
func (i *item) register(o observer) {
    i.observerList = append(i.observerList, o)
}

func (i *item) deregister(o observer) {
    i.observerList = removeFromslice(i.observerList, o)
}

func (i *item) notifyAll() {
    for _, observer := range i.observerList {
        observer.update(i.name)
    }
}

func removeFromslice(observerList []observer, observerToRemove observer) []observer {
    observerListLength := len(observerList)
    for i, observer := range observerList {
        if observerToRemove.getID() == observer.getID() {
            observerList[observerListLength-1], observerList[i] = observerList[i], observerList[observerListLength-1]
            return observerList[:observerListLength-1]
        }
    }
    return observerList
}

type customer struct {
    id string
}

func (c *customer) update(itemName string) {
    fmt.Printf("Sending email to customer %s for item %s\n", c.id, itemName)
}

func (c *customer) getID() string {
    return c.id
}

func main() {

    shirtItem := newItem("Nike Shirt")

    observerFirst := &customer{id: "abc@gmail.com"}
    observerSecond := &customer{id: "xyz@gmail.com"}

    shirtItem.register(observerFirst)
    shirtItem.register(observerSecond)

    shirtItem.updateAvailability()
}

// output

Item Nike Shirt is now in stock
Sending email to customer abc@gmail.com for item Nike Shirt
Sending email to customer xyz@gmail.com for item Nike Shirt

Kubernetes

func (s * sharedIndexInformer) AddEventHandlerWithResyncPeriod (handler ResourceEventHandler, resyncPeriod time.Duration) {
    ...
    s.processor.addListener (listener)
    ...
}


//Distribution
func (p * sharedProcessor) distribute (obj interface {}, sync bool) {
    p.listenersLock.RLock ()
    defer p.listenersLock.RUnlock ()

    if sync {
        for _, listener: = range p.syncingListeners {
            listener.add (obj)
        }
    } Else {
        for _, listener: = range p.listeners {
            listener.add (obj)
        }
    }
}

Command

Command is behavioral design pattern that converts requests or simple operations into objects. The conversion allows deferred or remote execution of commands, storing command history, etc.

Golang

package command

import "fmt"

type Command interface {
    Execute()
}

type StartCommand struct {
    mb *MotherBoard
}

func NewStartCommand(mb *MotherBoard) *StartCommand {
    return &StartCommand{
        mb: mb,
    }
}

func (c *StartCommand) Execute() {
    c.mb.Start()
}

type RebootCommand struct {
    mb *MotherBoard
}

func NewRebootCommand(mb *MotherBoard) *RebootCommand {
    return &RebootCommand{
        mb: mb,
    }
}

func (c *RebootCommand) Execute() {
    c.mb.Reboot()
}

type MotherBoard struct{}

func (*MotherBoard) Start() {
    fmt.Print("system starting\n")
}

func (*MotherBoard) Reboot() {
    fmt.Print("system rebooting\n")
}

type Box struct {
    button1 Command
    button2 Command
}

func NewBox(button1, button2 Command) *Box {
    return &Box{
        button1: button1,
        button2: button2,
    }
}

func (b *Box) PressButton1() {
    b.button1.Execute()
}

func (b *Box) PressButton2() {
    b.button2.Execute()
}

Kubernetes

https://github.com/kubernetes/kubernetes/blob/master/cmd/kubectl/kubectl.go

kubernetes use of command is based on the cobra/cli package (github.com/spf13/cobra).

func main() {
    command := cmd.NewDefaultKubectlCommand()
    code := cli.Run(command)
    os.Exit(code)
}

Iterator

Iterator is a behavioral design pattern that allows sequential traversal through a complex data structure without exposing its internal details.

type collection interface {
    createIterator() iterator
}

type userCollection struct {
    users []*user
}

func (u *userCollection) createIterator() iterator {
    return &userIterator{
        users: u.users,
    }
}

type iterator interface {
    hasNext() bool
    getNext() *user
}

type userIterator struct {
    index int
    users []*user
}

func (u *userIterator) hasNext() bool {
    if u.index < len(u.users) {
        return true
    }
    return false

}
func (u *userIterator) getNext() *user {
    if u.hasNext() {
        user := u.users[u.index]
        u.index++
        return user
    }
    return nil
}

type user struct {
    name string
    age  int
}

func main() {

    user1 := &user{
        name: "a",
        age:  30,
    }
    user2 := &user{
        name: "b",
        age:  20,
    }

    userCollection := &userCollection{
        users: []*user{user1, user2},
    }

    iterator := userCollection.createIterator()

    for iterator.hasNext() {
        user := iterator.getNext()
        fmt.Printf("User is %+v\n", user)
    }
}

Kubernetes

https://github.com/kubernetes/apimachinery/blob/v0.21.0/pkg/runtime/serializer/json/json.go

// CaseSensitiveJSONIterator returns a jsoniterator API that's configured to be
// case-sensitive when unmarshalling, and otherwise compatible with
// the encoding/json standard library.
func CaseSensitiveJSONIterator() jsoniter.API {
    config := jsoniter.Config{
        EscapeHTML:             true,
        SortMapKeys:            true,
        ValidateJsonRawMessage: true,
        CaseSensitive:          true,
    }.Froze()
    // Force jsoniter to decode number to interface{} via int64/float64, if possible.
    config.RegisterExtension(&customNumberExtension{})
    return config
}

// StrictCaseSensitiveJSONIterator returns a jsoniterator API that's configured to be
// case-sensitive, but also disallows unknown fields when unmarshalling. It is compatible with
// the encoding/json standard library.
func StrictCaseSensitiveJSONIterator() jsoniter.API {
    config := jsoniter.Config{
        EscapeHTML:             true,
        SortMapKeys:            true,
        ValidateJsonRawMessage: true,
        CaseSensitive:          true,
        DisallowUnknownFields:  true,
    }.Froze()
    // Force jsoniter to decode number to interface{} via int64/float64, if possible.
    config.RegisterExtension(&customNumberExtension{})
    return config
}

Strategy

Strategy is a behavioral design pattern that turns a set of behaviors into objects and makes them interchangeable inside original context object. The original object, called context, holds a reference to a strategy object and delegates it executing the behavior. In order to change the way the context performs its work, other objects may replace the currently linked strategy object with another one.

Golang

type Payment struct {
    context  *PaymentContext
    strategy PaymentStrategy
}

type PaymentContext struct {
    Name, CardID string
    Money        int
}

func NewPayment(name, cardid string, money int, strategy PaymentStrategy) *Payment {
    return &Payment{
        context: &PaymentContext{
            Name:   name,
            CardID: cardid,
            Money:  money,
        },
        strategy: strategy,
    }
}

func (p *Payment) Pay() {
    p.strategy.Pay(p.context)
}

type PaymentStrategy interface {
    Pay(*PaymentContext)
}

type Cash struct{}

func (*Cash) Pay(ctx *PaymentContext) {
    fmt.Printf("Pay $%d to %s by cash", ctx.Money, ctx.Name)
}

type Bank struct{}

func (*Bank) Pay(ctx *PaymentContext) {
    fmt.Printf("Pay $%d to %s by bank account %s", ctx.Money, ctx.Name, ctx.CardID)

}

Kubernetes

https://github.com/kubernetes/kubernetes/blob/master/pkg/registry/core/configmap/strategy.go

//strategy implements behavior for ConfigMap objects
type strategy struct {
    runtime.ObjectTyper
    names.NameGenerator
}

//Strategy is the default logic that applies when creating and updating ConfigMap
//objects via the REST API.
var Strategy = strategy {api.Scheme, names.SimpleNameGenerator}

//Strategy should implement rest.RESTCreateStrategy
var _ rest.RESTCreateStrategy = Strategy

//Strategy should implement rest.RESTUpdateStrategy
var _ rest.RESTUpdateStrategy = Strategy

func (strategy) NamespaceScoped () bool {
    return true
}

func (strategy) PrepareForCreate (ctx genericapirequest.Context, obj runtime.Object) {
    _ = Obj. (* Api.ConfigMap)
}

func (strategy) Validate (ctx genericapirequest.Context, obj runtime.Object) field.ErrorList {
    cfg: = obj (* api.ConfigMap).

    return validation.ValidateConfigMap (cfg)
}

//Canonicalize normalizes the object after validation.
func (strategy) Canonicalize (obj runtime.Object) {
}

func (strategy) AllowCreateOnUpdate () bool {
    return false
}

func (strategy) PrepareForUpdate (ctx genericapirequest.Context, newObj, oldObj runtime.Object) {
    _ = OldObj. (* Api.ConfigMap)
    _ = NewObj. (* Api.ConfigMap)
}

func (strategy) AllowUnconditionalUpdate () bool {
    return true
}

func (strategy) ValidateUpdate (ctx genericapirequest.Context, newObj, oldObj runtime.Object) field.ErrorList {
    oldCfg, newCfg: = oldObj (* api.ConfigMap), newObj (* api.ConfigMap)..

    return validation.ValidateConfigMapUpdate (newCfg, oldCfg)
}



//k8s.io/kubernetes/pkg/registry/core/configmap/storage/storage.go
//NewREST returns a RESTStorage object that will work with ConfigMap objects.
func NewREST (optsGetter generic.RESTOptionsGetter) * REST {
    store: = & genericregistry.Store {
        NewFunc: func () runtime.Object {return & api.ConfigMap {}},
        NewListFunc: func () runtime.Object {return & api.ConfigMapList {}},
        DefaultQualifiedResource: api.Resource ( "configmaps"),

        CreateStrategy: configmap.Strategy,
        UpdateStrategy: configmap.Strategy,
        DeleteStrategy: configmap.Strategy,
    }
    options: = & generic.StoreOptions {RESTOptions: optsGetter}
    if err: = store.CompleteWithOptions (options); err = nil {!
        panic (err)//TODO: Propagate error up
    }
    return & REST {store}
}

Chain Of Responsibility

Chain of Responsibility is behavioral design pattern that allows passing request along the chain of potential handlers until one of them handles request. The pattern allows multiple objects to handle the request without coupling sender class to the concrete classes of the receivers. The chain can be composed dynamically at runtime with any handler that follows a standard handler interface.

Golang

Let’s look at the Chain of Responsibility pattern with the case of a hospital app. A hospital could have multiple departments such as:

Reception
Doctor
Medicine room
Cashier

Whenever any patient arrives, they first get to Reception, then to Doctor, then to Medicine Room, and then to Cashier (and so on). The patient is being sent through a chain of departments, where each department sends the patient further down the chain once their function is completed.

// department
type department interface {
    execute(*patient)
    setNext(department)
}

// reception
type reception struct {
    next department
}

func (r *reception) execute(p *patient) {
    if p.registrationDone {
        fmt.Println("Patient registration already done")
        r.next.execute(p)
        return
    }
    fmt.Println("Reception registering patient")
    p.registrationDone = true
    r.next.execute(p)
}

func (r *reception) setNext(next department) {
    r.next = next
}

// doctor
type doctor struct {
    next department
}

func (d *doctor) execute(p *patient) {
    if p.doctorCheckUpDone {
        fmt.Println("Doctor checkup already done")
        d.next.execute(p)
        return
    }
    fmt.Println("Doctor checking patient")
    p.doctorCheckUpDone = true
    d.next.execute(p)
}

func (d *doctor) setNext(next department) {
    d.next = next
}

// medical
type medical struct {
    next department
}

func (m *medical) execute(p *patient) {
    if p.medicineDone {
        fmt.Println("Medicine already given to patient")
        m.next.execute(p)
        return
    }
    fmt.Println("Medical giving medicine to patient")
    p.medicineDone = true
    m.next.execute(p)
}

func (m *medical) setNext(next department) {
    m.next = next
}

// cashier

type cashier struct {
    next department
}

func (c *cashier) execute(p *patient) {
    if p.paymentDone {
        fmt.Println("Payment Done")
    }
    fmt.Println("Cashier getting money from patient patient")
}

func (c *cashier) setNext(next department) {
    c.next = next
}

// patient
type patient struct {
    name              string
    registrationDone  bool
    doctorCheckUpDone bool
    medicineDone      bool
    paymentDone       bool
}



func main() {

    cashier := &cashier{}

    //Set next for medical department
    medical := &medical{}
    medical.setNext(cashier)

    //Set next for doctor department
    doctor := &doctor{}
    doctor.setNext(medical)

    //Set next for reception department
    reception := &reception{}
    reception.setNext(doctor)

    patient := &patient{name: "abc"}
    //Patient visiting
    reception.execute(patient)
}

// Output
Reception registering patient
Doctor checking patient
Medical giving medicine to patient
Cashier getting money from patient patient

Visitor

Visitor is a behavioral design pattern that allows adding new behaviors to existing class hierarchy without altering any existing code. The Visitor pattern lets you add behavior to a struct without actually modifying the struct. Let’s say you are the maintainer of a lib which has different shape structs such as:

Square
Circle
Triangle

Each of the above shape structs implements the common shape interface.

Once people in your company started to use your awesome lib, you got flooded with feature requests. Let’s review one of the simplest ones: a team requested you to add the getArea behavior to the shape structs.

There are many options to solve this problem.

The first option that comes to the mind is to add the getArea method directly into the shape interface and then implement it in each shape struct. This seems like a go-to solution, but it comes at a cost. As the maintainer of the library, you don’t want to risk breaking your precious code each time someone asks for another behavior. Still, you do want other teams to extend your library somehow.

The second option is that the team requesting the feature can implement the behavior themselves. However, this is not always possible, as this behavior may depend on the private code.

The third option is to solve the above problem using the Visitor pattern. We start by defining a visitor interface like this:

type visitor interface {
   visitForSquare(square)
   visitForCircle(circle)
   visitForTriangle(triangle)
}

The functions visitForSquare(square), visitForCircle(circle), visitForTriangle(triangle) will let us add functionality to squares, circles and triangles respectively.

type shape interface {
    getType() string
    accept(visitor)
}

type square struct {
    side int
}

func (s *square) accept(v visitor) {
    v.visitForSquare(s)
}

func (s *square) getType() string {
    return "Square"
}

type circle struct {
    radius int
}

func (c *circle) accept(v visitor) {
    v.visitForCircle(c)
}

func (c *circle) getType() string {
    return "Circle"
}

type rectangle struct {
    l int
    b int
}

func (t *rectangle) accept(v visitor) {
    v.visitForrectangle(t)
}

func (t *rectangle) getType() string {
    return "rectangle"
}

type visitor interface {
    visitForSquare(*square)
    visitForCircle(*circle)
    visitForrectangle(*rectangle)
}

type areaCalculator struct {
    area int
}

func (a *areaCalculator) visitForSquare(s *square) {
    // Calculate area for square.
    // Then assign in to the area instance variable.
    fmt.Println("Calculating area for square")
}

func (a *areaCalculator) visitForCircle(s *circle) {
    fmt.Println("Calculating area for circle")
}
func (a *areaCalculator) visitForrectangle(s *rectangle) {
    fmt.Println("Calculating area for rectangle")
}

type middleCoordinates struct {
    x int
    y int
}

func (a *middleCoordinates) visitForSquare(s *square) {
    // Calculate middle point coordinates for square.
    // Then assign in to the x and y instance variable.
    fmt.Println("Calculating middle point coordinates for square")
}

func (a *middleCoordinates) visitForCircle(c *circle) {
    fmt.Println("Calculating middle point coordinates for circle")
}
func (a *middleCoordinates) visitForrectangle(t *rectangle) {
    fmt.Println("Calculating middle point coordinates for rectangle")
}

func main() {
    square := &square{side: 2}
    circle := &circle{radius: 3}
    rectangle := &rectangle{l: 2, b: 3}

    areaCalculator := &areaCalculator{}

    square.accept(areaCalculator)
    circle.accept(areaCalculator)
    rectangle.accept(areaCalculator)

    fmt.Println()
    middleCoordinates := &middleCoordinates{}
    square.accept(middleCoordinates)
    circle.accept(middleCoordinates)
    rectangle.accept(middleCoordinates)
}

Kubernetes

https://github.com/kubernetes/kube-openapi/blob/master/pkg/util/proto/openapi.go

type SchemaVisitor interface {
    VisitArray(*Array)
    VisitMap(*Map)
    VisitPrimitive(*Primitive)
    VisitKind(*Kind)
    VisitReference(Reference)
}

//Schema is the base definition of an openapi type.
type Schema interface {
    //Giving a visitor here will let you visit the actual type.
    Accept(SchemaVisitor)

    //Pretty print the name of the type.
    GetName() string
    //Describes how to access this field.
    GetPath() *Path
    //Describes the field.
    GetDescription() string
    //Returns type extensions.
    GetExtensions() map[string]interface{}
}

///k8s.io/kubernetes/pkg/kubectl/resource/builder.go
func (b *Builder) visitByName() *Result {
    ...

    visitors := []Visitor{}
    for _, name := range b.names {
        info := NewInfo(client, mapping, selectorNamespace, name, b.export)
        visitors = append(visitors, info)
    }
    result.visitor = VisitorList(visitors)
    result.sources = visitors
    return result
}

Resources

Kubediff: A Kubernetes Resource Diff Tool

Farhan — Mon, 04 Oct 2021 13:27:53 +0000

kubediff is a Kubernetes resource diff watcher, with the ability to send event notifications.

Usage

$ kubediff --config=/path/to/config

kubediff: A resource diff watcher for Kubernetes

kubediff is a Kubernetes resource diff watcher with the ability to configure event notifications
to webhook/Slack. It watches the cluster for any resource change (including custom CRDs) and logs them. You can also run it in normal mode, and can export the logs to your preferred logging stack.

Usage:
  kubediff --config=/path/to/config

Flags:
  --config   configuration folder for kubediff

Options

You can create a watch on a single namespace, or multiple namespaces. If you want update on all the events, you can simply run kubediff in watch mode and it will update you for all the events when an object is Created/Deleted/Updated.


# Mode (watch/diff). Watch mode logs all events (Create/Update/Delete). Diff mode logs only object diff on update,
mode: watch 

# Name of the resource. Resource name must be in group/version/resource (G/V/R) format
# resource name should be plural (e.g apps/v1/deployments, v1/pods)
resources:
  - kind: v1/pods
  - kind: v1/services
  - kind: apps/v1/deployments
  - kind: apps/v1/statefulsets
  - kind: networking.k8s.io/v1beta1/ingresses
  - kind: v1/nodes
  - kind: v1/namespaces
  - kind: v1/persistentvolumes
  - kind: v1/persistentvolumeclaims
  - kind: v1/configmaps
  - kind: apps/v1/daemonsets
  - kind: batch/v1/jobs
  - kind: rbac.authorization.k8s.io/v1/roles
  - kind: rbac.authorization.k8s.io/v1/rolebindings
  - kind: rbac.authorization.k8s.io/v1/clusterrolebindings
  - kind: rbac.authorization.k8s.io/v1/clusterroles

# add multiple namespaces which you'd like to watch. Use `all` to watch all namespaces.
namespaces:
  - kube-system
  - default

Install

Using helm:

When you have helm installed in your cluster, use the following setup:

helm repo add arriqaaq https://arriqaaq.github.io/charts
helm repo update
helm install --create-namespace --namespace kubediff kubediff arriqaaq/kubediff

You can also install this chart locally by cloning this repo:

helm install --create-namespace --namespace kubediff kubediff helm/kubediff

Using kubectl:

In order to run kubediff in a kind cluster quickly, just run

$ kubectl apply -f hack/deploy.yaml

Configuration:

You can also provide a custom config file:

resources:
  - kind: v1/pods             # Name of the resource. Resource name must be in group/version/resource (G/V/R) format
                              # resource name should be plural (e.g apps/v1/deployments, v1/pods)
  - kind: v1/services
  - kind: apps/v1/deployments
  - kind: apps/v1/statefulsets
  - kind: networking.k8s.io/v1beta1/ingresses
  - kind: v1/nodes
  - kind: v1/namespaces
  - kind: v1/persistentvolumes
  - kind: v1/persistentvolumeclaims
  - kind: v1/configmaps
  - kind: apps/v1/daemonsets
  - kind: batch/v1/jobs
  - kind: rbac.authorization.k8s.io/v1/roles
  - kind: rbac.authorization.k8s.io/v1/rolebindings
  - kind: rbac.authorization.k8s.io/v1/clusterrolebindings
  - kind: rbac.authorization.k8s.io/v1/clusterroles

namespaces:
  - all

Using Go:

# Download and install kubediff
$ go get -u github.com/arriqaaq/kubediff

# Add resources to be watched
kubediff --config=./test/

Resources

Read more on how it is implemented here.

https://github.com/arriqaaq/kubediff

Deep dive into Linux Networking and Docker - Bridge, vETH and IPTables

Farhan — Sat, 11 Sep 2021 12:00:40 +0000

This was originally published here: https://aly.arriqaaq.com/linux-networking-bridge-iptables-and-docker/

These series of articles are my log of learning about various networking concepts related to Container Orchestration Platforms (Docker, Kubernetes, etc)

Linux Networking is a very interesting topic. In this series, my aim is to dig deep to understand the various ways in which these container orchestration platforms implement network internals underneath.

Getting Started

A few questions before getting started.

1) What are namespaces?

TLDR, a linux namespace is an abstraction over resources in the operating system. Namespaces are like separate houses with their own sets of isolated resources. There are currently 7 types of namespaces Cgroup, IPC, Network, Mount, PID, User, UTS

Network isolation is what we are interested in, so we will be discussing in depth about network namespaces.

2) How to follow along?

All the examples in this article have been made on a fresh vagrant Ubuntu Bionic virtual machine.

Getting Started

A few questions before getting started.

1) What are namespaces?

Network isolation is what we are interested in, so we will be discussing in depth about network namespaces.

2) How to follow along?

All the examples in this article have been made on a fresh vagrant Ubuntu Bionic virtual machine.

Getting Started

A few questions before getting started.

1) What are namespaces?

Network isolation is what we are interested in, so we will be discussing in depth about network namespaces.

2) How to follow along?

All the examples in this article have been made on a fresh vagrant Ubuntu Bionic virtual machine.

vagrant init ubuntu/bionic64
vagrant up
vagrant ssh

Exploring Network Namespaces

How do platforms virtualise network resources to isolate containers by assigning them a dedicated network stack, and making sure these containers do not interfere with the host (or neighbouring containers)? Network Namespace. A network namespace isolates network related resources — a process running in a distinct network namespace has its own networking devices, routing tables, firewall rules etc.Let’s create one quickly.

ip netns add ns1

And voila! You have your isolated network namespace (ns1) created just like that. Now you can go ahead and run any process inside this namespace.

ip netns exec ns1 python3 -m http.server 8000

This was pretty neat! The exec $namespace $command executes $command in the named network namespace $namespace. This means that the process runs within its own network stack, separate from the host, and can communicate only through the interfaces defined in the network namespace.

Host NamespaceBefore you read ahead, I’d like to draw your attention on the default namespace for the **host **network. Let’s list down all the namespaces

ip netns

# all namespaces
ns1
default

You can notice the default **namespace that is created. This is the **host namespace, which implies whatever services that you run simply on your VM or your machine, is run under this namespace. This would be important to note moving forward.

Creating a Network Namespace

So with that said, let’s quickly move forward and create two isolated network namespaces (similar to two containers)

#!/usr/bin/env bash

NS1="ns1"
NS2="ns2"

# create namespace
ip netns add $NS1
ip netns add $NS2

Connecting the cables

We need to go ahead and connect these namespaces to our host network. The vETH (virtual Ethernet) device helps in making this connection. vETH is a local Ethernet tunnel, and devices are created in pairs.Packets transmitted on one device in the pair are immediately received on the other device. When either device is down, the link state of the pair is down.

#!/usr/bin/env bash

NS1="ns1"
VETH1="veth1"
VPEER1="vpeer1"

NS2="ns2"
VETH2="veth2"
VPEER2="vpeer2"

# create namespace
ip netns add $NS1
ip netns add $NS2

# create veth link
ip link add ${VETH1} type veth peer name ${VPEER1}
ip link add ${VETH2} type veth peer name ${VPEER2}

Think of VETH like a network cable. One end is attached to the host network, and the other end to the network namespace created. Let’s go ahead and connect the cable, and bring these interfaces up.

# setup veth link
ip link set ${VETH1} up
ip link set ${VETH2} up

# add peers to ns
ip link set ${VPEER1} netns ${NS1}
ip link set ${VPEER2} netns ${NS2}

Localhost

Ever wondered how localhost works? Well, the loopback interface directs the traffic to remain within the local system. So when you run something on localhost (127.0.0.1), you are essentially using the loopback interface to route the traffic through. Let’s bring the loopback interface up in case we’d want to run a service locally, and also bring up the peer interfaces inside our network namespace to start accepting traffic.

# setup loopback interface
ip netns exec ${NS1} ip link set lo up
ip netns exec ${NS2} ip link set lo up

# setup peer ns interface
ip netns exec ${NS1} ip link set ${VPEER1} up
ip netns exec ${NS2} ip link set ${VPEER2} up

In order to connect to the network, a computer must have at least one network interface. Each network interface must have its own unique IP address. The IP address that you give to a host is assigned to its network interface.But does every network interface require an IP address right? Well, not really. We’ll see that in the coming steps.

# assign ip address to ns interfaces
VPEER_ADDR1="10.10.0.10"
VPEER_ADDR2="10.10.0.20"

ip netns exec ${NS1} ip addr add ${VPEER_ADDR1}/16 dev ${VPEER1}
ip netns exec ${NS2} ip addr add ${VPEER_ADDR2}/16 dev ${VPEER2}

Remember, here we’ve only assigned network addresses to the interfaces inside the network namespaces (ns1 (vpeer1), ns2 (vpeer2)). The host namespaces interfaces do not have an IP assigned (veth1, veth2). Why? Do we need it? Well, not really.

Build Bridges, Not Walls

Men build too many walls and not enough bridges

Remember that when you have multiple containers running, and want to send traffic to these containers, we’d require a bridge to connect them. A network bridge creates a single, aggregate network from multiple communication networks or network segments. A bridge is a way to connect two Ethernet segments together in a protocol independent way. **Packets are forwarded based on Ethernet address*, rather than IP address (like a router). *Since forwarding is done at Layer 2, all protocols can go transparently through a bridge. Bridging is distinct from routing. Routing allows multiple networks to communicate independently and yet remain separate, whereas bridging connects two separate networks as if they were a single network.

Docker has a **docker0 **bridge underneath to direct traffic. When Docker service starts, a Linux bridge is created on the host machine. The various interfaces on the containers talk to the bridge, and the bridge proxies to the external world. Multiple containers on the same host can talk to each other through the Linux bridge.

So let’s go ahead and create a bridge.

BR_ADDR="10.10.0.1"
BR_DEV="br0"

# setup bridge
ip link add ${BR_DEV} type bridge
ip link set ${BR_DEV} up

# assign veth pairs to bridge
ip link set ${VETH1} master ${BR_DEV}
ip link set ${VETH2} master ${BR_DEV}

# setup bridge ip
ip addr add ${BR_ADDR}/16 dev ${BR_DEV}

Now that we have out network interfaces connected to the bridge, how do these interfaces know how to direct the traffic to the host? The route tables in both network namespaces only have route entries for their respective subnet IP range.

Since we have the VETH pairs connected to the bridge, the bridge network address is available to these network namespaces. Let’s add a default route to direct the traffic to the bridge.

# add default routes for ns
ip netns exec ${NS1} ip route add default via ${BR_ADDR}
ip netns exec ${NS2} ip route add default via ${BR_ADDR}

Done. Sweet! We have a proper setup to test if our containers can talk to each other. Let’s finally interact with the namespaces.

# add default routes for ns
ip netns exec ${NS1} ping ${VPEER_ADDR2}

PING 10.10.0.20 (10.10.0.20) 56(84) bytes of data.
64 bytes from 10.10.0.20: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from 10.10.0.20: icmp_seq=2 ttl=64 time=0.039 ms

ip netns exec ${NS2} ping ${VPEER_ADDR1}
PING 10.10.0.10 (10.10.0.10) 56(84) bytes of data.
64 bytes from 10.10.0.10: icmp_seq=1 ttl=64 time=0.045 ms
64 bytes from 10.10.0.10: icmp_seq=2 ttl=64 time=0.039 ms

MASQUERADE

We are able to send traffic between the namespaces, but we haven’t tested sending traffic outside the container. And for that, we’d need to use IPTables to masquerade the outgoing traffic from our namespace.

# enable ip forwarding
bash -c 'echo 1 > /proc/sys/net/ipv4/ip_forward'

iptables -t nat -A POSTROUTING -s ${BR_ADDR}/16 ! -o ${BR_DEV} -j MASQUERADE

MASQUERADE modifies the source address of the packet, replacing it with the address of a specified network interface. This is similar to SNAT, except that it does not require the machine’s IP address to be known in advance.Basically, what we are doing here is that we are adding an entry to NAT table, to masquerade the outgoing traffic from the bridge, except for the bridge traffic itself. With this, we are done with a basic setup on how docker actually implements linux network stack to isolate containers. You can find the entire script here.

Now let’s dive deep into how docker works with various networking setups.

How does Docker work?

Each Docker container has its own network stack, where a new network namespace is created for each container, isolated from other containers. When a Docker container launches, the Docker engine assigns it a network interface with an IP address, a default gateway, and other components, such as a routing table and DNS services.

Docker offers five network types. All these network types are configured through docker0 via the --net flag

1. Host Networking (--net=host*)*: The container shares the same network namespace of the default host.

You can verify this easily.

# check the network interfaces on the host
ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:98:b0:9b:6c:78 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
       valid_lft 85994sec preferred_lft 85994sec
    inet6 fe80::98:b0ff:fe9b:6c78/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:e5:72:10:c0 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

Run docker in host mode, and you will see it lists out the same set of interfaces.

# check the network interfaces in the container
docker run --net=host -it --rm alpine ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: enp0s3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 02:98:b0:9b:6c:78 brd ff:ff:ff:ff:ff:ff
    inet 10.0.2.15/24 brd 10.0.2.255 scope global dynamic enp0s3
       valid_lft 85994sec preferred_lft 85994sec
    inet6 fe80::98:b0ff:fe9b:6c78/64 scope link
       valid_lft forever preferred_lft forever
3: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:e5:72:10:c0 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

2. Bridge Networking ( — net=bridge/default*):* In this mode, the default bridge is used as the bridge for containers to connect to each other.The container runs in an isolated network namespace. Communication is open to other containers in the same network. Communication with services outside of the host goes through network address translation (NAT) before exiting the host. We’ve already seen above, the creation of a bridge network.

# check the network interfaces in the container
docker run --net=bridge -it --rm alpine ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
16: eth0@if17: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

You can notice that there is an eth0 veth pair that has been created for this container and the corresponding pair should exist on the host machine

# check the network interfaces on the host
ip addr

21: veth8a812a3@if20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master docker0 state UP group default
    link/ether d2:c4:4e:d4:08:ad brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::d0c4:4eff:fed4:8ad/64 scope link
       valid_lft forever preferred_lft forever

3. Custom bridge network ( — network=xxx*):* This is the same as Bridge Networking but uses a custom bridge explicitly created for containers.

# create custom bridge
docker network create foo
2b25342b1d883dd134ed8a36e3371ef9c3ec77cdb9e24a0365165232e31b17b6

# check the bridge interface on the host
22: br-2b25342b1d88: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default
    link/ether 02:42:49:79:07:30 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-2b25342b1d88
       valid_lft forever preferred_lft forever
    inet6 fe80::42:49ff:fe79:730/64 scope link
       valid_lft forever preferred_lft forever

You can see that on the custom creation of a bridge, a bridge interface is added to the host. Now, all containers in a custom bridge can communicate with the ports of other containers on that bridge. This provides better isolation and security.

Now let’s run two containers in different terminals

# terminal 1
docker run -it --rm --name=container1 --network=foo alpine sh

# terminal 2
docker run -it --rm --name=container2 --network=foo alpine sh

# check the network interfaces on the host
ip addr

22: br-2b25342b1d88: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:49:79:07:30 brd ff:ff:ff:ff:ff:ff
    inet 172.18.0.1/16 brd 172.18.255.255 scope global br-2b25342b1d88
       valid_lft forever preferred_lft forever
    inet6 fe80::42:49ff:fe79:730/64 scope link
       valid_lft forever preferred_lft forever
30: veth86ca323@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-2b25342b1d88 state UP group default
    link/ether 1e:5e:66:ea:47:1e brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet6 fe80::1c5e:66ff:feea:471e/64 scope link
       valid_lft forever preferred_lft forever
32: vethdf5e755@if31: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br-2b25342b1d88 state UP group default
    link/ether ba:2b:25:23:a3:40 brd ff:ff:ff:ff:ff:ff link-netnsid 1
    inet6 fe80::b82b:25ff:fe23:a340/64 scope link
       valid_lft forever preferred_lft forever

As expected, with bridge networking, both containers (container1, container2) have got their respective veth (veth86ca323, vethdf5e755) cable attached. You can verify this bridge simply by running:

# you can notice both the containers are connected via the same bridge
brctl show br-2b25342b1d88

bridge name bridge id       STP enabled interfaces
br-2b25342b1d88     8000.024249790730   no  veth86ca323
                            vethdf5e755

4. Container-defined Networking( — net=container:$container2*):* With this enabled, the container created shares its network namespace with the container called $container2.

# create a container in terminal 1
docker run -it --rm --name=container1  alpine sh

# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
33: eth0@if34: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

# create a container in terminal 1
docker run -it --rm --name=container2 --network=container:container1 alpine ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
33: eth0@if34: <BROADCAST,MULTICAST,UP,LOWER_UP,M-DOWN> mtu 1500 qdisc noqueue state UP
    link/ether 02:42:ac:11:00:02 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.2/16 brd 172.17.255.255 scope global eth0
       valid_lft forever preferred_lft forever

You can see that both the container share the same network interface.

5. No networking: This option disables all networking for the container

docker run --net=none alpine ip addr

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever

You can notice that only the lo (loopback) interface is enabled, nothing else is configured in this container.

In the next article, we will dive deeper (inshaAllah) into how docker manipulates iptables rules to provide network isolation.

yq - read YAML on terminal

Farhan — Fri, 06 Aug 2021 20:52:28 +0000

yq

yq lets you read YAML files easily on the terminal. You can find key/values easily.

arriqaaq / yq

yq lets you read YAML files easily on the terminal. You can find key/values easily

yq

yq lets you read YAML files easily on the terminal. You can find key/values easily.

Motivation

Reading yaml configurations for k8s file becomes ardent through the terminal. yq helps reading/searching through the YAML easy. It uses tview and is inspired by tson.

Installation

Grab the latest binary

$ cd "$(mktemp -d)"
$ curl -sL "https://github.com/arriqaaq/yq/releases/download/v0.1.0/yq_0.1.0_$(uname)_amd64.tar.gz" | tar xz
$ mv yq /usr/local/bin
# yq should be available now in your $PATH

Usage

NAME:
   yq 

USAGE:
# from file
$ yq < test.json

# from kubectl
$ kubectl get pod kube-dns -n kube-system -oyaml | yq

Bindings

YAMLtree

key	description
s	hide/show current node
S	collaspe/expand all value nodes
/ or f	search for key/values
q	quit

View on GitHub

Motivation

Reading yaml configurations for k8s file becomes ardent through the terminal. yq helps reading/searching through the YAML easy. It uses tview and is inspired by tson.

Installation

Grab the latest binary

$ cd "$(mktemp -d)"
$ curl -sL "https://github.com/arriqaaq/yq/releases/download/v0.1.0/yq_0.1.0_$(uname)_amd64.tar.gz" | tar xz
$ mv yq /usr/local/bin
# yq should be available now in your $PATH

Usage

NAME:
   yq 

USAGE:
# from file
$ yq < test.json

# from kubectl
$ kubectl get pod kube-dns -n kube-system -oyaml | yq

Bindings

YAMLtree

key	description
s	hide/show current node
S	collaspe/expand all value nodes
/ or f	search for key/values
q	quit

Writing An Operating System - The Boot Process (Part 1)

Farhan — Sat, 29 May 2021 13:04:46 +0000

This post was originally published here

My journey on learning to build a simple OS

How many times have you read an OS book but not been able to code one?Operating System (OS) books are tedious, but only theory makes it hard to understand how an OS actually works. Here is my attempt to write a simple OS and document some of the concepts learned.
‌
‌

Before You Start

On a mac, install Homebrew and then brew install qemu nasm
On some systems qemu is split into multiple binaries. You may want to call qemu-system-x86_64 binfile

QEmu

For testing these low-level programs without continuously having to reboot a machine or risk scrubbing your important data off a disk, we will use a CPU emulator QEmu.
I'm working on a Mac (with M1 chip). QEmu has some issues with M1 chip, so you can run these experiements inside a docker container. docker run -it ubuntu bash.
Run QEmu with -nographic and -curses arguments inside docker container to display the VGA output when in text mode

NASM

NASM is an assembler and disassembler for the Intel x86 architecture. It can be used to write 16-bit, 32-bit (IA-32) and 64-bit (x86-64) programs.
‌
‌
When we start our computer, initially, it has no notion of an operating system. Somehow, it must load the operating system --- whatever variant that may be --- from some permanent storage device that is currently attached to the computer (e.g. a floppy disk, a hard disk, a USB dongle, etc.).

The Boot Process

Booting an operating system consists of transferring control along a chain of small programs, each one more “powerful” than the previous one, where the operating system is the last “program”.

‌

BIOS

When the PC is turned on, the computer will start a small program that adheres to the Basic Input Output System (BIOS) [16] standard. This program is usually stored on a read only memory chip on the motherboard of the PC. BIOS is a collection of software routines that are initially loaded from a chip into memory and initialised when the computer is switched on. BIOS provides auto-detection and basic control of your computer’s essential devices, such as the screen, keyboard, and hard disks.

Note: Modern operating systems do not use the BIOS’ functions, they use drivers that interact directly with the hardware, bypassing the BIOS. Today, BIOS mainly runs some early diagnostics (power-on-self-test) and then transfers control to the bootloader.
‌

‌Boot Sector‌

‌
BIOS cannot simply load a file that represents your operating system from a disk, since BIOS has no notion of a file- system. BIOS must read specific sectors of data (usually 512 bytes in size) from specific physical locations of the disk devices, such as Cylinder 2, Head 3, Sector 5.
So, the easiest place for BIOS to find our OS is in the first sector of one of the disks (i.e. Cylinder 0, Head 0, Sector 0), known as the boot sector. To make sure that the "disk is bootable", the BIOS checks that bytes 511 and 512 of the alleged boot sector are bytes 0xAA55. If so, the BIOS loads the first sector to the address 7C00h, set the program counter to that address and let the CPU executing code from there. This is the simplest boot sector ever:

e9 fd ff 00 00 00 00 00 00 00 00 00 00 00 00 00
00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[ 29 more lines with sixteen zero-bytes each ]
00 00 00 00 00 00 00 00 00 00 00 00 00 00 55 aa

‌
Note that, in the above boot sector, the three important features are:

1) The initial three bytes, in hexadecimal as 0xe9, 0xfd and 0xff, are actually machine code instructions, as defined by the CPU manufacturer, to perform an endless jump.
2) The last two bytes, 0x55 and 0xaa, make up the magic number, which tells BIOS that this is indeed a boot block and not just data that happens to be on a drive’s boot sector. (in little-endian format)
3) The file is padded with zeros (’*’ indicates zeros omitted for brevity), basically to position the magic BIOS number at the end of the 512 byte disk sector.

The first sector is called Master Boot Record, or MBR. The program in the first sector is called MBR Bootloader.

So, BIOS loops through each storage device (e.g. floppy drive, hard disk, CD drive, etc.), reads the boot sector into memory, and instructs the CPU to begin executing the first boot sector it finds that ends with the magic number. This is where we seize control of the computer.
‌

‌The Bootloader

The BIOS program will transfer control of the PC to a program called a bootloader. A bootloader loads an OS, or an application that runs and communicate directly with hardware. To run an OS, the first thing to write is a bootloader. Here is a simple bootloader.

;
; A simple boot sector program that loops forever. ;
9

; Define a label, "loop", that will allow ; us to jump back to it, forever.
; Use a simple CPU instruction that jumps
; to a new memory address to continue execution. ; In our case, jump to the address of the current ; instruction.



loop:
    jmp loop


; When compiled, our program must fit into 512 bytes,
; with the last two bytes being the magic number,
; so here, tell our assembly compiler to pad out our
; program with enough zero bytes (db 0) to bring us to the ; 510th byte.


times 510-($-$$) db 0


; Last two bytes (one word) form the magic number, ; so BIOS knows we are a boot sector.

dw 0xaa55

‌
We compile the code with nasm and write it to a bin file:

nasm -f bin boot_sect_simple.asm -o boot_sect_simple.bin

Let's try it out, so let's do it:

qemu boot_sect_simple.bin

On some systems, you may have to run

You will see a window open which says "Booting from Hard Disk..." and nothing else. There you go, a simple boot loader is ready!

Continue reading more here

Sorry, copy pasting it from ghost was tough!

Latency need not always require scaling your Microservice (a story of an in memory cache)

Farhan — Wed, 04 Dec 2019 20:35:10 +0000

The single most important thing when building an Adtech platform which does OpenRTB bidding is sub milli-second response times. To serve billions of ad requests, a huge infrastructure is required. But most of the times, it is not really required to scale out your service with the incoming traffic,

When building robust Microservices and deploying them on Kubernetes or any other platform, one would have surely encountered latency issues between different services. As the traffic increases, the pods (or servers) need to be scaled up accordingly for a particular service.

But it is not always necessary that you need to linearly scale one service based on the traffic it receives from the other. Say there is a service whose job is to respond with the eCPM value, or a targeting service which does runs complex lua scripts in redis to figure out the targeted partners for a particular device. This data wouldn’t necessary change per-second. A user is not going to change states in a few mins neither is the eCPM value going to change every second. Many a times, data required from the other service doesn’t change continuously, and if this data can be cached briefly, a lot of traffic can be served faster without making a network call.

Let’s take two services for this example, Service A and Service B. Service B figures out some data from redis and provides it to Service A on request. So ideally when the traffic increases, both the services need to be scaled linearly to handle the load.

Service A

Service B

Let’s benchmark Service A and Service B and check their response times. Both services are being monitored using prometheus.

A simple ab test gives us the following result.

Increasing the concurrency to 40

Service A has higher latency than Service B (response times in ms)

As you can see, from the charts, as the RPS increases on Service A, the time taken for the client to obtain response from Service B increases too. But Service B prometheus stats do not show much of a difference, and the response time is pretty consistent. What is the reason for this behavior?

THE TCP GAMEPLAY

A lot happens under the hood for a TCP request, when two services interact.

Lot of things come into play when two services interact over TCP, the number of sockets involved(Socket layer), the network IO speed(Protocol/Interface layer), number of cores of the machine (apart from application code). Bottlenecks could happen at any of these layers. The sockets may run out because of a limit on the system, not enough threads available on a server to serve (hence shooting up the load average leading to delay in response). Hence Service B metrics only the record response times on the application layer, whereas Service A metrics (for Service B) takes into account the whole tcp cycle, which includes the connection time, tcp wait time etc.

How do we solve this problem in general? A few obvious approaches would be to either add more servers or upscale the hardware. But is it always required? Is the data being served from the other service needed in real time that every single time a request has to be made? These are few questions every application developer must consider. It is not necessary that the configurations or values that we fetch from the other service may update frequently (or every second). That is where things could be optimized.

KEEP EVERYTHING IN MEMORY

High frequency trading systems had always intrigued me as they are limited by the same problems. But they seem to overcome it and provide responses in real time. Here are a few points to note when building a HFT platform:

Optimize code for performance
Keep everything in memory
Keep hardware unutilized
Keep reads sequential, utilize cache locality
Do async work as much as possible

Keeping everything in memory is a good viable option for us since in most of our cases, the RAM remained underutilized compared to the cores. Remember, Service A requests data every single time from Service B, but Service B’s data doesn’t change frequently. When updates/writes are not very frequent, one could cache the response from one service locally for a few minutes to avoid requesting the other service for the same data again and again. Similar to how L2/L3 cache works on a processor to avoid requesting from main memory/disk all the time, the same idea can be applied at a service level to increase the traffic for better response times.

ZIZOU, an alternate answer

Keeping this in mind, I made a simple library called **Zizou** which serves as an in memory (sharded) cache, which can store keys locally in memory, with eviction policy enabled. This just acts as another layer of cache locally on the system, similar to what one would store on redis to avoid lookup’s from different services for the same stale data. And since the data is served from memory, it helps your service respond back in pretty quickly and frees up your resources for other important tasks.

Using it is pretty simple. You have simple Get/Set functions which keep a map of the key value pairs in memory with the expiry specified.

A background goroutine runs periodically which evicts deleted keys from memory to free up the heap.

While this may seem a trivial caching mechanism which many would have implemented, this simple hack has helped me save a lot of money in infrastructure because this avoids unnecessary scaling of services with increase in traffic, thereby reducing traffic on other Microservices in our cluster, and therefore not scaling the services unnecessary which would otherwise scale based on the custom resource RPS metrics in Kubernetes.

Here’s the code.

arriqaaq / zizou

In memory sharded cache implementation

zizou

In memory cache implementation with high concurrency

Features

Store millions of entries
High concurrent thread-safe access
Expiration support
Shard support to avoid locks on whole db during any concurrent read/writes/deletes

Example Usage

config:=zizou.Config{
    SweepTime:10*time.Minute
    ShardSize: 256,
}
cache,err:=zizou.New(&config)
cache.Set("foo", "bar", 10*time.Second)
item, found := cache.Get("foo")

Performance

     BenchmarkCacheGetExpiring-4                 20000000            70.0 ns/op
        BenchmarkCacheGetNotExpiring-4              20000000           119 ns/op
        BenchmarkCacheGetConcurrentExpiring-4       20000000            59.3 ns/op
        BenchmarkCacheGetConcurrentNotExpiring-4    20000000            91.3 ns/op
        BenchmarkCacheSetExpiring-4                 10000000           172 ns/op
        BenchmarkCacheSetNotExpiring-4              10000000           134 ns/op
        BenchmarkCacheSetDelete-4                    5000000           343 ns/op

TODO

Store data as a map of string and byte array to avoid lookups on object, which will be copied in memory
Make a circular shard ring, cache objects in individual shard for better concurrency…

View on GitHub

Live, laugh, and always Cache.

Resources:

For more, please follow TechLog

Layering a docker image

Farhan — Tue, 03 Dec 2019 01:47:28 +0000

Introduction

I’ve been a fan of immutable infrastructure ever since I used tools like Terraform and Packer. I wouldn’t say I understood the pros of it right away, but with time spent managing deployment of services in high traffic environments, one sort of understands the importance of being able to have an infrastructure which is scalable, predictable and can be recovered automatically. Who better to explain what immutable infrastructure is than Armon Dadgar himself?

Image layering

If you go through Section 2.3 of the book container best practices, they have a good section on various types of images. This brought back few of the steps I used to use previously on maintaining and building container images.

Poor quick drawing credits: sketchpad.io

Base Layer

This base layer contains the operating system, core system tools, (eg: such as vim, bash, software-properties-common, curl, vim etc) and tools necessary to install packages and make updates to the image over time. The best example of such base layers are the alpine images.

Dependency layer

Dependency image layers are where you would install the platform related dependencies (eg: such as Java runtime, mysql, Go, Erlang etc)

Application layer

Application layer is where the dependencies specific to the container would go (eg: Database configurations, Go application code etc)

One can add the application specific endpoints here or add something like dumb-init.

What this helps us do is to reuse the layers at one specific stage and to use it for multiple purposes. Images being layered in this fashion will help break down large sized images into smaller images with well defined layers. It will help in faster builds, maintenance and also for reusing them as components.

References

Please follow TechLog for more.

What’s inside etcd? A deep dive into the Kubernetes world

Farhan — Tue, 03 Dec 2019 01:37:26 +0000

Liquid error: internal

Having been away from working on K8s for a while, I decided to cut open K8s to see how it works to try falling in love with it again. This is going to be a series of blogs, with the first surgery being trying to figure out what’s stored inside the etcd cluster.

This is a story of finding the etcd pod inside the kubernetes cluster without diving deep into how etcd works.

Etcd is kubernetes vault. It stores the entire state of the cluster: its configuration, specifications, and the statuses of the running workloads. In this blog,

What is etcd?

Etcd is defined as a distributed, reliable key-value store for the most critical data of a distributed system. Etcd is written in Go.

How does etcd work inside kubernetes?

Our friends at Heptio have a great blog where they have shown the components involved during a simple Pod creation process. It’s a great illustration of the API Server and etcd interaction.

Installing etcd

Download Binary

    curl -L [https://github.com/etcd-io/etcd/releases/download/v3.3.11/etcdv3.3.11-linux-amd64.tar.gz](https://github.com/etcd-io/etcd/releases/download/v3.3.11/etcdv3.3.11-linux-amd64.tar.gz) -o etcd-v3.3.11-linux-amd64.tar.gz

Extract


    tar xzvf etcd-v3.3.11-linux-amd64.tar.gz

Operate ETCD

Run server


    ./etcd

Run client (on a different terminal)


    // Set a key
    ./etcdctl set key1 value1

    // Get a key
    ./etcdctl get key1

One can also use etcd labs to play around with etcd deployment. It is similar to docker playground.

The Kubernetes Test Cluster

I use minikube along with a virtualboxdriver for starting a docker container, so the next task would be to setup up a one node K8s cluster on my local machine to interact with the etcd store.

Start the cluster

    minikube start — vm-driver=virtualbox

Finding the etcd pod


    $ kubectl get pods -n kube-system

    NAME                               READY     STATUS    RESTARTS   AGE
    coredns-5644d7b6d9-2qg54           1/1       Running   0          2d
    coredns-5644d7b6d9-6mbqk           1/1       Running   0          2d
    etcd-minikube                      1/1       Running   0          2d
    kube-addon-manager-minikube        1/1       Running   0          2d
    kube-apiserver-minikube            1/1       Running   0          2d
    kube-controller-manager-minikube   1/1       Running   0          20s
    kube-proxy-kgt6n                   1/1       Running   0          2d
    kube-scheduler-minikube            1/1       Running   0          2d
    storage-provisioner                1/1       Running   0          2d

Connecting to the etcd pod


    kubectl exec -it etcd-minikube -n kube-system sh

After connecting to the shell, I tried running the etcdctl command to find all the keys.


    etcdctl get /

Turns out, it wasn’t as simple as this. Etcd uses an ADVERTISE_URL over which it exposes the process. I had to figure out all the flags which etcd uses at startup. I used the following command to get all the necessary flags.


    $ echo “$(ps aux)”

    USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
    root         1  2.9  2.3 10611260 46916 ?      Ssl  10:24   7:04

    etcd --advertise-client-urls=[https://192.168.99.100:2379](https://192.168.99.100:2379) --cert-file=/var/lib/minikube/certs/etcd/server.crt --client-cert-auth=true --data-dir=/var/lib/minikube/etcd --initial-advertise-peer-urls=[https://192.168.99.100:2380](https://192.168.99.100:2380) --initial-cluster=minikube=[https://192.168.99.100:2380](https://192.168.99.100:2380) --key-file=/var/lib/minikube/certs/etcd/server.key --listen-client-urls=[https://127.0.0.1:2379,https://192.168.99.100:2379](https://127.0.0.1:2379,https://192.168.99.100:2379) --listen-metrics-urls=[http://127.0.0.1:2381](http://127.0.0.1:2381) --listen-peer-urls=[https://192.168.99.100:2380](https://192.168.99.100:2380) --name=minikube --peer-cert-file=/var/lib/minikube/certs/etcd/peer.crt --peer-client-cert-auth=true --peer-key-file=/var/lib/minikube/certs/etcd/peer.key --peer-trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt --snapshot-count=10000 --trusted-ca-file=/var/lib/minikube/certs/etcd/ca.crt

To get etcdctl to talk to the etcd server, it needs the above flags since K8s sets up etcd along with authentication inside the pod. We first need to figure out the ADVERTISE_URL, which can be found using the --advertise-client-urls flag.

Getting all the keys


    $ ADVERTISE_URL=”[https://192.168.99.100:2379](https://192.168.99.100:2379)"

    $ ETCDCTL_API=3 etcdctl --endpoints $ADVERTISE_URL \
    --cacert /var/lib/minikube/certs/etcd/ca.crt \
    --cert /var/lib/minikube/certs/etcd/server.crt \ 
    --key /var/lib/minikube/certs/etcd/server.key \ 
    get / — prefix — keys-only

Funny output

    /registry/apiregistration.k8s.io/apiservices/v1.
    /registry/apiregistration.k8s.io/apiservices/v1.admissionregistration.k8s.io
    /registry/apiregistration.k8s.io/apiservices/v1.apiextensions.k8s.io
    /registry/apiregistration.k8s.io/apiservices/v1.apps
    /registry/apiregistration.k8s.io/apiservices/v1.authentication.k8s.io
    /registry/apiregistration.k8s.io/apiservices/v1.authorization.k8s.io
    /registry/apiregistration.k8s.io/apiservices/v1.autoscaling
    /registry/apiregistration.k8s.io/apiservices/v1.batch
    /registry/apiregistration.k8s.io/apiservices/v1.coordination.k8s.io
    /registry/apiregistration.k8s.io/apiservices/v1.networking.k8s.io
    /registry/apiregistration.k8s.io/apiservices/v1.rbac.authorization.k8s.io

Let’s make sense of this by storing the output it in a json file.


    $ ETCDCTL_API=3 etcdctl --endpoints $ADVERTISE_URL \
    --cacert /var/lib/minikube/certs/etcd/ca.crt \
    --cert /var/lib/minikube/certs/etcd/server.crt \ 
    --key /var/lib/minikube/certs/etcd/server.key \ 
    get / — prefix — keys-only -w json > out.json

It outputs all the KV pairs in a json format where they keys are encoded.


    {
     "header": {
      "cluster_id": 12197035334886545600,
      "member_id": 9217530203749069991,
      "revision": 19300,
      "raft_term": 2
     },
     "kvs": [{
      "key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEu",
      "create_revision": 12,
      "mod_revision": 12,
      "version": 1
     }, {
      "key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEuYWRtaXNzaW9ucmVnaXN0cmF0aW9uLms4cy5pbw==",
      "create_revision": 9,
      "mod_revision": 9,
      "version": 1
     }, {
      "key": "L3JlZ2lzdHJ5L2FwaXJlZ2lzdHJhdGlvbi5rOHMuaW8vYXBpc2VydmljZXMvdjEuYXBpZXh0ZW5zaW9ucy5rOHMuaW8=",
      "create_revision": 10,
      "mod_revision": 10,
      "version": 1
     }]
    }

Output for one key


    $ ETCDCTL_API=3 etcdctl — endpoints $ADVERTISE_URL — cacert /var/lib/minikube/certs/etcd/ca.crt — cert /var/lib/minikube/certs/etcd/server.crt — key /var/lib/minikube/certs/etcd/server.key get /registry/apiregistration.k8s.io/apiservices/v1.

    {
     "kind": "APIService",
     "apiVersion": "apiregistration.k8s.io/v1beta1",
     "metadata": {
      "name": "v1.",
      "uid": "fd15144e-ef8c-4a02–87fc-0fc72c178118",
      "creationTimestamp": "2019–11–22T15:19:43Z",
      "labels": {
       "kube-aggregator.kubernetes.io/automanaged": "onstart"
      }
     },
     "spec": {
      "service": null,
      "version": "v1",
      "groupPriorityMinimum": 18000,
      "versionPriority": 1
     },
     "status": {
      "conditions": [{
       "type": "Available",
       "status": "True",
       "lastTransitionTime": "2019–11–22T15:19:43Z",
       "reason": "Local",
       "message": "Local APIServices are always available"
      }]
     }
    }

The above keys are the configuration and status of all the resources within the cluster like:

Nodes
Namespaces
ClusterRoles
ClusterRoleBindings
ConfigMaps
Secrets
Workloads: Deployments, DaemonSets, Pods

Summary

This is my personal log on how to connect to the etcd pod inside a kubernetes cluster for debugging purpose in the future. It is interesting to see what information a K8s cluster stores in etcd when running. Helps in learning how to design complex systems in the future.

Please follow TechLog for more.