DEV Community: Caio Ferreira

Reflections about Supervised Learning on Security

Caio Ferreira — Sun, 23 Apr 2023 00:00:00 +0000

Supervised learning is a technique that aims to learn a hypothesis function $h$ that fits a behavior observed in the real-world, which is governed by an unknown function $f$ .

To learn this function, we use a set of example data points composed of inputs (also called features) and outcomes (sometimes called labels). These example data points were sampled from the real world behavior, i.e. from the function $f$ , at some time in the past. Our goal is that we can extract knowledge from the past to figure out this behavior on unseen data beforehand. This knowledge extracted from the set of examples is materialized on the $h$ function.

In Security, we can imagine some examples where this would be useful, like trying to learn if an HTTP request contains malicious payload or if some set of bytes is a malware or not. However, supervised learning is less often used in Security than in many other domains.

This happens because the principles of the supervised learning theory conflicts with the nature of Security, limiting its application. But, by understanding these principles, it is also possible to see how to best apply this technique and how it can maybe useful.

Stationary assumption

The most important principle in supervised learning is the stationary assumption. When the data that represents the real world behaviors follows the stationary assumption, it means that predicting the behavior using past example is approximately correctly.

The stationary assumption states that the behavior that is being learned don’t change through time. This has some important consequences:

We expect that each data point is independent of each other. This is important, because if there were causal effects between data points, then the features of a data point $x_1$ , caused by $x_0$ , would vary with a probability that is a combination of the probability distribution and the effect of $x_0$ , hence the distribution would not remain the same over time, because $x_0$ and $x_1$ would vary in different ways. For example, in a box with 1 blue ball and 2 red balls, the probability of picking a blue ball when drawing the first one from the box is 33% while a red one would be 66%. However, if the first one is indeed blue, then the probability of drawing a red ball as the second one is 100%. The first data point (blue ball draw) changed the probability of the second data point.
We expect that each data point is identically distributed, i.e. each data point should be drawn from the same probability distribution. We could learn the shopping behavior using data ranging from Black Friday to New Years, however the users’ behavior in this time is completed different from the rest of the year, therefore the data used to learn has a different probability distribution than the unseen data on which we are going to make predictions.

Any dataset that follows these two characteristics is said to hold the i.i.d assumption (independent and identically distributed). The importance for our training datasets on supervised learning to be i.i.d is because it connects the past to the future, without it, any inference made on the available data would be invalid.

Understand the i.i.d assumption is specially important for Security Machine Learning because it is one of the areas where causality and behavior shifts are most present. So, exactly how this assumption affects our ability to do Supervised Machine Learning?

Causality

Many threat behaviors have causal nature, and therefore we should have a lot of care when preparing our datasets and choosing our validation methods.

A good example is malware classification, where you could have many samples from various families from different years. Each family generation influences each other, and sometimes they have similar characteristics. A special bad situation that could happen with this is that during splitting of the dataset between training and testing, without taking into account the time relation of the families and samples, then you could end up training the model with future information and testing against past samples. This would produce falsely accurate results that would not generalize in the real-world.

There are ways to deal with this, but it depends on the type and strength of the causal relation between the data. For this case, ensuring that the newest malware samples are used for test should be enough.

Behavior Shift

We could create a model to learn a threat behavior like the profile of a botnet, however once we started responding effectively to it, adversaries would adapt and our model would become useless because the new botnets would have a totally different behavior. This would be the case of a change in the probability distribution from which the features are drawn, leading to the break of our assumption.

Looking to the other side

Although causality may be addressed by good data preparation, preventing a model to be become outdated due to behavioral shift is almost impossible. However, this problem isn’t a new one in Security, such that one best practices is to instead of trying to detect and block malicious action, defenders should define what a legitimate system behavior looks like and block everything else. This can be summarized as: allow lists are more secure than block lists.

We can apply the same philosophy for supervised learning, by modeling profiles of legitimate behavior, which usually are more stable. Then, new data points are classified against these multiple profiles models, finally a meta-classifier is used to choose which one is the best fit or if it’s an outlier. This has the ability to catch any new threat behavior that deviates from the know legitimate profiles.

This combination of multiple models is called ensemble learning, which has shown to improve models performance, like when comparing a Decision Tree model to a Random Forest one.

As a side bonus, since this way of building models does not depend on knowing threat behaviors, it avoids the common problem in mining data for Security that usually produce highly unbalanced datasets. We can train each profile using the true positive data of others profiles as false examples, assuming each profile is mutually exclusive.

The main challenge in this approach is that adversaries may try to mimic legitimate profiles, however like the authors of Notos showed, this can be useful sometimes, as in their cases this would imply in adversaries using a more stable network infrastructure that would be easily defeated by static block lists. Therefore, forcing adversaries to mimic a legitimate profile could also reduce their capabilities.

Conclusion

In conclusion, supervised learning is a powerful technique that allows us to extract knowledge from past data points to predict future behavior. However, in the Security domain, applying it comes with unique challenges due to the presence of causality and behavior shifts. Understanding the stationary assumption and the importance of having an i.i.d dataset is crucial, because it informs us how to best prepare our datasets, choose our validation methods and, most importantly, what are the best behaviors to be modeled using supervised learning.

Luckily, by modeling profiles of legitimate behavior, and catching new threat behavior that deviates from known legitimate profiles, we can build intelligent allow lists.

While there are challenges, with proper preparation and understanding, supervised learning can be a valuable tool in Security.

References

Notos: Building a Dynamic Reputation System for DNS

Implementing a safe and sound API Key authorization middleware in Go

Caio Ferreira — Sun, 06 Feb 2022 02:40:00 +0000

A common requirement that I face on multiple projects is to safeguard some API endpoints to administrative access, or to provide a secure way for other applications to consume our service in a controlled and traceable manner.

The usual solution for it is API Keys, a simple and effective authorization control mechanism that we can implement with a few lines of code. However, when doing, so we also need to be aware of threats and possible attacks that we may suffer, specially due to the usual privileges that these keys provides.

Therefore, we are going to analyze common points of concern and design a solution that improve our security posture while keeping it simple.

API Keys threats

There are two main concerns when implementing an API Key authorization scheme: key provisioning and timing attacks. Let’s review each threat before designing solutions to address them.

Key Provisioning

The key storage is directly related to how applications expect these secrets to be provided to them. Environment variables are the most common solution used on modern services since they are widely supported and don’t incur a high reading cost (in contrast to files) allowing for dynamic changes to be easily detected.

However, the way developers usually define the environment variables are through scripts or configuration files, for example using a Kubernetes Secret manifest. This introduces a serious threat of API Keys being committed to git repositories, which in the event of data leakage from the internal VCS management system would expose these credentials.

Note: remember that once committed, even if the keys are deleted from the source files, the information is already on the repository history and is easily searchable with tools like TruffleHog.

Therefore, please do not commit your API Keys to git!

Timing Attacks

Once your application is configured with the available API Keys, you need to verify that the end-user provided key (let’s call this the user key) is correct. Doing so with a naive algorithm, like using == operator, will make the verification end on the first incorrect character, hence reducing the time taken to respond.

A timing attack takes advantage of this scenario by trying to guess the correct characters of a secret based on how long the application took to respond. If the guess is right, the response will take slightly longer than if it’s wrong.

Naturally, since equality checks are orders of magnitude faster than the network roundtrip, this type of attack is extremely difficult to perform because it depends on a statistical analysis of many response samples. By looking at the time distribution produced by two different characters, one can infer that if they are different, inferring that the greater one is the correct value. For an extensive discussion of statistical techniques to help perform this attack see Morgan, Morgan 2015.

Middleware design and implementation

Having these threats in mind, we can design a suitable solution. Let’s start with the most simple API Key middleware implementation possible and iterate from it.

func ApiKeyMiddleware(cfg conf.Config, logger logging.Logger) func(handler http.Handler) http.Handler {
    apiKeyHeader := cfg.APIKeyHeader // string
    apiKeys := cfg.APIKeys // map[string]string

    reverseKeyIndex := make(map[string]string)
    for name, key := apiKeys {
        reverseKeyIndex[key] = name
    }

    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            apiKey, err := bearerToken(r, apiKeyHeader)
            if err != nil {
                logger.Errorw("request failed API key authentication", "error", err)
                RespondError(w, http.StatusUnauthorized, "invalid API key")
                return
            }

            _, found := reverseKeyIndex[apiKey]
            if !found {
                hostIP, _, err := net.SplitHostPort(r.RemoteAddr)
                if err != nil {
                    logger.Errorw("failed to parse remote address", "error", err)
                    hostIP = r.RemoteAddr
                }
                logger.Errorw("no matching API key found", "remoteIP", hostIP)

                RespondError(w, http.StatusUnauthorized, "invalid api key")
                return
            }

            next.ServeHTTP(w, r)
        })
    }
}

// bearerToken extracts the content from the header, striping the Bearer prefix
func bearerToken(r *http.Request, header string) (string, error) {
    rawToken := r.Header.Get(header)
    pieces := strings.SplitN(rawToken, " ", 2)

    if len(pieces) < 2 {
        return "", errors.New("token with incorrect bearer format")
    }

    token := strings.TrimSpace(pieces[1])

    return token, nil
}

A middleware is a function that takes an http.Handler and returns an http.Handler. In this code, the function ApiKeyMiddleware is a factory that creates an instance of the middleware with the provided configuration and logger. The config.Config is a struct populated from environment variables and logging.Logger is an interface that can be implemented using any logging library or the standard library. You could pass only the header and map of keys, but for clarity we choose to denote the dependency from this middleware to the configuration.

After extracting the fields that it relies on, the function creates a reverse index of the API Keys, which is originally a map from a key id/name to the key value. Using this reverse index it’s trivial to verify if the user key is valid by doing a map lookup on line 18.

However, this approach expects the API Keys as plaintext values and is susceptible to timing attacks, because its validation algorithm is not constant time.

Using key hashes for validation

To improve the key provisioning workflow, we can use a simple yet effective solution: expect the available keys to be hashes. Using this approach we can now commit our key hashes to our repository because even in the event of a data leak they could not be reversed to their original value.

Let’s use the SHA256 hashing algorithm to encode our keys. For example, if one of them is 123456789 (please, do not use a key like this :D) then its hash will be:

15e2b0d3c33891ebb0f1ef609ec419420c20e320ce94c65fbc8c3312448eb225

Now you can add this hash to your deployment script, Kubernetes Secret, etc., and commit it with peace of mind.

Next, we need to handle this new format on our middleware. This is what the code will look like now:

func ApiKeyMiddleware(cfg conf.Config, logger logging.Logger) func(handler http.Handler) http.Handler {
    apiKeyHeader := cfg.APIKeyHeader // string
    apiKeys := cfg.APIKeys // map[string]string

    reverseKeyIndex := make(map[string]string)
    for name, key := apiKeys {
        reverseKeyIndex[key] = name
    }

    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            apiKey, err := bearerToken(r, apiKeyHeader)
            if err != nil {
                logger.Errorw("request failed API key authentication", "error", err)
                RespondError(w, http.StatusUnauthorized, "invalid API key")
                return
            }

            _, ok := apiKeyIsValid(apiKey, reverseKeyIndex)
            if !ok {
                hostIP, _, err := net.SplitHostPort(r.RemoteAddr)
                if err != nil {
                    logger.Errorw("failed to parse remote address", "error", err)
                    hostIP = r.RemoteAddr
                }
                logger.Errorw("no matching API key found", "remoteIP", hostIP)

                RespondError(w, http.StatusUnauthorized, "invalid api key")
                return
            }

            next.ServeHTTP(w, r)
        })
    }
}

// apiKeyIsValid checks if the given API key is valid and returns the principal if it is.
func apiKeyIsValid(rawKey string, availableKeys map[string][]byte) (string, bool) {
    hash := sha256.Sum256([]byte(rawKey))
    key := string(hash[:])

    name, found := reverseKeyIndex[apiKey]

    return name, found
}

// bearerToken function omitted..

Here we extracted the logic to validate the key into a function that, before checking the equality of the user key against the available ones, encodes the user key using the same SHA256 algorithm.

This simple step improved a lot our security posture without adding much complexity. Now we can have the benefits of version control, like change history and easy detection when someone changes a key hash.

This approach works well when there are few keys to be managed, and you want to follow a GitOps approach. However, if you need to scale the key management, allow for self-service key requests and automatic rotation, you may want to look for a solution like Hashicorp Vault. Even using an external secret store I still believe this strategy, to rely on key hashes to be valid, because your external secret store can persist both the original key and the hash, and the access policy for the application can have fewer privileges in such a way that it can only read the hashes.

Constant time key verification

Once we have a better strategy to provision our keys, we need to defend ourselves against them being exfiltrated by timing attacks. The solution for this kind of vulnerability is to use an algorithm that takes the same time to produce a result whether the keys are equal or not. This is called a constant time comparison, and the Go Standard Library offers us an implementation in the crypto/subtle package that is perfect to solve most of our problems. Hence, we can update our code to use this package:

func ApiKeyMiddleware(cfg conf.Config, logger logging.Logger) (func(handler http.Handler) http.Handler, error) {
    apiKeyHeader := cfg.APIKeyHeader
    apiKeys := cfg.APIKeys
    apiKeyMaxLen := cfg.APIKeyMaxLen

    decodedAPIKeys := make(map[string][]byte)
    for name, value := range apiKeys {
        decodedKey, err := hex.DecodeString(value)
        if err != nil {
            return nil, err
        }

        decodedAPIKeys[name] = decodedKey
    }

    return func(next http.Handler) http.Handler {
        return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
            ctx := r.Context()

            apiKey, err := bearerToken(r, apiKeyHeader)
            if err != nil {
                logger.Errorw("request failed API key authentication", "error", err)
                RespondError(w, http.StatusUnauthorized, "invalid API key")
                return
            }

            if _, ok := apiKeyIsValid(apiKey, decodedAPIKeys); !ok {
                 hostIP, _, err := net.SplitHostPort(r.RemoteAddr)
                    if err != nil {
                        logger.Errorw("failed to parse remote address", "error", err)
                        hostIP = r.RemoteAddr
                    }
                    logger.Errorw("no matching API key found", "remoteIP", hostIP)

                    RespondError(w, http.StatusUnauthorized, "invalid api key")
                    return
            }

            next.ServeHTTP(w, r.WithContext(ctx))
        })
    }, nil
}

// apiKeyIsValid checks if the given API key is valid and returns the principal if it is.
func apiKeyIsValid(rawKey string, availableKeys map[string][]byte) (string, bool) {
    hash := sha256.Sum256([]byte(rawKey))
    key := hash[:]

    for name, value := range availableKeys {
        contentEqual := subtle.ConstantTimeCompare(value, key) == 1

        if contentEqual {
            return name, true
        }
    }

    return "", false
}

// bearerToken function omitted...

Now, the function apiKeyIsValid uses subtle.ConstantTimeCompare to verify the user key against each available key. Since subtle.ConstantTimeCompare operates upon byte slices we don’t cast our hash to string anymore and also our reversed index has gone in place of a decoded map.

The decoding is necessary because the string representation of our key hashes are actually a hexadecimal encoding of the binary value. Hence, we cannot just cast the string to byte slice because Go assumes all strings to be UTF-8 encoded.

Note: for an example on how using a cast instead of the correct decoding function, the result of []byte("09") is 110000111001 while hex.DecodeString("09") produces 1001. Check out the live example here.

The major disadvantage of this solution is that now we need to iterate over all available keys before finding out if the key is incorrect. This doesn’t scale well if there are too many keys, however one simple workaround would be to require the client to send an extra header with the key ID/name, e.g. X-App-Key-ID, with which you can find the key in O(1) and then apply the constant time comparison.

However, there is one subtle (pun intended) behavior from subtle.ConstantTimeCompare that we must be aware before deploying our solution to production. When the byte slices have different lengths, the functions returns earlier without performing the bitwise operations. This is natural because it does an XOR between each pair of bits from each slice, and with slices of different sizes, there would be bits from one slice without a matching pair to be combined with. Because of it, an adversary could measure that keys with the wrong length have a smaller response time than keys with the correct length, hence leaking the key length. It would only be a vulnerability if you use a short key that is easily brute-forced, but with a simple 30 character key using the UTF-8 printable characters you would have 30^95 = 2.12089515 × 10^140 possible keys.

Finally, we’ve built a simple, secure and efficient API Key solution that should handle a lot of uses cases without additional infrastructure or complexity. Using a basic understanding of threats and the Golang standard library, we could do a security-oriented design instead of leaving security as an after-though in an iterative way.

Photo by Silas Köhler on Unsplash

The Kubernetes dynamic client

Caio Ferreira — Fri, 28 May 2021 22:00:00 +0000

The Kubernetes dynamic client

Introduction

Kubernetes won the battle for the cloud-native platform and the characteristic that makes me enjoy the most working with it is its extensibility. By providing an open model through the kube-apiserver, without splitting an internal and external interface, we can interact with the cluster and any other system to integrate both from the same application (Controller) and even use custom resources to describe our unique operations, know as the Operator Pattern.

Although one could use any HTTP Client to interact with the API Server, this is no simple task. There are many resources with different response structures and possible operations if we only consider the core resources on Kubernetes. Hence, Kubernetes itself provides a set of clients for easier integration through the k8s.io/client-go project.

The most used client provided by this project is the k8s.io/client-go/kubernets.ClientSet, which is a typed client. What that means is that this interface provides exclusive methods for each resource on Kubernetes (think of Pods, Deployments, Services, everything!) and operation (Create, Get, List, Watch, Update, Patch and Delete). It is obvious why you should, whenever possible, prefer to use this client.

However, there are situations where this can be limiting. It is when k8s.io/client-go/dynamic.Interface, the dynamic client, will enter the game. This client has a twofold purpose:

First, it allows working with custom resources while avoiding strong dependencies. If you want to build some automation or workflow using another Operator as the building block, like ExternalDNS, CertManager, or Prometheus (Operator) usually you would need to add these projects as dependencies to use their Go types and register them on your client instance. This obviously introduces a lot of burdens as you now need to manage their ever-evolving versions and try to keep the version you have installed on the cluster matching the version on your go.mod.

Secondly, you can work with multiple or unknown resources. When your operator implements a generic logic that can interact with any common Kubernetes resource (from RBAC to Pods) and even custom resources, the dynamic client may be your only solution. A few examples are the garbage collection controller relies heavily on it and if you would want to add support for an arbitrary custom resource on a project like KubeWatch.

Therefore, let’s dive into this resourceful (pun intended) component of the k8s.io/client-go project and see how we can leverage it.

Basic operations with the dynamic client

The code below assumes to be running inside a Kubernetes cluster.

Many operations with the dynamic client is similar to the typed client, like creating a new instance can be done by providing the config to its constructor:

func newClient() (dynamic.Interface, error) {
    config, err := rest.InClusterConfig()
    if err != nil {
        return nil, err
    }

    dynClient, err := dynamic.NewForConfig(config)
    if err != nil {
        return nil, err
    }

    return dynClient, nil
}

Since the dynamic client has no knowledge about the resource you want to consume, it does not provide helper methods like CoreV1().Pod . Instead, you need to first provide a schema.GroupVersionResource, which is a Golang type that provides the necessary information to construct an HTTP request to the cluster API Server.

For example, if you want a function to list all MongoDB resources from the MongoDB Community Operator:

var monboDBResource = schema.GroupVersionResource{Group: "mongodbcommunity.mongodb.com", Version: "v1", Resource: "mongodbcommunity"}

func ListMongoDB(ctx context.Context, client dynamic.Interface, namespace string) ([]unstructured.Unstructured, error) {
    // GET /apis/mongodbcommunity.mongodb.com/v1/namespaces/{namespace}/mongodbcommunity/
    list, err := client.Resource(monboDBResource).Namespace(namespace).List(ctx, metav1.ListOptions{})
    if err != nil {
        return nil, err
    }

    return list.Items, nil
}

Note that if you are dealing with a namespaced resource then .Namespace(namespace) is obligatory, even if you will use an empty string to list on all namespaces.

In this snippet, we can see the main companion of the dynamic client: unstructured.Unstructured. This is a special type that encapsulates an arbitrary JSON while also complying with standard Kubernetes interfaces like runtime.Object , but most importantly it provides a set of helpers on the unstructured package to manipulate this data.

Expanding our example, if we would scale a MongoDB by an proportion we could do so like:

// ScaleMongoDB changes the number of members by the given proportion,
// which should be 0 =< proportion < 1.
func ScaleMongoDB(ctx context.Context, client dynamic.Interface, name string, namespace string, proportion uint) error {
    if proportion > 1 {
        return fmt.Errorf("proportion should be between 0 =< proportion < 1")
    }

    mongoDBClient := client.Resource(monboDBResource).Namespace(namespace)
    mdb, err := mongoDBClient.Get(ctx, name, metav1.GetOptions{})
    if err != nil {
        return err
    }

    members, found, err := unstructured.NestedInt64(mdb.UnstructuredContent(), "spec", "members")
    if err != nil {
        return err
    }

    if !found {
        return fmt.Errorf("members field not found on MongoDB spec")
    }

    scaled := int(members) * (1 + int(proportion))

    patch := []interface{}{
        map[string]interface{}{
            "op": "replace",
            "path": "/spec/members",
            "value": scaled,
        },
    }

    payload, err := json.Marshal(patch)
    if err != nil {
        return err
    }

    _, err = mongoDBClient.Patch(ctx, name, types.JSONPatchType, payload, metav1.PatchOptions{})
    if err != nil {
        return err
    }

    return nil
}

Here we leverage unstructured.NestedInt64 to access only the field that we are interested in, keeping our coupling to the MongoDB CRD to a minimum while also being able to manipulate the resource data with type safety.

The unstructured package has lots of helpers like this, not only for reading but also for writing to any field on the resource.

Performing all the usual operations on Kubernetes (get, list, watch, create, patch, and delete) follow the same approach: provide the scheme.GroupVersionResource and handle the unstructured.Unstructured result.

Controller with a dynamic client

More advanced but frequent use of a Kubernetes client is to build a controller that reacts to changes on the actual cluster state to bring it to the desired state.

Usually, we leverage an Informer, a component provided by k8s.io/client-go, that runs a handler when changes are detected, created from a typed client. Luckily the dynamic package also provides an Informer component that we can use.

For example, if we want to capture when a MongoDB is deleted to clean the associated PersistentVolumeClaims:

package main

import (
    "fmt"
    utilruntime "k8s.io/apimachinery/pkg/util/runtime"
    "k8s.io/apimachinery/pkg/util/wait"
    "k8s.io/client-go/dynamic"
    "k8s.io/client-go/dynamic/dynamicinformer"
    "k8s.io/client-go/tools/cache"
    "k8s.io/client-go/util/workqueue"
    "time"
)

const maxRetries = 3

var monboDBResource = schema.GroupVersionResource{Group: "mongodbcommunity.mongodb.com", Version: "v1", Resource: "mongodbcommunity"}

type MongoDBController struct {
    informer cache.SharedIndexInformer
    stopper chan struct{}
    queue workqueue.RateLimitingInterface
}

func NewMongoDBController(client dynamic.Interface) (*MongoDBController, error) {
    dynInformer := dynamicinformer.NewDynamicSharedInformerFactory(client, 0)
    informer := dynInformer.ForResource(monboDBResource).Informer()
    stopper := make(chan struct{})

    queue := workqueue.NewRateLimitingQueue(workqueue.DefaultControllerRateLimiter())
    informer.AddEventHandler(cache.ResourceEventHandlerFuncs{
        DeleteFunc: func(obj interface{}) {
            key, err := cache.DeletionHandlingMetaNamespaceKeyFunc(obj)
            if err == nil {
                queue.Add(key)
            }
        },
    })

    return &MongoDBController{
        informer: informer,
        queue: queue,
        stopper: stopper,
    }, nil
}

func (m *MongoDBController) Stop() {
    close(m.stopper)
}

func (m *MongoDBController) Run() {
    defer utilruntime.HandleCrash()

    defer m.queue.ShutDown()

    go m.informer.Run(m.stopper)

    // wait for the caches to synchronize before starting the worker
    if !cache.WaitForCacheSync(m.stopper, m.informer.HasSynced) {
        utilruntime.HandleError(fmt.Errorf("timed out waiting for caches to sync"))
        return
    }

    // runWorker will loop until some problem happens. The wait.Until will then restart the worker after one second
    wait.Until(m.runWorker, time.Second, m.stopper)
}

func (m *MongoDBController) runWorker() {
    for {
        key, quit := m.queue.Get()
        if quit {
            return
        }

        err := m.processItem(key.(string))

        if err == nil {
            m.queue.Forget(key)
        } else if m.queue.NumRequeues(key) < maxRetries {
            m.queue.AddRateLimited(key)
        } else {
            m.queue.Forget(key)
            utilruntime.HandleError(err)
        }

        m.queue.Done(key)
    }
}

func (m *MongoDBController) processItem(mongodb string) error {
    // Clean up PVCs
    return nil
}

Most of this code is standard, like the work queue, informer event handlers, and item processing, of a controller using the typed client.

Hence, leveraging the decoupling provided by the dynamic client really comes with a low overhead in terms of complexity.

Testing with the dynamic client

If we were to scale the use of the dynamic client it is paramount that it is as easy to test as the typed client.

As in the controller case, the dynamic package provides an equivalent fake client that allows for stubbing objects and asserting actions performed using it.

package main

import (
    "context"
    "k8s.io/apimachinery/pkg/runtime"
    dynamicfake "k8s.io/client-go/dynamic/fake"
)

func TestDynamicClient(t *testing.T) {
    // Setup an Object as mock on the client
    // Write it like its YAML manifest
    mdb := &unstructured.Unstructured{}
    mdb.SetUnstructuredContent(map[string]interface{}{
        "apiVersion": "mongodbcommunity.mongodb.com/v1",
        "kind": "MongoDBCommunity",
        "metadata": map[string]interface{} {
            "name": "mongodb-test",
            "namespace": "default",
        },
        "spec": map[string]interface{}{
            "members": 3,
        },
    })

    dynamicClient := dynamicfake.NewSimpleDynamicClient(runtime.NewScheme(), mdb)

    // Run any logic that depend on the dynamic client
  NotifyMongoDBs(context.Background(), dynamicClient)

    AssertActions(t, dynamicClient.Actions(), []ExpectedAction{
        {
            Verb: "list",
            Namespace: "default",
            Resource: "mongodbcommunity",
        },
    })

}

Using the unestructured.Unestructured type we can create stub Kubernetes objects using the same syntax as in YAML, but with maps.

After performing the tested logic we can use dynamicClient.Actions() to see all operations that were performed by our code. However, manually asserting these actions on every test often lead to unreadable code and brittle assertions.

Hence, I often use a special assertion function AssertActions that verify if every expected action can be found in the performed actions. An important note is that this function does not perform an exact list match, i.e. if a delete operation was performed using the client the test would not break, the only condition for the AssertActions to fail is if the list operation provided on the expected list isn’t found. One could change the asserting function or make a sibling function that validates only if the expected actions were performed.

Although the current implementation is verbose, this function has the benefit of working both with the dynamic and the typed client.

type ExpectedAction struct {
    Verb string
    Name string
    Namespace string
    Resource string

    // Patch action
    PatchType types.PatchType
    PatchPayload []map[string]interface{}
}

func AssertActions(t *testing.T, got []kubetesting.Action, expected []ExpectedAction) {
    if len(expected) > len(got) {
        t.Fatalf("executed actions too short, expected %d, got %d", len(expected), len(got))
        return
    }

    for i, expectedAction := range expected {
        if !AssertExpectedAction(got, expectedAction) {
            t.Fatalf("action %d does not match any of the got actions", i)
        }
    }
}

func AssertExpectedAction(got []kubetesting.Action, expectedAction ExpectedAction) bool {
    for _, gotAction := range got {
        switch expectedAction.Verb {
        case "get":
            getAction, ok := gotAction.(kubetesting.GetAction)
            if !ok {
                continue
            }

            if getAction.GetName() != expectedAction.Name {
                continue
            }

            if !validateNamespaceAndResource(getAction, expectedAction) {
                continue
            }

            return true
        case "list":
            listAction, ok := gotAction.(kubetesting.ListAction)
            if !ok {
                continue
            }

            if !validateNamespaceAndResource(listAction, expectedAction) {
                continue
            }

            return true
        case "watch":
            watchAction, ok := gotAction.(kubetesting.WatchAction)
            if !ok {
                continue
            }

            if !validateNamespaceAndResource(watchAction, expectedAction) {
                continue
            }

            return true
        case "create":
            createAction, ok := gotAction.(kubetesting.CreateAction)
            if !ok {
                continue
            }

            if !validateNamespaceAndResource(createAction, expectedAction) {
                continue
            }

            return true
        case "update":
            updateAction, ok := gotAction.(kubetesting.UpdateAction)
            if !ok {
                continue
            }

            if !validateNamespaceAndResource(updateAction, expectedAction) {
                continue
            }

            return true
        case "delete":
            deleteAction, ok := gotAction.(kubetesting.DeleteAction)
            if !ok {
                continue
            }

            if deleteAction.GetName() != expectedAction.Name {
                continue
            }

            if !validateNamespaceAndResource(deleteAction, expectedAction) {
                continue
            }

            return true
        case "patch":
            patchAction, ok := gotAction.(kubetesting.PatchAction)
            if !ok {
                continue
            }

            if patchAction.GetName() != expectedAction.Name {
                continue
            }

            if !validateNamespaceAndResource(patchAction, expectedAction) {
                continue
            }

            if patchAction.GetPatchType() != expectedAction.PatchType {
                continue
            }

            patchBytes, err := json.Marshal(expectedAction.PatchPayload)
            if err != nil {
                continue
            }

            if !bytes.Equal(patchAction.GetPatch(), patchBytes) {
                continue
            }

            return true
        }
    }

    return false
}

func validateNamespaceAndResource(action kubetesting.Action, expectedAction ExpectedAction) bool {
    return action.GetNamespace() == expectedAction.Namespace && action.GetResource().Resource == expectedAction.Resource
}

This asserting function allows for more conditions to be added, like verifying list/watch restrictions and create/update bodies.

Conclusion

The Kubernetes ecosystem is rich and every now and then we stumble upon this kind of treasure. I strongly recommend reading through the documentation not only of the k8s.io/client-go but also other like sigs.k8s.io/controller-runtime project and the Kubernetes Reference API documentation.

Introducing Cache in your System

Caio Ferreira — Mon, 04 May 2020 19:00:00 +0000

Photo by Joshua Coleman on Unsplash

Caching is one of the most popular tools used to scale systems and anyone looking to maintain high throughput, resilient and cost-effective products should understand how to use it because it is financially impractical to apply only compute resources in order to meet the access demands.

Knowing the basics about it and what parameters you should be looking when choosing your solution is rarely addressed and hence is the purpose of this article.

Don’t rush your decision

If one would search for caching it will find a plethora of tutorials teaching how to set up your solution. They make it look so easy to use a cache in your application that one may do it without thinking twice. Be careful, every choice comes with costs and tradeoffs, caching is no different.

Usually, we can find bad cache designs when we stumble on the most important metrics for it: hit ratio and miss ratio. As a brief overview, we can define these metrics as follow:

Hit ratio : when the cache has a key and can provide the value for the system to use, we call it a hit. The metric is simply the number of hits / number of lookups.

Miss ratio : when the cache hasn’t the key and the value must be computed for the system to use, we call it a miss. The metric is simply the number of misses / number of lookups.

The number of lookups is simply the total quantity of cache accesses, which is lookups = hits + misses.

You can know that the cache is not being efficient if it has a low hit ratio and, therefore, a high miss ratio. What low or high means will depend on your problem. Discovering your baseline metrics can only be achieved through our first guideline.

Guidelines

Monitor your cache

You should set up a way to capture the hit and miss ratio of your cache solution. How you will do it is highly dependable on the chosen implementation, but most of it should have an easy way of extracting these statistics and if not, consider looking for others.

The most important about having monitoring is exactly that you will be able to experiment with different algorithms and tradeoffs presented in the below guidelines and strive to improve your ratios. Hence, formulate a hypothesis and let your data drive your solution.

Tradeoffs: Performance and Resilience vs Consistency

Caching is one of the simplest and more powerful ideas in computation. Understanding and applying its basic cases is easy but can become extremely hard sooner than you imagine. Therefore the classical phrase, “there is only two difficult things in computer science: cache invalidation and naming things”. But, why cache invalidation is so hard? Because it usually is critical and has many moving parts.

The moving parts come from the fact that caches improve application performance because they bring the data closer, which also means we now have gone off the rails with the most important principle for Consistency: have one source of truth. This has another effect which is resilience, since now if our main source of information goes off, our application can survive a little longer with its cached values.

The critical segment arises usually when you need to update the data on your cache. In order to give your user meaningful information, you need to understand the access patterns to choose the right eviction algorithm and parameters which will balance performance and correctness, choosing the wrong one will probably damage your product. Besides that, there is the case when you need to force clean your cache and the distributed nature of it can cause a lot of pain in the process of invalidating each cache node.

Therefore, adding a cache to your solution is a trade-off between Performance+Resilience versus Consistency and so the first question you should ask yourself is “Can my system live with potentially old and invalid data?” If your answer is Yes, then you can continue here, otherwise, caching will do you more harm than good.

Keyspace

So you decided that you really want a cache. The first decision you will have to make is about your keyspace, i.e. what you will use as a cache key to index your costly computed values?

It is important because you need to analyze your key cardinality, which means how many distinct values your key can have. For example, a boolean key has a cardinality of 2 (true or false) whereas a customer id can have thousands of possible values. This is really important to understand because a low cardinality key would limit the amount of data your cache could store but your hit ratio would be really high. On the other side, if a key has an extremely high cardinality (tending to uniqueness, never repeating itself) your cache could grow exponentially and you may end up with a low hit ratio, demanding much computation and providing little performance improvements.

Hence you want to choose a high cardinality key, avoiding never-repeating ones, but not to small, avoiding limited ones. One way many uses to achieve this balance is to use complex keys (like a map or list), mixing a medium cardinality key like customer id with a low cardinality one like state names.

One scenario that you should be careful is with memoization. For those coming from OO lands, it is a technique to cache the values computed by a function. It uses the function’s arguments as keys and caches the result. But, since a function may change over time and you may use a solution that wraps the function in a place far from the local here it is implemented, there is a great risk that some feature or refactoring adds unique arguments (like a timestamp) or reduce the arguments to low cardinality ones (a small enum and a boolean). You should implement memoization near to the implementation and/or use cache solutions that allow you to choose the keys from the argument list, which is the best solution since one could not accidentally chance that.

Cache algorithms and strategies

Next, you need to understand your access pattern in order to choose your cache algorithm and possible strategies. Each one will have trade-offs, you can combine some of them and you should experiment because in this area data will be better to guide you.

FIFO (First in First out): this is the simplest algorithm where the cache works like a queue and evict the first block to enter, independent of how many times it was used. Through time, you will have the most used keys remaining in the cache, since even if a block was evicted, since it is highly used, duplicated blocks of this key will be presented at the queue. This strategy is really simple to implement and has low overhead but also has an inefficient usage of memory in comparison to other algorithms.
LRU (Least Recently Used): probably the most used cache algorithm, it tracks when some block was used and evict the one with fewer accesses. Hence it keeps the most used keys in the cache but with better memory usage. As a tradeoff, it has a more complex and costly implementation since it has to add and track age bits in the cache blocks.
LFU (Least Frequently Used): imagine that you are using an LRU cache and you have 100 accesses in the last second. There were 80 hits in the key A, 19 hits in key B and 1 hit in key C. If your cache is full, in the next miss that needs to load a new key, your cache would not evict C, because it was the most recent one accessed. This can be really bad since we are probably removing a more usage key (A or B) in favor of C. That is the problem that the LFU algorithm addresses since it keep the most frequent usage keys, it would evict the key C from our example because it has a small access frequency. This is a really interesting model and probably is better suited to most use cases, but as a pattern, it also has more overhead than LRU because how it has to keep track of how many times a block was accessed in relation to how many accesses happened to the cache.
TTL (Time to Live): one really common strategy is time to live, an algorithm that evicts blocks that are older than a certain pre-defined timespan. It is used for more volatile data and usually with two cases: for low cardinality keys that can’t grow the memory footprint to the point where one block would be evicted or in combination with other cache strategies (like the ones mentioned above) to provide more refresh opportunities.
Stale data : we say that a block is stale when it passes its expiration time and should be evicted or refreshed. The stale data strategy is an augmentation of TTL that instead of eliminating the block from the cache once the time to live expires, it runs a refresh function (in case of memoization it reruns the function) that will compute a new value for the key. During this time, the stale (old) data is served for those that access the cache. It can also be the case where if the refresh function fails, it simply maintains the block in the cache, that may be evicted by some algorithm, but is not eliminated by its expiration. This can be a really powerful strategy for increased resilience if your system support living with a possibly long-living stale data.

Besides these there is also more modern algorithms like Windowed TinyLFU (used by Caffeine), LIRS and ARC. Note that various discussions about the cache algorithm will reference the theoretical Bélady’s algorithm, so it is good to have a look at it.

Local vs Distributed

One last question you might need to answer is if you are going to use a local or distributed cache. This will be the most important question in financial terms, so look close to your needs.

Local : it means that you will maintain the data in the memory of an application instance. This is the most simple setup, but it can impose a burden in the allocated RAM, maybe forcing you to upgrade to a bigger instance, which can be pretty expensive. It also can have suboptimal performance-wise, since the worst-case scenario for a cache with a size limit of 1000 keys is to have just it stored across all instances, that is, the same data is replicated in all local caches. This intersection diminishes the total performance gain across your system.

Distributed : in this case one would use a solution like Redis or Memcache, running separately from the application, and being shared by all instances. This solution optimizes the resource usage (the RAM is allocated exclusively for the cache), allow for bigger cache entries and improves data density (there is, no more replicated entries), but even then it can be less performant than the local solution since it demands a network roundtrip to access the database. Besides that, the cost to maintain new instances and the skill needed to operate these should not be overlooked.

All these points should be taken into consideration when choosing the design for your cache in your system.

Conclusion

Therefore, before looking into tutorials and getting started articles about how to set up your cache, think about and discuss your problem and how you could configure the cache to deliver the most value to your system. You can use these and other guidelines as starting points for the debates and focus on guiding your decisions based on the data measured from your implementation.

I hope you found it useful and you can find more of my thoughts at caioferreira.dev

The Lambda Path

Caio Ferreira — Tue, 08 Oct 2019 19:00:00 +0000

Photo by Roman Mager on Unsplash

Introduction

Being passionate by functional programming I am often asked about how and where one can learn more about this style. Hence, I decided to compile some resources on the subject to suggest a path to this beautiful world.

I split the content into three levels: Beginner, Intermediate and Advanced. Since this is thought of as a starter to master path, I gave preference to contents in Javascript, which is a well-known language that allows for a great extent of functional patterns.

Some purists probably will hate me for this choice.

In this path I tried to add resources on fundamental functional concepts and tools, then an introduction to some functional patterns like Combinators, Category Theory, Railroad Programming and finally a material to deepening into complex subjects.

After all, I also added some tips on dedicated functional languages that I found easy to play with or that I simply love to work with.

The Yellow Bricks Road

Beginner

Functional Programming Principles - Article about basic principles of the functional paradigm, like pure functions and immutability

Functional programming in Javascript - Highly recommended. Video series from the renowned Youtube channel FunFunFunction about functional tools and concepts in Javascript. The explanation is deep and clear about the subjects.

Eloquent Javascript: Functional Programming - The Eloquent Javascript’s chapter about functional programming. It is a hands on to building fundamental tools by hand, understanding its applications and usages.

Becoming Functional - An excellent book about the functional paradigm as a whole, presenting topics from functional principles like pure functions to advanced techniques like pattern matching. Unfortunately, the example is in Java.

Intermediate

Functional Design Patterns - An amazing talk about how to combine different pieces of pure functions into more complex applications.

Javascript Combinators - a talk from one of the greatest names in Functional Javascript, Reginald “Raganwald” Braithwaite, where he explores the concept of composition and introduces the Combinators pattern, a powerful functional tool inspired by Lambda Calculus.

Professor Frisby Mostly Adequate Guide to Functional Programming - An excellent resource on more complex functional patterns using Category Theory. This approach is less common on Front End and probably we can get most of the benefits of the functional paradigm without the complexity of an entire branch of mathematics.

Advanced

Category Theory for Programmers - The most recognized and advanced resource on Category Theory for development ends.

To Grok a Mockingbird & Why Y? Deriving the Y Combinator in JavaScript - two incredible articles from Reginald Braithwaite about the application of the Combinators pattern.

Going full functional

If you enjoyed your travel and want to keep going into this path, I recommend that you play with some fully functional programming language. I suggest Elm, ReasonML, and Clojure.

The first two (Elm and ReasonML) are functional languages that compile to JavaScript and are used to build front end applications. A special note to ReasonML which has great integration with React, being a language developed by Facebook.

The last one is Clojure, a JVM language, dynamically typed and from the LISP family. It is used primarily on the back-end and is well suitable for describing complex domain since it’s syntax is very flexible and can be customized in a way to better describe your problem space.

You can find more about these languages here:

Elm: Documentation and Code Example
ReasonML: Documentation and Code Example
Clojure: Best Book for new Clojurists and Code Example

Summary

The Lambda World is a vast and beautiful one, you will find many ways of thinking and solving problems. That is the main reason why I love FP, it not only brings new tools to your workbench but also constantly shifts your mental models. In this segment I could not leave the most impactful talk on my perspective on the work and profession on software engineering: Simple Made Easy by Rich Hickey, the creator of Clojure.

I hope these tips help you and fair travel!

Why I Love Gatsby

Caio Ferreira — Tue, 16 Apr 2019 18:51:50 +0000

Photo by Patrick Fore on Unsplash

Recently I launched my own blog and it was an amazing experience. In about a Sunday I had made 90% of the site and enjoyed every moment. This was thanks to Gatsby and here I will show you why!

Introduction

When I decided to share more about what I am doing and learning during my career I immediately thought about a blog. Being a passionate reader and dev.to fan, my first step was to jot down my ideas, structure them and release it at the Dev Community. It was great and the feedback was exciting.

But for me, technical writing wasn't just about sharing knowledge and build an image. It is like a diary, a personal mirror to look to myself and my history as a professional. I would like it to reflect me and my steps as a go through each one. So, being such a particular subject, I felt that I need something more individual, hence, my own page.

Reasons to ❤️

Build a personal blog has lots of options like Wordpress, Blogspot, Jekyll, and others. In this miscellanea, the JAM Stack caught my attention a long time ago. It is a web architecture to build fast, easy and scalable websites based on Javascript + API's + Markup. It essentially builds content from Markdown and other API sources in build time and can add dynamic behavior with client-side Javascript. Most JAM frameworks are also called static site generator, like the above mentioned Jekyll and Hugo. Since Jekyll is built in Ruby and Hugo in Go, they need to use template engines to design its pages, which leads us to the first reason.

React

Working mostly with Front-End development in React, it seemed natural for me that it could be used as a template engine and, as such, when I first met Gatsby it fit perfectly. With React we have a well know template API that is simple JSX/HTML and easy componentization and composability.

Architecture

Gatsby is an open source framework that allows us to build performant sites using React and multiple data sources, like Markdown, CMS (especially headless ones), web APIs and multiple kinds of files (JSON, YAML, CSV, etc). All data sources are abstracted behind the GraphQL API, creating a uniform way to access and manage data. This provides an extensible and decoupled architecture that brings with it a large plugin library for the data end and all the tools available in the React ecosystem.

Open source

In terms of community, documentation, and content, Gatsby is one of the best open source projects that I ever saw. If you are planning to use it, I strongly suggest that you read the official tutorial, it is awesome and will get you an excellent overview of its architecture. Besides that, the community provides lots of starters repositories to help you with your project. In my case, the gatsby-starter-blog provides a setup with features that any blog would like: SEO tools, RSS feed, Google Analytics, website manifest, offline support, CSS for theming and typography configuration.

Customization

All the features listed above are provided through plugins, for example, the SEO tools are provided with the Helmet plugin (which allows us to configure the HTML head with a title and keywords). These allow us to easily customize our project. In my blog I also added the plugin for SVG support (which allows me to import an SVG like a javascript module), a bunch of plugins better image rendering (including low resolution samples during page load) that you can find here, code styling and syntax highlighting with PrismJS plugin.

But the customization that I most enjoyed was the combination of Styled Components (through this plugin) with Typography.js (also with a plugin, but which came with the starter) because Styled Components enables me to use common CSS tools like media queries and pseudo elements and also Javascript functions, which allowed me to use the rhythm Typograph.js function to set margins and paddings in a harmonic way. Also, as the starter came with CSS Module support, I could create a global stylesheet that contained CSS variables and made easy for making the theme consistent.

Developer Experience

You just need 3 commands to use Gatsby:

    $ npm install -g gatsby-cli
    $ gatsby develop # inside the starter repository
    $ gatsby build # when you are finished implementing your site

And for adding plugins you just need to install an npm package and place its name in the gatsby-config.js file. It is easy, simple e fun to use! 😄

Simple and Fast Delivery

From developing to the final website you are only one command away: gatsby build. The generated code is simply static HTML and CSS and can be deployed to any static hosting service like Github Pages. Therefore, you can easily build and deploy your site for free.

However, for my blog, I decided to host on Netlify. Their service is so simple that not even documentation is needed, although existent with you need, because you can click to deploy your site and they will take you to point to a Github repository, choose a command to build your site (therefore, we can add gatsby-cli as a devDependency and create an npm script to build the site using the node_modules binary) and "boom", you will have it in a random domain.

Besides that, if you also want a custom domain you can buy it directly through them and your site will also be running in HTTPS using a Let's Encrypt TLS certificate provided and configured by Netlify.

It is also a platform with developer experience in mind and with lot's of features to expand your site to a web application if you want.

Pro Tips

If you are planning to use Gatsby to build your blog I have some extra tips!

Seek inspiration

Today the dev blog that I most consume is Overreacted by Dan Abramov and since he used Gatsby to also build his blog I learned a lot from reading his source code opened in Github. It is extremely helpful to have a ground in what you like in a blog if you, like me, don't have much UI design skills.

Run Lighthouse

Chrome comes with an incredible tool for auditing websites in 4 pillars: Performance, Accessibility, Best Practices, and SEO. One of the Gatsby killing features is that its defaults delivery you a high score site in this evaluation, but run each is really helpful to catch any mistakes that you may let pass. Like me, that forgot to add aria-label 's to social media link icons.

Theme

Have a theme for your blog will help to make a brand for yourself and improve the reader experience when using your website. As I'm a disaster with choosing colors, the Coolors was my salvation and delivered a nice color schema.

Summary

Finally, Gatsby is not just useful but also really fun to play with. I enjoyed every moment using it and in a Sunday I managed to implement 99% of the website, leaving only minor tweaks to be corrected after. In less then a busy week, I put my blog on air and enjoyed the view!

If you liked the post you can see more at caioferreira.dev and in case of any question or opinion please leave a comment and let's talk about it!

Using Technical Debt as your next Tool

Caio Ferreira — Thu, 28 Mar 2019 13:31:03 +0000

Quick Summary

In this post I will show you what is the debt that we collect with during the software lifecycle, what are its causes and how to pay it back.

Introduction

Often we are faced with a dilemma in software development: implement the best solution for the feature or delivery it quickly but assuming some workarounds and code smells? Whichever side you choose, will be a cost.

This cost is even greater if you work on legacy projects or high changing environments. I am a software engineer @ B2W, the biggest e-commerce on Latin America, and because of its long history, we deal with +6 years legacy applications, in which dozens of developers worked through this time. Sum up the e-commerce environment with its high competitiveness that drives constant changes and new features then you will have a millionaire decision at your hands.

Hence, understating how to manage this cost can be a key asset to achieve a product that has both high quality and a short time to market.

The decision cost

To understand how to manage the decision cost, we must first learn what cost is. Every decision we make in a software project, not just about best solution vs quick & dirty, incurs in a time cost. Sometimes we cannot take this time and need to target production rather earlier than later. In this situation, we exchange bad code for time. Apply this behavior through time and we get a large debt.

Therefore, Technical Debt is a metaphor in software development, based on the financial debt, about the accumulation of low-quality code in a project over time.

Extending the metaphor, we can see financial debt as a resource uptake from some institution, therefore, in the software scope, the resource we borrow is time instead of money and the institution that provides us with this is our code base rather than a bank. Thus, in technical debt calculation, the main amount is the time cost to refactor the codebase to a clean design and the interest represents the extra time that will be taken in the future if the team has to work on the messy code.

Even more debt

Although the best solution vs quick & dirty decision is an excellent example to introduce technical debt, it isn't its only source. As Fowler (2009) points out, this case is just a Prudent and Deliberate debt, where the team is aware of the cost it is incurring and know how to pay it back. But we can have four types of debt in total.

Each debt type has a source associated with it:

Prudent & Deliberate - the team know how to build good software and has the opportunity to deliver more product value if they speed up. They analyze the tradeoff and judge that cost/earn ratio is enough to justify the bad code, otherwise delaying the solution. Then, they plan how to refactor the debt and implement it as early as possible.
Reckless & Deliberate - the team has the skill to build a well-designed solution but do not has the tools or support to do so. This happens when a team is purposely taking debt but without a plan to repay it or to do so in a code that probably will never change. This scenario is more common on unmotivated teams, legacy projects and due to management pressure for fast delivery instead of a constant and quality delivery.
Reckless & Inadvertent - the team doesn't know good design practices and incur on costs due to lack of training, experience, and leadership.
Prudent & Inadvertent - it is a common case that after we delivery a feature or project we realize that the best design approach would be other than what we did. This situation is unpredictable and can happen to any team due to lack of domain knowledge, obscure requirements, and technology limitations.

Of course, this exploration does not exhaust the possible causes for technical debt but includes the most frequent ones.

Living on high debt

Since the debt causes can be so many, projects can experience an ever-increasing technical debt, to the point where adding new functionality, fix current bugs and operate the application becomes impossible.

This type of scenario can be catastrophic, because while the team lives with this application they will lose agility, will observe an increasing bug count, have loss of motivation, increased stress, long production problems (long living bugs), customer complains and possible single points of failure in case only a few people know how to develop and operate the project.

When an application reaches this point we say that it went into technical bankruptcy.

However, using the right strategies we can make the highlighted curve steeper, leading the actual curve nearest to the idealized curve.

Paying back

In order to avoid that applications achieve high debts or to handle projects already with high debt, we need to build a repayment plan. This can have any sort of effort in order to refactor and pay the time due, but the industry set some strategies that have proven effective.

Technical Backlog: each task should have a brief description, the reason that the change is important for the project and which part of the code base must change. Like any other task, we should estimate the effort needed to build a good and clean solution. When estimating the cost, the team should add an interest cost that is directly proportional to the probability of this code to change in the future and hence prioritizing the most costly task. A precise estimation is really difficult but a rudimentary approximation is enough to guide the decisions.
- With this approach the technical debt becomes visible to all stakeholders and the decision of doing such a task can be made upon effort and future impact.
- The task cost is easily traceable.
- Don't mix technical and feature tasks.
Refactoring costs included on the feature estimation: explain the costs is necessary since we follow the principle that no feature should be implemented above bad code and therefore this code should be refactored first (Boy Scout Rule). This is a fundamental principle once bad code already incur in interests and adding a new feature using it will increase this interest exponentially, to the point where the cost to work on this code will be so high that will be impossible to make further changes.

Besides that, there are two other critic strategies that can be used depending on how high debt the project is in.

Buffer Tasks: the task has a portion of the team's sprint. It can be used to unplanned refactorings, unforeseen problems, discoveries, etc. To avoid the task to be wasted on non-relevant work members of the team should always propose how they pretend to use the time and discuss with the rest of the team.
Clean-up Releases: periodically or in critic scenarios, a team can make releases only with technical refactorings. This strategy is only useful if there is already a refactoring list to be made. Furthermore, the business team should give support as this will probably delay new features. The teams should consider using this when a greater effort is necessary, like in big architectural changes, infrastructural changes, build & deploy, etc.

Conclusion

Therefore, Technical Debt can be key tool from the product perspective. It allows business people to improve time to market and customer satisfaction with the cost of bad code.

However high debt can bring to huge problems to the project, hence we must strive to be Prudent and Deliberated about our debt, always taking it with a repayment plan together, using all kinds of strategies to keep the debt under control.

To achieve this all stakeholders must be aware of the debt's nature and existence and bring the development team together in the decision.

References

https://www.infoq.com/presentations/debt-aware-culture
https://martinfowler.com/bliki/TechnicalDebtQuadrant.html
https://www.infoq.com/minibooks/emag-technical-debt?utm_source=minibooks_about_TechnicalDebt&utm_medium=link&utm_campaign=TechnicalDebt
https://books.google.com.br/books?id=pKVFDwAAQBAJ&lpg=PA18&ots=Gc9vC8vf80&dq=Software Quality Assurance Pressman 2014&hl=pt-BR&pg=PA18#v=onepage&q=Software Quality Assurance Pressman 2014&f=false

Difference between state and State

Caio Ferreira — Sun, 09 Dec 2018 11:55:00 +0000

Photo by Annie Spratt on Unsplash

Today we will try to solve the ambiguity in the concept of state, highlighting the differences in the two main notions about it. The post describes the nature of each one, some use cases and how they fit in the object and functional paradigms.

Introduction

On the last couple of months, I dove into the topic of State Machines and how we can design UI’s with this concept in order to provide better semantic and predictability to our application. When reading and talking about it often I have to stop and clarify which of the two ideas about state I am referring to:

the idea of a collection of data at a point in time
the idea of the representation of an entity modeled as a state machine.

For the sake of comprehension, we will use state to refer to the first and State to the last.

Requirements

We will use Typescript for our yummy examples so some familiarity with it would be good.

The state as a travel bag

The first notion we became comfortable with when learning about state in software development is the entity “travel bag”. Basically, we see a state as a collection of data in a specific point in time. Throughout the application lifecycle, this data is manipulated and altered in order to reflect the business process. For example:

    class Pizza {
        private dough: Dough; // it's an enum that could be traditional or thin
        private ingredients: Array<Ingredient>;

        // entity controls
        private isBeingnPrepared: boolean;
        private isBaking: boolean;
        private baked: boolean;

        constructor() {
            this.isBeingnPrepared = true;
            this.isBaking = false;
            this.baked = false;
        }

        // getters and setters

        async public bakePizza(): void {
            const oven = new OvenService();

            try {
                this.isBeingnPrepared = false;
                this.isBaking = true;

                await oven.bake(this);

                this.baked = true;
            } catch (error) {
                throw error;
            }
        }
    }

From this point on, the pizza state, and hence the application state, has changed, because its data was updated. However, two booleans can be arranged in four different ways, and some of them are invalid states. In the object-oriented paradigm we would avoid this by encapsulating such data in an object and modeling its operations only through methods that guarantee atomic and consistent changes.

Until this use case, our model seems to be fine. But, the time comes to implement the next step in the pizzeria flow, the delivery.

    class Pizza {
        private dough: Dough;
        private ingredients: Array<Ingredient>;

        // entity controls
        private isBeingnPrepared: boolean;
        private isBaking: boolean;
        private baked: boolean;
        private isBeingDelivered: boolean;
        private hasBeenDelivered: boolean;

        constructor() {
            this.isBeingnPrepared = true;
            this.isBaking = false;
            this.baked = false;
            this.isBeingDelivered = false;
            this.hasBeenDelivered = false;
        }

            // getters and setters

            // bake behavior

        async public deliveryPizza() {
            if (!this.baked) {
                throw new PizzaNotBakedException();
            }

            const deliveryService = new DevelieryService();

            try {
                this.isBeingnPrepared = false;
                this.isBeingDelivered = true;

                await deliveryService.send(this);
            } catch (error) {
                throw error;
            }
        }

        public notifyDelivery(wasSuccessful) {
            if(wasSuccessful) {
                this.hasBeenDelivered = true;
            }
        }
    }

What raises a flag in this code is the use of a guard condition at the start of the delivery function that checks if the pizza is baked. If not, it throws an exception. This seems really simple, and if this were the only condition, it would be fine. But, a pizza could already be left for delivery, as such, we don’t want to try to send it again. So, we add another guard condition to our function:

    class Pizza {
        private dough: Dough;
        private ingredients: Array<Ingredient>;

        // entity controls
        private isBeingPrepared: boolean;
        private isBaking: boolean;
        private baked: boolean;
        private isBeingDelivered: boolean;
        private hasBeenDelivered: boolean;

            // constructor

            // getters and setters

            // bake behavior

        async public deliveryPizza() {
            if (!this.baked) {
                throw new PizzaNotBakedException();
            }

            if (this.isBeingDelivered) {
                throw new PizzaAlreadyLeftException();
            }

            const deliveryService = new DevelieryService();

            try {
                this.isBeingPrepared = false;
                this.isBeingDelivered = true;

                await deliveryService.send(this, this.notifyDelivery);
            } catch (error) {
                throw error;
            }
        }

            // notify delivery behavior
    }

If we elaborate all the scenarios which a pizza can be, this kind of implementation with lots of branches and conditions expressed by if/else statements grows exponentially. It increases our code cyclomatic complexity and diminishes maintainability as such code is more fragile, harder to read and understand.

It gets worse when this kind of conditional start to spread across the code, as in the bake function, which needs to be updated in order to not try to bake it again.

    class Pizza {
        private dough: Dough;
        private ingredients: Array<Ingredient>;

        // entity controls
        private isBeingPrepared: boolean;
        private isBaking: boolean;
        private baked: boolean;
        private isBeingDelivered: boolean;
        private hasBeenDelivered: boolean;

        // constructor

            // getters and setters

        async public bakePizza(): void {
            if (this.baked) {
                throw new PizzaAlreadyBakedException();
            }

            const oven = new OvenService();

            try {
                this.isBeingPrepared = false;
                this.isBaking = true;

                await oven.bake(this);

                this.baked = true;
            } catch (error) {
                throw error;
            }
        }

        // delivery behavior

            // notify delivery behavior
    }

Although this kind of design serves several proposes, in special on more simple or data-centric scenarios, in fast evolution and process-centric domains it evolves on a mess of code execution paths and unsynchronized conditionals through different functions.

The state as an entity travel bag has a use and it is to carry the associated information to the model. Try to control the behavior of this entity through the same concept ends up overloading it with responsibility and creating a silent trap for our design.

The problem faced here is that the application architecture allows for invalid behavior through invalid states, and when it does eventually some use case will expose the bugs created by this freedom. Besides that, this approach takes the system invariants, in this case, the Pizza cooking flow, and scatter then inside many implementation points instead of enforcing them in the design.

Side note: if you are versed in Algebraic Data Types you can see this as a Product Type with cardinality which tends to infinity.

Representational State

Once we have the problem of control the entity information and behavior being done by the same construct, the state, our response could not be more simple: let’s break these responsibilities.

Therefore, we need a new pattern to handle our entity’s behavior.

But, the alternative pattern we propose when designing your application is not at all new. It is the State Pattern, describes in many ancient books about OO. And this books will tell you the same, that the State Pattern seeks to delegate an entity behavior to a specific implementation which is the current State and at the end of the method calculate the entity’s next State, which will now represent the entity, replacing its behaviors implementation on the fly. After all, this pattern is a translation of a state machine to the idiom of the nouns. An alternative implementation for our Pizza example can be as below:

    interface IPizza {
        bakePizza();
        deliveryPizza();
        notifyDelivery(wasSuccessful: boolean);
    }

    type Pizza = PreparingPizza | BakedPizza | DeliveringPizza | DeliveredPizza;

    class PreparingPizza implements IPizza {
        private dough: Dough; // it is an enum that could be traditional or thin
        private ingredients: Array<Ingredient>;

        constructor(dough: Dough, ingredients: Array<Ingredient>) {
            this.dough = dough;
            this.ingredients = ingredients;
        }

        // setters

        getDough() {
            return this.dough;
        }

        getIngredients() {
            return this.ingredients;
        }

        public async bakePizza(): Promise<BakedPizza> {
            const oven = new OvenService();

            try {
                await oven.bake(this);

                return new BakedPizza(this);
            } catch (error) {
                throw error;
            }
        }

        public async deliveryPizza() {
            throw new PizzaNotReadyForDelivery();
        }

        public notifyDelivery(wasSuccessful) {
            throw new PizzaNotReadyForDelivery();
        }
    }

    class BakedPizza implements IPizza {
        private dough: Dough;
        private ingredients: Array<Ingredient>;

        // constructor
        constructor(pizza: PreparingPizza) {
            this.dough = pizza.getDough();
            this.ingredients = pizza.getIngredients();
        }

        // getters and setters

        public async bakePizza(): Promise<BakedPizza> {
            throw new PizzaAlreadyBakedException();
        }

        public async deliveryPizza(): Promise<DeliveringPizza> {
            const deliveryService = new DevelieryService();

            try {
                await deliveryService.send(this);

                return new DeliveringPizza(this);
            } catch (error) {
                throw error;
            }
        }

        public notifyDelivery(wasSuccessful) {
            throw new PizzaNotLeftForDeliveryYey();
        }
    }

    class DeliveringPizza implements IPizza {
        private dough: Dough;
        private ingredients: Array<Ingredient>;

        // constructor

        // getters and setters

        public async bakePizza(): Promise<BakedPizza> {
            throw new PizzaAlreadyBakedException();
        }

        public async deliveryPizza(): Promise<DeliveringPizza> {
            throw new PizzaAlreadyLeftForDeliveryException();
        }

        public notifyDelivery(wasSuccessful) {
            if(wasSuccessful) {
                return new DeliveredPizza(this);
            }
        }
    }

    class DeliveredPizza implements IPizza {
        private dough: Dough;
        private ingredients: Array<Ingredient>;

        // constructor

        // getters and setters

        public async bakePizza(): Promise<BakedPizza> {
            throw new PizzaAlreadyBakedException();
        }

        public async deliveryPizza(): Promise<DeliveringPizza> {
            throw new PizzaAlreadyLeftForDeliveryException();
        }

        public notifyDelivery(wasSuccessful) {
            throw new PizzaAlreadyDeliveredException();
        }
    }

With this implementation, we enforce the domain invariants with our type system through the interface and the Pizza union type. With it we gain less cyclomatic complexity since we don’t have so many branches in our code and, by design, we don’t allow for invalid States to happen. Besides that, each State carries an internal data, its travel bag. As such, these patterns are not excluding, but rather composable.

In the trend on the front-end what we are usually seeing is more a functional paradigm approach to the state machines. The entity, represented as a state machine, is now just a different data structure for each State that can be interpreted by the pure functions that implement the domain behaviors. These functions than can internally delegate its call to others functions specialized in each State. This separation of the state machine implementation of the behavior is natural as it follows the idiom for functional architectures.

What remains in both cases are the nature of the State as an entity’s representation. It works on its behalf and delimits the possible behaviors it can expose.

For example, a Pizza could never be in Baked and Delivered States at the same time. Now, it isn’t an implementation that guarantees that it is the design itself. Such abstractions, that models the domain, the heart of our product, couldn’t depend on implementation details to be valid, they must depend on the abstractions itself.

Side note: if you are versed in Algebraic Data Types you can see this as a Union Type with finite cardinality in the order of less then a dozen.

Evolving the abstraction

One could implement a State oriented design by using a simple enum, a proper state machine implementation or a more advanced concept, a statechart.

It is true that many domains can be modeled using the two first approaches to code a State, but sometimes we are faced with a high complexity scenario where this abstraction implementation would not scale with the development of the application.

For that reason that in 1987 David Harel proposed a new technique that expanded the grounds of the state machine definition, introducing tools like state hierarchy, parallelism, clustering, history, etc. He called it statecharts and it is a formalism that helps us scale the development of a State design, be implementing it thoroughly or just taking some tools.

I highly recommend reading more about statecharts as it can shift your mindset about how to approach problems.

Summary

Now we can differentiate state from State and avoid accidental complexity by using the right construct to model our domain. Its worth nothing if I don’t say that there is no silver bullet and these are tools to deliver a job. We have been experimenting with this design style on my team and it has been helpful since our scenario is really complex and fast pacing.

If you have any questions or want to discuss these and other topics more in deep please comment or you can reach me at Twitter, @caiorcferreira .

Thanks for reading!

References

Hosting an API in Firebase writen with Typescript and Nest

Caio Ferreira — Thu, 24 Aug 2017 01:03:40 +0000

This week I decided to test if Firebase is suited to a personal project and to do it I needed to discovery how to host an API writen in Typescript with Nest framework and, UOU, it was not an easy task! Because of this, I decided to write down the success path and registry some problems that I had.

As a disclaimer, I know it's a hell of a specificy project and not so many people (if anyone) will build with a stack like this, but if you want you can speed this process with this repository, which I write while making this post as a starter repo.
Now, even if you won't build something with this stack you can get some tips that I will leave at the end of this post and may help you bootstrapting a project with a different stack.

Pre-requisites

You will need the following packages installed:

Firebase Tools
@google-cloud/functions-emulator
WebPack
Nodemon

TIP 1

I had seen a lot of people having problems in running the firebase emulator, which is used to run the project locally . I suggest to install in the following way:

If you already have firebase tools installed, uninstall it with $ npm remove -g firebase-tools.
Then, install the package @google-cloud/functions-emulator with $ npm install -g @google-cloud/firebase-functions.
Then, install firebase tools again: $ npm install -g firebase-tools.

Building project scaffold

Create your project folder and enter it

$ mkdir nestjs-firebase
$ cd nestjs-firebase

Use the Firebase CLI to create the initial project

$ firebase init

When you run this command, you will be promped with a configuration menu, then, follow the steps bellow:

Mark the Functions and Hosting features with space and press enter.
Set your Firebase project
Enter n to not install npm dependencies yet
Press enter to use the default hosting directory as public.
Enter n to disable single-app configuration

Then, open your favorite code editor. I will be using Visual Studio Code. You will see the following folder structure:

Now enter functions folder with $ cd functions and update the package.json file inside it with the following.

{
  "name": "nestjs-firebase",
  "description": "NestJS app with Cloud Functions for Firebase",
  "scripts": {
    "build": "webpack",
    "serve": "webpack && firebase serve --only functions,hosting",
    "serve:dev": "nodemon -w *.ts --exec npm run serve",
    "deploy": "webpack && firebase deploy --only functions,hosting"
  },
  "dependencies": {
    "firebase-admin": "~4.2.1",
    "firebase-functions": "^0.5.9",
    "@nestjs/common": "*",
    "@nestjs/core": "*",
    "@nestjs/microservices": "*",
    "@nestjs/testing": "*",
    "@nestjs/websockets": "*",
    "@types/express": "^4.0.37",
    "express": "^4.15.4",
    "redis": "^2.7.1",
    "reflect-metadata": "^0.1.10",
    "rxjs": "^5.4.0",
    "typescript": "^2.3.2"
  },
  "devDependencies": {
    "@types/node": "^7.0.5",
    "ts-loader": "^2.3.3",
    "webpack-node-externals": "^1.6.0"
  },
  "private": true
}

Then, run $ npm install. After, create the src folder $ mkdir src and delete index.js. Next, create webpack.config.js $ > webpack.config.js with this.

// webpack.config.js
'use strict';

const nodeExternals = require('webpack-node-externals');

module.exports = {
    entry: './src/server.ts',
    output: {
        filename: 'index.js', // <-- Important
        libraryTarget: 'this' // <-- Important
    },
    target: 'node', // <-- Important
    module: {
        rules: [
            {
                test: /\.tsx?$/,
                loader: 'ts-loader',
                options: {
                    transpileOnly: true
                }
            }
        ]
    },
    resolve: {
        extensions: [ '.ts', '.tsx', '.js' ]
    },
    externals: [nodeExternals()] // <-- Important
};

Now, create the this structure inside src folder, with the following content.

// server.ts
import * as functions from 'firebase-functions';
import * as express from 'express';

import { NestFactory } from '@nestjs/core';
import { ApplicationModule } from './modules/app.module';

const server = express();
const app = NestFactory.create(ApplicationModule, server);

app.init();
exports.api = functions.https.onRequest(server);

// app.module.ts
import { Module } from '@nestjs/common';

@Module({})
export class ApplicationModule {}

Finally, update firebase.json like this.

{
  "hosting": {
    "public": "public",
    "rewrites": [{
      "source": "**",
      "function": "api"
    }],
    "ignore": [
      "firebase.json",
      "**/.*",
      "**/node_modules/**"
    ]
  }
}

And, if you want to use the root url in your API, you need to delete or rename public/index.html.

TIP 2

If you are on Linux (I don't know if this happens in others OS), you can have problems when running $ firebase serve. It is caused by permission problems and you can solve this in two ways:

Run the serve and deploy commands from firebase with sudo.
Change the owner of the firebase-tools folder in the global node_modules to your user.

I recommend the second way because, otherwise, when the nodemon restart the server it will not kill the previous process, so every restart the firebase tool will see the port 5000 as used and will start to listen in another process and this will pile up in your machine.

Conclusion

Now, you can develop your project and use run it with this commands:

To build the app $ npm run build
To run the server $ npm run serve
To run the server for development $ npm run serve:dev
To deploy it to Firebase $ npm run deploy

Wish I helped someone and, if you have questions or suggetions, please leave your comment.

Thank you!

Fonts

https://medium.com/netscape/firebase-cloud-functions-with-typescript-and-webpack-7781c882a05b

https://github.com/ultrasaurus/firebase-functions-typescript

https://stackoverflow.com/questions/44871075/redirect-firebase-hosting-root-to-a-cloud-function-is-not-working

Tips

Tip 1 - Firebase Emulator problem
Tip 2 - Firebase serve commands problem