DEV Community: rdoria1

My experience building backends with Booster Framework

rdoria1 — Wed, 16 Dec 2020 22:57:02 +0000

When developing a new backend application, there is bunch of things we need to think about:

Which language am I going to use?
Do I need any framework? Which one?
Do I know how to model my problem? (models, relations etc..)
Where am I going to deploy my application?
What kind of DB do I need? (SQL, noSQL etc..)
How will I expose my service?
Do I need a cloud provider? Which resources do I need?

Needless to say this is overwhelming, specially nowadays with the amount of options available to solve each problem.

The truth is that this is very time consuming. Looking for the best posible available option is almost always a very expensive decision and it's a trap.

Instead, I believe it's always better to have a default choice. The problem is, how do we choose one? Well, it is basically an option we have high trust and confidence it will just work, something that is reliable to solve the problem you have. Once we have something working we can improve from there.

The key point here is that it is delusional to think that at the beginning of a project we should test all the posible solutions to make the best possible decision from the very beginning.

I'm starting to think my default choice to develop backend applications can be the Booster Framework.

I'm mostly familiar with AWS when it comes to Cloud providers so let's imagine a regular development process

I open my IDE, start writing some code
Open AWS console (or change terraform if using infra as code) then change some param in Dynamo DB
Then I remember I need to change another params in my API gateway then I open another tab with the AWS console
Then I go back to my code and forget what I was doing

Do you see where I'm going? This constant context switching between Infrastructure and Business logic (the code in my IDE) can be expensive and overwhelming specially at the beginning of a project or when crafting a prototype.

I needed to build an account backend application quickly, a simple backend to add and subtract values, get balances etc.. so I decided to do it with Booster, I was done in like 2 hours. I didn't need to think about Infrastructure. Everything is deployed seamlessly, no context switching, no AWS console tabs, no Infra to handle. In this sense, it is very similar to the Serverless experience we get when writing a Lambda function in AWS for instance, but this even goes beyond that.

Give it a try https://www.booster.cloud/

Redefining Event Sourcing

rdoria1 — Fri, 30 Oct 2020 17:40:32 +0000

Event Sourcing is complex and involves multiple concepts, patterns, and architectures. At The Agile Monkeys, we have been working with it for several years and we would like to share our vision of the main concepts around Event Sourcing.

Events

Things that have happened (yes in the past).

e.g. a customer has placed an order.

State

Something that evolves and changes over time with a finite number of possible values.

e.g. A SKU can have multiple states (in stock, out of stock, soon to be available, discontinued, etc.)

Events + State

Events = Things that have happened that changed some state.

e.g. Customer placed an order and therefore the order is in the PLACED status and the inventory of the item decreased by 1 etc.

Event Storage

Permanent Storage where events are persisted.

e.g. Can be a DB (NoSQL, SQL…) or maybe object storage (S3...)

Event Sourcing Contract: 3 Main Concepts

Store the full history of events (aka state changes) in the Event Storage: of course, we mean only the events we intended to store, the ones we designed to persist.
Events are chronologically ordered: This allows to reconstruct the state of the system at any given point in time.
Immutable events: it’s an append-only system (no update nor removal).

Data Removal

Because of data privacy policies (like GDPR in Europe), there are some cases where users want to delete their data. Allowing this would break the Event Sourcing Contract we just described (immutable events).

How can we support this use case?

We can encrypt the data. When users request to delete their data, we are going to delete the encryption key.

This design solves both problems:

By dumping the key we are making data unusable which fulfills the purpose of deleting the data from the user perspective.
We are not opening unwanted doors by actually deleting the data (no permissions, no repo with delete methods, etc.) so we are still respecting the contract we described.

Event Sourcing usually works with

Domain-Driven Design
CQRS
Event-Driven Architecture
MicroServices

Event sourcing excels when used together with the 4 concepts above.

Domain-Driven Design

Uses conceptual maps (called object models) of the domain (easily maps to an actual business concept/idea) that incorporates both behavior and data.

Creates common concepts/vocabulary to easily communicate with other technical teams and business people.

e.g In the e-commerce business, a must-have object model would be an order (defined by its ID, customer ID, date, status, etc..)

1. Domain Event

Captures something that happened that changed the state of a domain model.

e.g. A person changed their address, which changes the Address domain model given an AddressChanged event.

We can then use the knowledge gained from the changes to do something useful with it.

e.g. Since the person moved to a new place, we can send them reminders to update their address for all their bills.

2. Event Sourcing + DDD

Stores the full history of domain events in the Event Storage.

Event Sourcing + Event-Driven Architecture

Misconception: They are not the same!

Event-Driven Architecture: System which is based on components communicating mainly or exclusively through notification events.

When talking about “event” in event sourcing we are actually referring to the state change and not a “notification” as in the event-driven architecture (queues, bus, streams, async communication in general).

domain event != notification event

Event Sourcing + Microservices

When working with microservices architecture, we need to notify multiple services about state changes (domain events) so they can react to them.

Usually, this is done with notification events (so we are using an event-driven architecture).

In this case, notification events are domain events.

e.g. Domain events persisted in the Event Storage. In an event-driven architecture, we need to notify Service A and B about them. In this case, we can call them notification events, as we decided to use async communication for this purpose.

Reading Data

Don’t use your Event Storage as a Read Model.

You might be tempted to query your Event storage to find the latest state for a given event.
This can be slow for large systems. It might work for very small systems but reconstructing the full history to find the latest state for regular usage of your system doesn’t sound like a very good idea.

1. CQRS - Read Model

A pattern that requires having separate classes for reading and writing data.

This allows us to have different models to read and write data.
This also means the repositories (and therefore the DBs) can be different for reading and writing data. For instance, we can write to a relational DB (the writing part is the Event storage) and then read from a NoSQL DB.

CQRS generally has 2 main concepts: commands (write) and queries (read).

A command can return a value: Most likely the returned value will have to do with operation confirmation or identifiers and nothing to do with state changes and this operation is expected to be asynchronous. A domain event will be put into a queue (stream etc..) and be processed at a later stage.
A query does not change any state, its only goal is to read.

This separation makes the following scenario possible.

Let’s say we decide our Event Storage will be a NoSQL DB. Our Read Models will live in our Services and they will be separate DBs, they can be relational DBs.
The Event Storage will remain our source of truth but each Read Model will be designed to render the data as needed per each service. That’s its only goal.

Snapshotting

Since we are storing everything in the Event storage, one very common question is “How do we avoid reading all the data to find the latest state for a given domain model?” The solution is to create snapshots from time to time. Snapshots are actually part of the Event Storage itself. Let’s give an example to make it clearer.

One of the classic examples of event sourcing is the bank card we use to perform withdrawal and credit operations. Domain events will persist these operations. In order to know the current balance of the account, we need to reconstruct all the operations from the beginning. Snapshotting here can be used to have the balance computed every 5 operations. In the worst case, we would need to get the latest snapshot + 4 domain events to know the current balance, thus decreasing the number of reads significantly.

1. Snapshotting vs Read Model: Replay events

If Snapshotting is used to reconstruct the latest state for a given domain model, why would we use Read Models?

When working with a micro-service architecture they are used for different purposes.

Imagine adding a new micro-service, therefore a new ReadModel DB. We need to reconstruct the latest history of domain events from the Event Storage so we’ll use the Replay events feature to achieve this.

Replay Events is a feature of event sourcing and is a byproduct of 2 of the main concepts we defined: full history of domain events + ordered domain events. We can replay the full history of what happened and get the latest state for a given domain model.

This can be a very time-consuming operation if we are talking about millions of domain events. The fastest way to do this is using Snapshotting.

In this case we can seamlessly add a new microservice. We are using snapshots for a different purpose as we are still using read models to read data from a given Microservice and not the snapshot from the Event storage.

The same situation applies if we decide to completely dump a Read Model and change it for a new one. Our source of truth remains the Event Storage and we just need to replay all the events and create our new read model.

Benefits of Event-Sourcing

Data-driven business
When starting a new business/new product etc.. we just don’t know how much our data is worth. Even if the goal is clear at that point in time, it’s always a better idea to keep all the data to possibly use it at some point in the future (storing data is cheap now) Example: Let’s imagine we are creating a new e-commerce site and we are in charge of specifically adding features and then the checkout process to cart items. In this case, what matters the most is the final state of my cart before the user places the order. The fact that the user has added and then removed items doesn’t really affect the checkout process and the correct functioning of the website. Nevertheless, we should keep the full event history of adding/removing items from the cart. Even if not useful now, it can help us analyze the customer behavior and see why they added and deleted those items.
Audit/Logging
By reliably persisting the full history of events we will be able to debug our applications more easily by finding the full history of events for a given domain model.
Since it’s an append-only system, we never lose data. We can’t delete any data.
High-Security Standard naturally by design: We can’t update or remove data. If a system gets hacked, data could still be appended meaning the current state of a given domain model would change. In any case, not losing any data is a great benefit.
Great fit for analytics: Since we have the full history of everything that happened, we analyze the past and use data to drive the business.

Disadvantages of Event-Sourcing

Mental shift for developers: There are a bunch of concepts that need to be mastered here, event sourcing, DDD, CQRS, event-driven architecture etc. It can be overwhelming.
It should not be used for all scenarios. Like any other pattern, it doesn’t make sense in some cases.
Unless we use the Event Storage for the read models (which is definitely not the best practice or not even possible in some cases), we are introducing data redundancy which can lead to potential inconsistencies and also eventual consistency.

This is how we work with event sourcing in Booster.

Saga Patterns by Example

rdoria1 — Tue, 11 Aug 2020 15:49:26 +0000

Moving from Monolithic application to microservices

A very common scenario nowadays is migrating Monolith applications to a new Microservice Architecture.

We are not going to focus on explaining these architectures in this article. Instead, we are focusing on a very specific issue when it comes to dealing with Microservices: Data Consistency.

With MicroServices we are usually going to follow a One Database per service pattern. Imagine business transactions spanning across multiple services, meaning one transaction completion or failure will impact subsequent transactions. We need to maintain data consistency.

We can’t use ACID local transactions, we need to solve this issue differently.

Saga

We’ll need Saga: Sequence of local transactions.
Each local transaction will

Trigger the associated action (This will generally be a DB update, but can also be something like sending an email to a customer etc..)
Publish a message/event to trigger the next local transaction

If the local transaction fails, there will be a series of compensating transactions to undo preceding local transactions. Undoing things doesn’t necessarily mean deleting data in case the action is a DB transaction, but rather updating some state to FAILED.

Enough theory, let’s get to a specific business which we will use to explore the specifics of what we described and to understand Saga.

Enrollment Credit Card

Let’s imagine a Bank has an Enrollment to New Credit Card process. Whenever a customer wants to ask for a new Credit Card, they will enter a bunch of data through some Forms.

This data will get to a Card Service that will create the data needed to create Digital Credit Card on the fly (in the meantime the actual physical card gets processed and sent)
For this purpose it will first check the provided customer’s data is accurate and can be validated through the Identity Service (maybe this service will use a third party service like Equifax to check the customer’s identity)
Then, the Calculation Service will determine the credit limit for the new credit card based on internal calculations and other factors
Once everything is successfully completed the Card Service can finally return all the data for the new Digital Credit Card to the customer that was computed in the beginning.

Choreography Pattern

Saga comes into 2 different flavors. First, we’ll explore the simplest and most direct one: The Choreography pattern.

Use Case

The best way to describe this is to show the diagrams of the sequences of transactions happening just after the customer has entered all the required data in the form.

The input data will get to the Card Service that will create a new record in the Card DB. Note we have a status value, which will play a key role here. For now it’s set with VERIFYING value.

Once the record is successfully persisted, a new event will be streamed by Card Service and consumed by Identity Service.

Identity service will consume the event, then it will execute its internal logic and persist a new record in the Identity DB. The isVerified attribute will be a boolean stating if the identity was indeed verified or not.

Once the record is successfully persisted, a new event will be streamed and will be consumed by Card Service and Calculation Service.

The event consumed by Card Service will actually trigger an update of the record persisted in the first step. The status value will be updated to IDENTITY_VERIFIED. We are notifying the main Service that the identity was verified.

The event will also be consumed by Calculation Service. This one will execute some internal logic so that the goal is to calculate the amount Limit for the potential new Credit Card and persist this new data to a new record in the Amount DB.

Once the record is successfully persisted a new event with the amount limit data will be streamed to be consumed by the Card Service.

Finally, once the event is consumed by the Card Service we will again update the initial record with its final state. The amountLimit value will be updated and the status will change to VERIFIED. This status value means the new Credit Card Data is ready to be returned to the customer so they can start using their new Digital Credit Card.

Implementation details

Let’s imagine how we would implement this in the cloud, more specifically with Azure. We are showing just a small part of the implementation as we can extrapolate this to the other services.

As we can see from the diagram above, the services are implemented as Spring boot services. Whenever a new request gets to the Card Service, a new record is persisted in a Cosmos DB Card container.

The interesting part of this architecture comes when integrating Change Feed. We won’t get into too many details on how it works (the official documentation is great) but in a nutshell:

Enabled by default
It exposes the Cosmos DB logs to the end user so you can consume it and integrate it with other resources to be notified about any new insertion or update.
Sorted: Comes in the order of modification
Can be integrated with a number of resources

The easiest way to consume the change feed is the integration with Azure Functions, which also comes in handy with our purpose.

Once a new record in the Card Container is persisted, the Change feed will trigger a new Azure function with an event containing the new record. The azure function will receive this event and will do the following:

Deserialize the content so we get the customerId
Find the record associated with the given “customerId” in the Input Container. Actually, the Change Feed event received will only contain the data persisted in the Card Container. This is not enough for the next step which is the Identity Service. We will need identity data that was part of the original request. In this case we are fetching this data by directly reading a given Input container, but it might be something else such as an HTTP request from another Service.
HTTP request to the Identity Service
Wait for the Response and depending on the Response Code and Body, stream a new event through Event Hub to be consumed by the Card Service which will transition the Card Container record from VERIFYING to IDENTITY_VERIFIED.

The implementation of the Azure function that consumes the change feed is really simple. Here is a skeleton of this function implemented in Java.

package com.function;

import com.azure.cosmos.*;
import com.azure.cosmos.models.CosmosItemResponse;
import com.azure.cosmos.models.PartitionKey;
import com.microsoft.azure.functions.annotation.*;
import com.microsoft.azure.functions.*;
import com.model.Input;
import org.json.JSONObject;

import java.util.Collections;

/**
 * Azure Function with Cosmo DB Card container Trigger from Change Feed.
 */
public class CardChangeFeedTrigger {

    private CosmosContainer container;

    private static final String databaseId = "Enrollments";
    private static final String containerId = "Input";

    @FunctionName("CardChangeFeedTrigger")
    public void run(
        @CosmosDBTrigger(name = "items",
        databaseName = "Enrollments", collectionName = "Card",
        createLeaseCollectionIfNotExists = true,
        connectionStringSetting = "AzureCosmosDBConnection") String[] items,
        final ExecutionContext context) {

        setupCosmoDB();

        for (String item : items) {            

            // Read customerId from Change Feed.
            JSONObject jsonObject = new JSONObject(item);
            String customerId = (String)jsonObject.get("customerId");


            // Read Input from Cosmo Container
            CosmosItemResponse<Input> input = container.readItem(customerId, new PartitionKey(customerId), Input.class);
            context.getLogger().info("New item received " + input.getItem());

            // Now we do an HTTP CALL to the Identity Service with data customerID and data from Input Model

            // Based on the Answer 
            // 1. "IDENTITY_VERIFIED" event back to CardService if HTTP call succeeded.
            // 1. "IDENTITY_FAILED" event back to CardService if HTTP call failed.
            // 2. "IDENTITY_NOT_VERIFIED" event back to CardService if Identity not verified.
        }

    }

    private void setupCosmoDB() {
        CosmosClient client = new CosmosClientBuilder()
                .endpoint(System.getenv("COSMO_DB_HOST"))
                .key(System.getenv("COSMO_DB_MASTER_KEY"))
                .preferredRegions(Collections.singletonList(System.getenv("REGION")))
                .consistencyLevel(ConsistencyLevel.EVENTUAL)
                .buildClient();
        CosmosDatabase database = client.getDatabase(databaseId);
        container = database.getContainer(containerId);
    }
}

Advantages

Easiest way to implement SAGA
Each transaction is independent
It’s ideal when not dealing with many local transactions (e.g. no more than 4)

Disadvantages

It is difficult to track and maintain this logic if several transactions and services are involved. In the implementation example we showed above let’s analyze how this would work:
We introduce a new Azure Function to consume the change feed. In terms of ownership it could be tricky to define who should “own” this new function. Does it fall under the Card Service? Or under the Identity Service?
Also, as we saw before, the event sent through Event Hub needs to be consumed by the Card Service meaning we’ll need to add a new logic in Card Service so it becomes a consumer of this message. Imagine having this same pattern with multiple services, N services would need to send events to Card Service, it would become difficult to maintain at some point.

Orchestration Pattern

The second approach of SAGA comes with an Orchestrator. In this pattern, the coordination of all the transactions is fully abstracted into a separate service, an Orchestrator. We’ll use the same example as the one for Choreography pattern but we are adding a new Service called Verification Service.

Implementation details

This Verification Service will actually internally check if the customer is eligible for a new Card and check for any outstanding loan or any potential red Flag.

We added this service so we have parallels and sequential operations:

Once a new request is generated it will get to Card Service
Card Service will trigger the Orchestrator which will be in charge of coordinating the rest of the transactions
We have 2 verifications that need to happen and they will both be triggered in parallel, the Verification and the Identity Service.
Once both are successfully completed, the orchestrator will trigger the Calculation Service to compute the amountLimit.

We’ll use durable functions to implement the Orchestrator.

Durable functions

The documentation on Durable functions is very useful, but in a nutshell:

Extension of Azure functions
Serverless State machines - used for Long-running orchestrations
It simplifies complex transactions and coordination
It maintains a local state
It’s code only (No json schemas or config files like other orchestration systems)
Currently available in Python, JavaScript and C#

The following image describes the structure of the project to create an orchestrator. We implemented the project in Javascript.

Under the same project (that we usually create with VSCode) we’ll have 3 different kinds of Azure Functions to create a whole Orchestrator workflow. Each function has its own folder.

Orchestrator Function

The EnrollmentCreditCardOrchestrator folder will have the code of the orchestrator function itself. As seen it only contains 2 files:

function.json
index.js

The json file is a config file and as seen it’s a very small file specifying the function defined in the index.js file is an Orchestration Function (for further details, please check the documentation which is great).

The Orchestration function looks like the following:

const df = require("durable-functions");

module.exports = df.orchestrator(function* (context) {
    const request = context.df.getInput();
    context.log("Received New enrollment request. person: " + (request.person ? JSON.stringify(request.person) : undefined) + 
    " || card: " + (request.card ? JSON.stringify(request.card) : undefined) + ".");

    // Verify the request has all the mandatory fields
    verifyRequest(request);

    const tasks = [];

    const verificationTask = context.df.callActivity("VerificationActivity", request.person)
    tasks.push(verificationTask);

    const identityTask = context.df.callActivity("IdentityActivity", request.person);
    tasks.push(identityTask);

    // Wait for verificationTask and identityTask to finish
    yield context.df.Task.all(tasks);

    context.log("Verification Task " + JSON.stringify(verificationTask.result));
    context.log("Identity Task " + JSON.stringify(identityTask.result));

    const calculationRequest = {
        "creditData": verificationTask.result,
        "identity": identityTask.result 
    }

    const calculationCreditCard = yield context.df.callActivity("CalculationActivity", calculationRequest);

    return calculationCreditCard;
});

function verifyRequest(request) {
    if (!request) {
        throw new Error("A request object is required.");
    }
    if (!request.person.id) {
        throw new Error("Gobernment ID is required.");
    }
    if (!request.person.firstName) {
        throw new Error("A first name is required.");
    }
    if (!request.person.lastName) {
        throw new Error("A last name is required.");
    }
    if (!request.person.workType) {
        throw new Error("A type of work is required.");
    }
    if (!request.person.workPlace) {
        throw new Error("A work place is required.");
    }
    if (!request.person.salary) {
        throw new Error("A salary is required.");
    }
    if (!request.card.type) {
        throw new Error("A Card type is required.");
    }
    if (!request.card.brand) {
        throw new Error("A Cart brand is required.");
    }
}

The way we call the actual services is through Activities and more precisely through the context.df.callActitity function which is another kind of function we’ll define later on.

The context object is the one containing the request when triggering the orchestrator.

“VerificationActivity” and “IdentityActivity” are triggered in parallel, whereas “CalculationActivity” sequentially. How do we do this?

Verification and Identity activities are called without the yield word. This means they run in parallel.
Line 18 specifies we want to wait for both activities to be completed before moving forward
Once both successfully return with their respective payload, we actually call the Calculation activity synchronously, we get the result from it and then return the final result calculationCreditCard to the user who triggered the Orchestration.

Activity Function

As seen in the project structure screenshot, each activity function will have its own folder. Each folder will have its config file specifying the function in index.js is an activity function.

The index.js will look something like the following:

const axios = require("axios");
const identityURL = process.env["IDENTITY_SERVICE_URL"]
const CircularJSON = require('circular-json');

module.exports = async function (context, request) {
    try {
        const result = await axios.post(identityURL, request)
        context.log("body returned..." + CircularJSON.stringify(result))

        return result.data;
    } catch (err) {
        context.log(`Identity Service encountered an error: ${err}`);
        throw new Error(err);
    }
}

In this specific case we imagined the services are already implemented somewhere else so the only goal of the activity function would be to use some HTTP library (like axios in this case) to make the HTTP request to them. In the practice if we are dealing with a new brand service you can also implement the actual logic of this service in the Activity Function.

The activity functions and orchestration functions live in different VMs.

HTTP Trigger Function

This is the last kind of function we need, the only purpose of this function is to actually expose the orchestrator as an HTTP endpoint so it can be called from any client easily.

const df = require("durable-functions");

module.exports = async function (context, req) {
    const client = df.getClient(context);
    const instanceId = await client.startNew(req.params.functionName, undefined, req.body);

    context.log(`Started orchestration with ID = '${instanceId}'.`);

    return client.createCheckStatusResponse(context.bindingData.req, instanceId);
};

As a result, when triggering the orchestrator we’ll get a bunch of URLs as a result (Check documentation)

When calling the statusQueryGetUri we will get the current status of the orchestrator

Advantages

Easy to maintain the whole flow in a single place
Avoid cyclic dependencies, again everything happens in a single place.
If designing a smart orchestrator, the complexity of adding new transactions remains linear.

Disadvantages

New Service to maintain: In distributed teams pattern, adding this new service can be tricky to decide which team should own it. It contains logic and domains from all the services.
It’s also challenging to know a little bit of the logic of all the services, or at the least clearly understanding the return payloads from services and understanding how to react if something fails and how it affects all the services.