DEV Community: Luis Felipe Ciochetta

Idempotence - what is and how to implement

Luis Felipe Ciochetta — Tue, 14 Feb 2023 12:24:52 +0000

In the context of development, idempotence means that calling a certain function once or multiple times should have the same effect on the system's state.

Being idempotent does not mean the return value is always the same, but that the function, if it has changed the system state, does not make any further changes if it is immediately called again with the same parameters.

So, for example, a function that returns the balance of an account is idempotent, even if it returns different values, because other functions may have changed the system state between one call and another.

Idempotence in our APIs

When using our APIs, to ensure idempotency in calls, we use the pattern known as Idempotence-Key, which means that when making a call to our API, it is important to specify an ID that your system understands as corresponding to the operation.

The way we implement Idempotence-Key in our system is through the a property named "correlationID" inside the body.

For example:

You want to use our API to create a charge

In your system, you should have a table/collection that represents the creation of the charge, create a document and get its ID

In the call to our API, add a correlationID property with that ID.

When using our webhooks, it is necessary to ensure that the route that handles the webhooks is idempotent, as it is common practice to resend webhooks to ensure that at least one of them has arrived and been processed. At the end of this document, there are examples of how you can implement this.

Why is idempotence relevant?

A recurring problem in event-driven systems is duplicating events, which can occur for many reasons, such as communication failures, parallel invocations of the functions that generate the events, etc.

Resilient systems are built taking into consideration that these problems can and will frequently happen, and preparing adequately so that communication failures do not cause a bad internal state.

So, the main importance of idempotence is to make your system resilient and reliable.

When should a function be idempotent?

As with everything in development, it heavily depends on the project goal and business rules.

However, there are some patterns:

REST APIs

PUT routes

All routes with the PUT verb, which make changes to entities.

For example: A route that serves to change the address of a client.

As long as the new address is the same, it shouldn't matter to the system state how many times that route is called.

GET routes

All routes with the GET verb, which are meant to read information.

For example: A route that serves to read the current balance of a bank account.

Even if the balance changes over time, it is safe to call this route multiple times without causing changes to the system state.

DELETE routes

All routes with the DELETE verb, which are meant to delete entities.

For example: A route that serves to delete a charge.

Once called, the function will delete the bank account, subsequent calls should not delete other accounts, but return a message saying that this account has already been deleted.

Implementing Idempotence

These are some patterns for implementing idempotence:

GetOrCreate Pattern

The idea behind this pattern is to check if the entity that the function is trying to create already has an equivalent created in the database.

Assuming this function:

const createCustomerBankAccount = function(customer, params){

    // Model validation
    ...

    // Creating the account in the database
    db.customerBankAccount.create({ ...params, customer: customer.id })
}

Applying this pattern, it can be rewritten as follows:

const getOrCreateCustomerBankAccount = function(customer, params){

    // Before making creation validations, we can look for the most important parameters in our database and return an already existing account
    const existingCustomerBankAccount = db.customerBankAccount.get({
        customer: customer.id,
        bankId: params.bankId
    })

    if (existingCustomerBankAccount){
        return existingCustomerBankAccount;
    }

    // Model validation
    ...

    // Creating the account in the database
    db.customerBankAccount.create({ ...params, customer: customer.id })
}

Call History Pattern

The idea behind this pattern is to save all calls made to the route that deals with events and upon receiving a new call, check if there is not a similar call that was made previously.

Assuming this function:

const createCustomerBankAccount = function(customer, params){

    // Model validation
    ...

    // Creating the account in the database
    db.customerBankAccount.create({ ...params, customer: customer.id })
}

Applying this pattern, it can be rewritten as follows:

const createCustomerBankAccount = function(customer, params){

    const newRequest = { customer, ...params }

    const existingRequest = db.events.get(newRequest);

    if (existingRequest){
        return;
    }

    // Model validation
    ...

    // Creating the account in the database
    db.customerBankAccount.create({ ...params, customer: customer.id })

    // Saving the event for future calls
    db.events.create({...newRequest})
}

Or, we can use hashing to save storage:

const createCustomerBankAccount = function(customer, params){

    const newRequestHash = MD5(JSON.stringify({ customer, ...params }));

    const existingRequest = db.events.get(hash: newRequestHash);

    if (existingRequest){
        return;
    }

    // Model validation
    ...

    // Creating the account in the database
    db.customerBankAccount.create({ ...params, customer: customer.id })

    db.events.create({ hash: newRequestHash})
}

Idempotence-Key Pattern

The idea behind this pattern is to use a key to represent an operation, and upon receiving a duplicate key, not repeat the operation.

We use a similar pattern to this in our APIs through the correlationID property.

Assuming this function:

const createCustomerBankAccount = function(customer, params){

    // Model validation
    ...

    // Creating the account in the database
    db.customerBankAccount.create({ ...params, customer: customer.id })
}

Applying this pattern, it can be rewritten as follows:

const createCustomerBankAccount = function(customer, params){

    const { idempotenceKey } = params;

    // Check if the user sent an idempotence key
    if (idempotenceKey){

        const existingBankAccount = db.customerBankAccount.get({
            idempotenceKey
        })

        // Checking if there is any registered account with this key and returning it if we find
        if (existingBankAccount){
            return existingBankAccount;
        }

    }


    // Model validation
    ...

    // Creating the account in the database
    db.customerBankAccount.create({ ...params, customer: customer.id });
}

If you want to work in a startup in its early stages, This is your chance. Apply today!

Woovi is a Startup that enables shoppers to pay as they please. To make this possible, Woovi provides instant payment solutions for merchants to accept orders.

If you want to work with us, we are hiring!

Learning Observability (w/ OpenTelemetry and Jaeger)

Luis Felipe Ciochetta — Thu, 16 Dec 2021 23:41:03 +0000

Hey folks, there has been a long time since I have last posted anything here and I wanted to start writing again.

A quick disclaimer, as always, I am learning as I write this article, so I may get something wrong, I would love some feedback if you catch anything

My idea here is to make some projects to teach myself how to create highly observable services

First Project: Fibonacci

To start, I've decided to follow the golang OpenTelemetry tutorial.

And used the jaeger example to create the tracer provider.

The result for this project is a simple program that calculates Fibonacci based on the input you provide and sends the data about each individual operation to jaeger

It looks like this in the jaeger UI:

The important concepts behind this project are traces, spans, and tracer providers.

Traces

Traces are an abstraction for an operation or call inside your system, and they are composed by spans.

Traces can track the life-cycle of a operation through many processes (I will probably get there in the next project).

In the case of this program, this is the trace strucutre:

Specifically in golang, the context library is used to create this structure, so you will need a context for each separate trace.

Spans

Spans are the subdivisions of a trace, the way I see it, they are meant to represent each function execution inside a given trace.

Each span can contain attributes, that are key/value pairs that should represent the state of that function when they're called.

Tracer Provider

Tracer providers are configurations for the project about where to send the tracing data and how to send it.

In the case of this project, that would be jaeger.

Second Project: Square

The second project was customized, I did not write the steps for it in as much detail as these documentations, but it's fairly simple code (the link for the repositories is at the end of the post).

The idea was to create two services and a trace that would have spans in both services.

So I have created this two services:

API: has an http server that receives requests and calls the other service
Square: squares the number sent to it

This is how a request looks like for the user >

And this is how it looks on jaeger >

And the structure for the trace >

And as a nice side effect, you also get this graph that shows the dependencies for your services and the number of requests sent from a service to another >

And for this second project there was a single new relevant concept: propagators

Propagators

Propagators are the tool used to share context;

The metadata is injected in the headers for the request, then the server receiving the request extracts it and adds to it's own spans.

The result is this graphic where you know how much time the trace spent in each service and in each operation.

Final Words

So, I covered the basics of observability, and I can see how this would be relevant on onboarding members to your team and also to debugging

Next I will probably try to learn other observability tools like prometheus and grafana

Repositories

ciochetta / learn-jaeger

repository for learning jaeger

ciochetta / go-square

square service for my learning jaeger article

Useful links

https://opentelemetry.io/docs/concepts/what-is-opentelemetry/

https://www.jaegertracing.io/docs/1.29/getting-started/

https://github.com/open-telemetry/opentelemetry-go

https://github.com/open-telemetry/opentelemetry-go/blob/main/example/jaeger/main.go

Book notes: Hooked - Chapter 1

Luis Felipe Ciochetta — Wed, 07 Apr 2021 11:32:43 +0000

Chapter 1 - The Habit Zone

This chapter is a 22 pages long sales pitch. I was really annoyed by it. If you plan to extract value from reading this book, you can just skip chapter one.

Habits are formed from solving our needs, our brain starts to associate actions with pleasant experiences (solving problems) and this might become a behavior.
Habit-making products increase the amount of cash you can generate from a single customer.
People are less sensitive to the price of products that are part of their day-to-day life.
Leaving a product you have formed a habit around is very difficult.

^ Say you want to stop using Instagram, then you would have to think of another way to store and share your photos with your friends that also use Instagram and think of new ways of contacting them. So Instagram, for being a habit-forming product has you hooked to it.

To use habit-forming as a strategy you have to focus on increasing the frequency of use of your product and the value you generate each time. Some behaviors are bound to not repeat as much as others, then you would need to focus more on delivering value.

Book notes: Hooked - Introduction

Luis Felipe Ciochetta — Mon, 05 Apr 2021 11:40:10 +0000

Hey folks

I've decided I should probably read some of the books I've bought in the last years before I bought any new, so I am starting today the reading of "Hooked: how to build habit-forming products".

As far as I was told, this is a book mostly about design, not coding or anything related.

Introduction

Habits are behaviors we have but don't think much about.
Companies should focus on creating habit-forming products.

The hook model is made of four parts:

1 - trigger: what starts the action, can be divided into external (ads, emails etc) and internal (psychological needs)
2 - action: the thing people do to interact with a product (click a button). Actions should be easy to do.
3 - variable reward: the result of the action. This result must change in order to get people hooked, the reward must not be predictable.
4 - investment: the user puts something into the product (money, time, data, etc) and that improves his experience for his next hook cycle.

Learning Event-Driven Architecture

Luis Felipe Ciochetta — Mon, 15 Mar 2021 12:49:34 +0000

Hello folks!

I've been studying event-driven architecture for about a month now, and I wanted to write about it so I can better fix the knowledge in my head.

Quick disclaimer

I am using Apache Kafka in my studies because that is what we use in the company I work with, but I am sure that most concepts I've learned can be applied to other tools.

Also, as I have mentioned, I am a beginner in this subject, I could be very wrong about some things, and if I am, I would love some feedback about it.

What

Event-driven architecture is a way of thinking about software architecture where things (events) happen in the business and our system reacts to it.

This architecture has three main components:

event emitters (aka producers, aka publishers) that are responsible for capturing and sending events to the system.
event listeners (aka consumers, aka subscribers) that react to the events applying business logic.
brokers which are the glue that receives the events from the emitters and sends them to the listeners.

As far as I know, event-driven architecture is always based on microservices, meaning, each of these components is a separate service.

Why

Event-driven architecture solves the problem of having a huge volume of data that needs to be asynchronously processed while supporting future change in the system.

Most of the examples I've found talk about payment processing, where you can have N orders being processed by different services so the information can come out of order and have multiple outcomes that need to be handled in a unique way.

Maybe you want to have systems that both notify the user about a failed payment and send information to a business dashboard;

You achieve this with both monolithic and regular microservices architectures, however, this will lead to more dependency between your modules or services.

But if you have an event-driven architecture you could easily just write a new service that handles this exact same data (let's say you also need a client dashboard now) without interfering with how the other two systems work.

How

I will grossly over-simplify the process of implementing this architecture but I intend to make other posts on the subject explaining it better.

1 - Pick a message software.

There is plenty of message queue software out there, do some research on their pros and cons.

Pick the one that makes more sense to your needs.

2 - Map your business needs and business events add them to your architecture.

Like in the example in the previous section, you have both events and business needs, add them to your design. so it should look like this >

Item purchase > ? > Notifiying customer

You will need to do something like this process for each of your business needs. In this case, the business need is to have customers notified of their successful purchases.

3 - Design a path in which the data is transformed from events to fulfilling your needs.

Now is time to design the services that you will need to make this event into a notification.

In this case, we should need only two more steps, one to validate the order and one to notify the customer.

So our project would look something like this >

Item purchase event
Order validation service acts upon this event, generating another event for validated order (or invalid).
Notifying services acts upon this event and send the notification to the user.

4 - Set communication interfaces for the services.

For each of these events, we need a (preferably documented) data structure that represents what happened, so other services can later act upon the events.

5 - Done, start coding.

The architecture phase is done, you can move to develop the system.

Once you finished doing all of this work, you will most likely have a lot of new ideas on how to improve your architecture, and that's why you should always keep in mind that can and will occur, your systems and data structures should be easy to alter.

Book notes: Clean Code - Chapter 9

Luis Felipe Ciochetta — Thu, 11 Mar 2021 01:03:39 +0000

Chapter 9 - Unit Tests

We should try to write clean tests, meaning that we should make the tests as easy to maintain as the functions they are testing. If the functions are changing as business needs evolve, so should the tests.

Test code is just as important as production code. It is not a second-class citizen. It requires thought, design, and care. It must be kept as clean as production code.

Tests enable bold changes to the code base.

^ This is kinda obvious, but I'm keeping it here to remember myself.

Clean tests should be as elegant as clean production code. They should be designed to be read.
Tests should follow the BUILD-OPERATE-CHECK pattern.
There are things you can do in a testing environment that you cannot in production. But they are not related to code cleanliness, but to resource usage.
We should try to minimize the assert statements per test.
We should have one concept per test.

F.I.R.S.T. rules:

Fast: Tests should run fast, so people always want to run them.
Independent: You should be able to run any test independently.
Repeatable: Tests should run in any environment.
Self-validating: Tests should have a boolean output, you should not have to think or investigate to know if it passed.
Timely: Tests should be written just before production code.

Book notes: Clean Code - Chapter 8

Luis Felipe Ciochetta — Tue, 09 Mar 2021 10:25:07 +0000

Chapter 8 - Boundaries

We can and should keep third party code abstracted so we have control over which properties and methods are accessible
We should create tests for third-party code so we create a better understanding of it and how we can use it on our application. This would also make us sure that if these libraries update in the future, that portion we are using in our code did not broke.

^ I don't think this is very practical, most libraries are not that easy to write relevant tests on. I can see why this is good and will try to do this in a future project but I am not optimistic.

When writing code that will integrate with code that doesn't yet exists and have no interface, we can create a class that has the interface we wish we had to mock that integration. Once we have the real code, we can adapt this class and abstract the implementation of the newly written module.

Book notes: Clean Code - Chapter 7

Luis Felipe Ciochetta — Mon, 08 Mar 2021 23:58:33 +0000

Chapter 7 - Error Handling

Error handling is important, but if it obscures logic, it’s wrong.

Exceptions are better than returning error codes. When using exceptions we can just follow the logic path until the end, and if there is an exception it will be caught later.

^ So, he mentions this in a previous chapter, I have also included this here just so I can find it later

When writing our tests, we should try to create tests that generate exceptions. This will force us to put all the correct logic inside the try block for our function.
We should provide meaningful context to our exceptions
We should not return null if something went wrong, if we return null, then we will have to check for null in every interaction

^ Ouch, can't count on my fingers the amount of times I wrote a function that returns null last month

Passing null as an argument is even worst than returning it.

Book notes: Clean Code - Chapter 6

Luis Felipe Ciochetta — Mon, 08 Mar 2021 00:36:25 +0000

Chapter 6 - Objects and Data Structures

Keeping implementation of functions private is a matter of abstraction. The user of a class does not need to know the details on how the command issued will be taken care of.
The worst option is to just create getters and setters, we should think about the best way to expose the data and how it will be used.

Objects hide their data behind abstractions and expose functions that operate on that data. Data structure expose their data and have no meaningful functions.

Hybrids between objects and data structures are bad. they make adding new data and new functions hard. In order to avoid creating such hybrids, we can create data structures and then encapsulate these data structures within objects that have functions.
We can add commands to objects that call commands on their internal properties to avoid making chain calls.

Book notes: Radical Candor - Part 1, Chapter 1

Luis Felipe Ciochetta — Wed, 03 Feb 2021 02:03:50 +0000

Hello folks

I was about to finish my reading of "Radical Candor" when I realized that I should have documented my whole study process like I am doing with other subjects I am studying.

So I am starting the book over and this time I will write notes and publish them here.

Disclaimer: if you never heard of this book, it is a book about managing people, it is not a technical book about programming

Notes

Part 1 - A new management philosophy

Chapter 1 - Build radically candid relationships

"This is not babysitting, it's called management and it is your job"

^ So this is one of the first things she says in the book and well, I am someone that has complained in the past about having to babysit adults because they just could not focus.

Emotional labor (aka babysitting) is a huge part of management.
Bosses have three responsibilities: guidance, team-building, and results.
Good relationships are the key to good management.
There is a direct correlation between a manager's responsibilities and relationships with his direct reports. good guidance, team building, and achieving results make for better relationships, and failing to fulfill these responsibilities leads to bad relationships.
There are two dimensions that can predict good relationships, "caring personally" and "challenging directly".

^ These two dimensions are very important, they are the foundation from where the Radical Candor framework emerges.

Caring personally here means that you should know people who work with you on a personal level, not caring only about what affect their job.

^ Ironically, if this framework is correct, caring about the stuff that is not job-related has a big impact on the contributor's job;

Challenging directly here means telling the truth, delivering feedback about bad situations, making hard choices, firing people. Basically, not being afraid to piss people off.
When you have a relationship with someone where you care personally and challenge directly, you have a radically candid relationship.
Radically candid relationships make for better communication, trust, and group cohesion.
The job of a boss doesn't need to go in the direction of crushing his direct report's souls, it can be to mitigate the misery and bring joy to the work.
People do not become bosses because they don't care about other people, and yet, this is how lots of people feel about their bosses.
There is a fear of caring personally because it does not seem professional. Being professional should not mean hiding your feelings or thoughts, neither changing your personality. People are not comfortable doing those things. So you should not hide who you are and create a safe space for other people to do the same.
You, as a boss, should not feel superior to your direct reports, being the boss is a job, not a value judgment.
Challenging people is to embrace conflict as a way to grow, to give people the chance to correct themselves.
Conflict may lead to hurt feelings, do not dismiss or disrespect the feelings of other people, instead, show them you care personally, and that's the source of the conflict.

"The hardest part of building trust is to invite people to challenge you just as directly as you are challenging them."

You should encourage people to challenge you in the same intensity, the one that can lead to hurt feelings. This can be hard, but it will lead you to learn about yourself.
Radical candor is not about being rude or front-stabbing people. This would not be caring personally. Nor is it about talking about everything that comes to mind, it's about meaningful conversations.
Both caring personally and challenging directly are different from person to person. Not everyone feels comfortable with the same levels of intimacy and conflict.

Book notes: Clean Code - Chapter 5

Luis Felipe Ciochetta — Fri, 29 Jan 2021 16:54:25 +0000

I will jump right into the notes, the description for the note formatting is in the first post of this series

Chapter 5 - Formatting

Good formatting is good for communication, and we should strive to have good communication
Small files are better to understand (and therefore better to work with) when compared to big files
Vertically close lines of code should have a close logical relation

Variables should be declared as close to their usage as possible

Functions that are called by another function in the same file should be right below the caller
Big lines of code are bad, 120 characters/line is fine, more than that is not
The team has the final word on the formatting rules for the codebase

Book notes: Clean Code - Chapters 1-4

Luis Felipe Ciochetta — Wed, 27 Jan 2021 02:31:42 +0000

Hello folks!

I am starting a new series of posts about the books I am reading, these will be my notes on the book

I thought about reading the entire book before publishing any notes, but I think this would take a long time, and if anyone has feedback, it would make the cycle bigger than it could be.

So I will publish my notes as I read the book.

The book I am currently reading is Clean Code, an absolute must-read in our profession, I am sure most people that will read this have already read the book, but I hope I can help these people remember its content.

Structure

This is how I've structured my notes

Notes

first note
second note

...

n note

Direct quotes

This is a quote (not really)

Others

Everything that is not a quote or a not is a personal thought

Notes

Chapter 1 - Clean Code

Good code matters (obviously)
It's our responsibility as developers to communicate the risks of having a bad code to business people

"Code, without tests, is not clean. No matter how elegant it is"

We should leave the code cleaner than when we've checked it out

Chapter 2 - Meaningful Names

Avoid meaningless names like "a", "a1", "b"
Avoid noise words like "Data" or "Info"
The length of a variable name should correspond to the size of its scope

"There can be no worse reason for using the name c than because a and b were already taken."

I laughed hard at this^

"One difference between a smart programmer and a professional programmer is that the professional understands that clarity is king. Professionals use their powers for good and write code that others can understand."

"Classes and objects should have noun or noun phrase names like Customer, WikiPage, Account, and AddressParser"

Pick one word per concept, don't use "get" and "fetch" for the same thing
Pick one concept per word, don't use "get" for both fetching and retrieving properties
People who will read our code are programmers, so we should use technical names over domain names; Ideally, the name will express both technical usage and domain
Add meaningful context; Using prefixes, namespaces, objects to hold information and with the name of other variables
Don't add meaningless context; Don't prefix every single variable in your application with its name
Shorter names are better when they are clear enough

Chapter 3 - Functions

Functions should be small, as small as possible
Functions should not hold nested structures; if, else and loops should have only one line, and if more is necessary, this should be a function call with a descriptive name
Functions should only do one thing and deal with abstraction only one level below its name;
If a function does only steps one level below of abstraction, it is only doing one thing; After all, we write functions to decompose large concepts

"Functions that do one thing cannot be reasonably divided into sections"

Functions should as few arguments as possible, having a lot of arguments make a function trickier to test
Monadic functions (functions with 1 argument) should have two forms, you either ask a question to a function with its argument, and it has a return value that is that argument transformed, or you should have no return value and the function should change the state of the system
Passing booleans to a monadic is a bad practice, if you are doing it, you probably should split that function in two
When a function needs more than two arguments, you can likely wrap then into an object
Functions should have no side effects

"Side effects are lies. Your function promises to do one thing, but it also does other hidden things."

Functions should either do something or return something but not both
We should use exceptions instead of returning error codes; When we return errors, we have to deal with them immediately, when we throw exceptions, we can deal with them later
Try/catch blocks should each call their own functions

"Error handling is one thing"

Therefore, if we want our functions to do one thing, a function that handles errors should do only that

Chapter 4 - Comments

"Comments are always failures. We must have them because we cannot always figure out how to express ourselves without them"

Before writing a comment, we should before try to better express ourselves through code

^ This thing he says about comments is very important, he comes back to it a lot.

If a comment is not maintained with the code, sooner or later, it will be lying about the nature of the function or class it is talking about
Lying comments are worst than no comment at all

"It is possible to make the point that programmers should be disciplined enough to keep the comments in a high state of repair, relevance, and accuracy. I agree, they should. But I would rather that energy go toward making the code so clear and expressive that it does not need the comments in the first place."

It is fine to write comments warning other developers of the possible consequences of using or changing a function
It is fine to leave some TODO comments. But you should regularly try to eliminate them (implementing the changes they describe)
Redundant comments are bad //and useless
Mandated comments are bad, don't tell other programmers to write huge comment headers for each function if the functions themselves are expressive and clear enough
Journal/log comments are bad, this kind of comment should be kept in commit messages
Comments announcing the start or the end of a subject are bad, unless you use them rarely and meaningfully
Comments announcing the end of a subject or a function are bad
Commented-out code is bad
Comments should be located near the code they talk about, this makes the maintaining of such comment easier
Comments should not have a lot of information