DEV Community: João H. Capucho

Thoughts on API Design

João H. Capucho — Mon, 30 Jan 2023 18:56:46 +0000

I really enjoy talking about REST API. Yes, for some people, REST is the past-generation stuff, and GraphQL is the cool kid in the block. Even though over the past months I've used GraphQL every day, REST still has my heart, and I have to be careful to avoid my bias toward it.

If you had to use SOAP as a communication protocol, REST was most likely your savior from the deep trenches of old-school distributed systems. Sadly, even though I'm not that old in the area, I've worked for an old bank, which exposed me to all kinds of ancient things you can imagine (yes, even COBOL).

This article is not a extensive guide on best practices, is just some quick thoughts that might benefit beginners in the REST journey. If you're an experience developer, this will most likely be boring.

Goals

One of the most important subjects to understand how to make your API comprehensive to your users/consumers is to have clear goals. In his book "The Design of Web API's" Arnaud Lauret defines five questions that one needs to answer to start reasoning on how to deliver a proper API.

Who can use the API?
- Here's where you list the profiles of people who are going to use your API;
What can they do?
How do they do it?
- Where you will decompose the previous question into steps
What do they need to do it?
- What is needed in each step, and where does the input comes from
What they get in return
- Which is where you state everything that is returned by each step and how it will be used;
Goals
- Where you will combine the hows, inputs, and outputs into a concise objective

Answering these questions will enable you to clarify what are the things your API must deliver. You can define a table where you fill those things, and you can run through a product manager and other colleagues to get feedback on any spots that you might be missing.

Error handling

Identify the possible errors of your API. Document all the cases, and error codes that you might return. A really nice to have is exhaustive errors. It's really annoying when you send a request, then the consumer receives an error. When the error is fixed and you re-submit the request, you get a new error. You can easily a errors array and add all the issues into a single response, so the consumer can clearly see all the issues within the request. This is especially needed for API's that are used in forms.

A good pattern I've found is to return something along these lines:

{
 error_code: SOME_ERROR_CODE,
 error_message: 'Something wrong is not right',
 validations: [{
   property: 'entity.created_at',
   code: 'INVALID_DATE',
   message: 'whatever message'
 }]
}

Be predictable

Lauret has a definition of consistent API's that I quite liked it. He splits into 4 levels:

Consistency within an API
Consistency across an organization/company/team
Consistency with the domain of an API. How you represent the entities should be matching the domain experts' expectations
Consistency with the rest of the world. Meeting standards, like ISO's, RFC's, and protocols.

So, be consistent wherever you can. Entity names, API responses (including error ones), path, headers, etc.

Before I forget, it would be lovely if folks stopped using X- for custom headers. There's an RFC deprecating this for around ten years now. I don't blame you, no one talks about this.

Adaptability

I loved when I worked in an API where we enabled our customers to get our reports in whichever format they wanted. JSON? You have it. PDF? Right now. CSV? Go crazy, just send me the header.

Here you can also include localization, internationalization, etc. I mean, your users might still use a really old imperial system that doesn't make any sense at all. If you send them a response in meters they will be completely lost, so you could enable them to have the answer in yards.

Your REST API can also enable lazy load to improve performance for your customers, they can easily expand the data sets they're interested at from a given entity. This is one of the main selling points for GraphQL from many folks, but can also be done in REST API's. Stripe API for reference.

Pagination

Please, pretty please, don't return something that only gives the consumer the next_page_token. This doesn't enable the consumer to go back and forth on the data.

You can have a quite simple response and still be useful:

"pagination": {
  "count": 10,
  "total": 250,
  "current_offset": 10
}

Then you know that the next page has the offset parameter as current_offset + count. A good thing is to have a parameter to limit the number of elements returned as well.

Documentation

One of the main things, and often overlooked, is to have clear documentation. For me, one good example is the Stripe API. They provide not only clear documentation but easy-to-follow examples, with commands that you can copy and paste to try it out. From my humble perspective, this should start to be in place from the moment you start designing your API. Otherwise, details will often be overlooked later on.

One other relevant portion of your documentation is that you should avoid the trap of only having it in an "engineer-oriented" format. It's always beneficial to have it in a human-friendly format (your JSON Schema is not so human-friendly).

Versioning

When applying semantic versioning to your APIs, all that matters are two digits: BREAKING_CHANGE.NON_BREAKING_CHANGE. However, your customers don't really care about the non breaking changes. So you can use the versioning only on the first value.

There are many ways you can version your API. Path, domain, query parameters, custom headers, content negotiation. Pick your flavor. I honestly tend to prefer the version to be in the path, such as /v1/users.

Request tracing

Use correlation_id's across your requests to enable yourself to trace the request flow across multiple services. If you're not familiar with the concept, you can read it in here.

You will be thankful when debugging.

Idempotency Key

Your customers could have retry operations when things fail, automatic reprocessing of messages. This could end up creating problems if there's no way to check if a operation has already been done. Idempotence is the property where an operation can be done multiple times without affecting the end result beyond the first application. Enabling your customers to take control over their idempotent requests is super useful. Stripe API has a good example on it if you want to dig further

5 books I'm happy that I've read in 2022

João H. Capucho — Sun, 15 Jan 2023 10:32:36 +0000

After a year goes by, it's always nice to take some to reflect on what we've done, and what we would like to do it differently for the next year.

After a long time in my life focusing on reading mostly technical books, in 2021 and 2022 I've tried to read more on other topics. Specially around emotional health, career and productivity. I will state here in a few sentences what I've read that impacted me the most, the books I'm usually recommending to my friends.

The Passionate Programmer - Chad Fowler

This is my second read of this book. I've got this as a gift from a mentor back in 2015. I find it one of the best books regarding our career as software engineers. Specially for those ones that enjoy playing some instrument, the author who has been a jazz player, makes lots of connections on our software engineer lives and a musicians life. It's broken down in quite small chapters with easy to grasp action points we can start taking on our daily work lives to improve our deliveries, but also the value we aggregate in our careers. It's definitely one of my favorite books in IT.

Laziness does not exist - Devon Price

A must read for all of us who have been dedicating most of our lives to our jobs. Many are the reasons one can put their work life above everything else and sometimes we start becoming our careers. I took a serious step back past year, where I didn't want to be "João, software engineer" anymore. This shouldn't be the first thing people relate to me when they think about me. That's not how I want to live my life. This book gave me a better emotional foundation to create distance between my personal life and work life. It's helping me to pursue my hobbies, freeing myself from the constant and obsessive behavior that I should always be productive, that everything I do should be towards growing in my career.

Atomic Habits - James Clear

This book is most likely to cause the "urgh" reaction for most people that think that self-help books should be locked away. For me, it was a great complement on "The Power of Habit". It gives you more practical advice on how to keep a routine, how to introduce new habits in your daily life, and overall, made it easier for me to stick my routine.

We all have routines, even if it's jumping on to the sofa after a full day of work. The main difference is on whether is the routine you want or not. Here I'm not going to preach on whether it's good for you or not. The only thing I care is if it's what you want. If you wanted to have a different routine in your life, this book might help you.

One of the great things I've learnt here is habit tracking. It's quite nice to see how you're being able to stick to the new habits you want to introduce, until they're fully integrated in your life.

The sexual politics of meat - Carol Adams

As a vegan person, looking vegetarianism and veganism through the lens of feminism was eye opening. This was the first classic vegetarianism book I've read in my life (after being a vegetarian for 3 years and vegan for 1).

It's quite a journey through history, patriarchy and movements from distinct people, embracing a cruelty-free life. It's disturbing to see how women and animals are constantly reduced to body parts for consumption. The way we change our vocabulary to transform those into absent referents "to keep the our 'meat' separated from any idea that she or he was once an animal...". Reading this solidified even more that this is not a life I want for myself.

Focus - Daniel Goleman

I really enjoyed reading Daniel Goleman's "Emotional Intelligence" book. It was what triggered my desire to have a deeper understanding on the subject and also pushed me towards seeing a therapist to heal myself. So, this book, was just a curiosity about what else the author could teach me.
I honestly feel in love with the book because it confirmed me something I've lived practically. A meditative practice is substantial to being able to have the ability to focus. I've used the Headspace app since the beginning of the pandemic to help me cope with the period, and I've saw many advantages of that in my life. For this specific subject and reference, I wrote an article called "The zen of focus" where I describe the journey with lots of references from this book. I highly recommend reading it if you want to understand the journey I had and how it helped me throughout the Covid-19 pandemic and still helps me to this day.

PostgreSQL - Partitioning & Inheritance

João H. Capucho — Sun, 30 Oct 2022 12:03:23 +0000

This article briefly exposes the basic concepts of how table partitioning and inheritance works in PostgreSQL.
Some prior knowledge on basic queries and DDLs is necessary.
There will be no deep dive in how the feature works under the hood, neither how to migrate your existing data to this solution.

Introduction

Since university, PostgreSQL has had a special place in my heart. I was heavily influenced back then by this incredible a professor I had. Until this day in my career, I've never seen someone with SQL capabilities like hers.

I can still recall this day, where after all her lectures about referential constraints and all sorts of integrity one can think about, she raises the question about what DB we used & liked the most. As we were a bunch of naïve young folks, breathing the LAMP (Linux, Apache, MySQL, PHP) era, the most common answer was MySQL.

Then she explained all the issues MySQL had on enforcing some constraints (I won't remember which ones since I never actually used MySQL professionally). Then, she took us to the lab and had made us test those constraints in both DBs. On that day, I feel in love with PostgreSQL.

However, it's a love that I neglected for a long period of time. But over the last months, I've had lots of fun playing with it again. That's why I decided to write a little about it.

Partitioning your data

According to Daniel Abadi, "Data Partitioning is the technique of distributing data across multiple tables, disks, or sites in order to improve query processing performance or increase database manageability".

If you want to dive deeper into how to handle data-intensive architectures and the problems around them, I would highly recommend reading Martin's Kleppman "Design Data-Intensive Applications". It's definitely my reference book for system design nowadays.

Now let's dig into how we can achieve partitioning at a table level using inheritance.

Using inheritance

Wait, ain't inheritance an Oriented Object programming thing like those Java public class Dog extends Animal type of stuff? Well, our folks working on PostgreSQL found a neat way to partition our data based on inheritance.

Let's start with the following table:

CREATE TABLE t_archive (id serial, year integer, data text );

But this table would grow too much (here we're not talking about indexes), and we want to optimize the search. A simple way is to partition the data into different tables. But then you would need to know which table to query from, right? Well, not exactly.

If we use inheritance, we can sort of avoid that trap. Let's create a few tables that will inherit from t_archive:

CREATE TABLE t_archive_2021 () INHERITS (t_archive);

CREATE TABLE t_archive_2020 () INHERITS (t_archive);

CREATE TABLE t_archive_2019 () INHERITS (t_archive);

Now all the tables have the same structure as follows:

When you query the parent table, all child tables will also be scanned.

Let's try now with a filter based on the field we want?

Well, it wasn't as I expected. I would like to have a single table being looked at. We don't care about data in other years beside 2021, right? What we want here is to scan a single table. How to achieve that?

In order to do so, we need to explicitly define checks on the table, so PostgreSQL can base itself on them when creating the query plan. Let's add a few checks to our tables:

ALTER TABLE t_archive_2021 ADD check (year >= 2021 AND year < 2022);
ALTER TABLE t_archive_2020 ADD check (year >= 2020 AND year < 2021);
ALTER TABLE t_archive_2019 ADD check (year < 2020);

Now, when we run the query, the data lookup will only check for the correct table.

Changing parent table

When altering the parent table, for instance, adding a column, all the changes will reflect in its child tables. By running the following command:

ALTER TABLE t_archive ADD COLUMN example text;

The result looks like this:

Conclusion

Hopefully with this brief exposure to the concept, you will be able to at least remember about it when the partition topic comes up at a design you're doing. Not that you would use it, but at least you would one more tool in your database Swiss army knife in case you need it.

See you later :)

A Philosophy of Software Design - Review

João H. Capucho — Wed, 31 Aug 2022 17:22:02 +0000

by John Ousterhout

tl;dr;

Even for the more seasoned engineer, this book is a good reminder of design practices that needs some refreshing every once in a while. At a low cost (< $20 for a technical book), and easy to read, will provide you a pleasant read for 1-2 weeks.

For the ones beginning their careers, is a good introduction to Software Design principles that won't require deep computer science knowledge, with guidelines that will improve your output.

Overall: 7.5/10.
If you wanna buy the book, click here :)

Introduction

The book already starts defining what it wants to tackle: software complexity. The author defines complexity as "anything related to the structure of a software system that makes it hard to understand and modify the system.". From this point on, the author will go through the "complexity symptoms" and what practices software designers can apply to minimize the problem.

One of the first things I've appreciated is that the author won't take the position from the software writer; instead, it will look through the software reader's eyes. These two are different personas, separated by time, context, and knowledge. The latter requires context that is potentially implicit in the written code.

What I've liked

After setting the tone, the author will start describing his design philosophy based on his experience teaching at Stanford. These design principles are good practices, sometimes already detailed by other authors, but written in an easy-to-digest manner, even if you're a beginner in software development.
In addition to his principles, the author will describe "red flags" that you can use to identify potential problems in your approach. John was also careful enough to create a summary at the end of the book with the red flags and which pages you can find more about them. He did the same for the Design Principles he distills in the book.

Within his design philosophy, the author will also challenge the status quo defined by classic authors, especially R. Martin. This book will discuss a few things from Martin's "Clean Code" - how to write functions, modules, comments, etc. -. When I compare both, John's seems closer to our reality and more pragmatic.

What I didn't like as much

And what about what I didn't like as much? One thing that I'm not so sure whether I liked or not is that the author comes back to the same example too many times. Why don't I know if I liked it or not? When I saw it was the same example again, I would get annoyed. However, I will most likely remember the piece of code for the rest of my life.

Another thing that stuck with me was: that the author states that you shouldn't expose your internal data structures to the API consumers. I don't think we would disagree with that. However, later on, he mentions that you should not use implementations that are a mismatch from your variable interfaces. I found this piece a bit conflicting with the first statement. I can agree that you can do that for internal variables in your module. Yet, for exposed variables, I'm not so sure about it. This is what enables me to be free to do internal changes in my module without introducing breaking changes to my customers.

Simple takeaways

The author goes through many things that I find valuable for any developer. Of course, they might impact you in different ways depending on at which point you're in your career. Among those, I will list a few (summarized, of course) that I find valuable to be reminded of even if you already know:

Complexity will always be present. It's a constant exercise to simplify code throughout a software lifecycle.
Write deep modules with simple APIs, and design them as much as possible for common cases. Omit unimportant details.
Do not fall under the temporal decomposition trap. Your code needs to reflect the knowledge required to perform tasks, not the execution order.
Define errors out of existence, and make interfaces simple.
Centralize error handling for simplicity.
If pieces of code are related, think about whether you should bring them together
Consistency is key for good readability. Don't start changing existing conventions without taking the effort to understand why they exist, and what value your new convention brings. Also, don't introduce yet another convention if you won't be able to remove the old one entirely.

Conclusion

In summary, this book is worth the time. Easy to understand, objective, and with small chapters that make it easy to assimilate things. For the more experienced engineer, many things could be a repetition of what you saw in other classics and already put into practice in your daily work. For you, the first chapters is where you will find the most valuable discussions.
However, there are still valuable things that you most likely can take from it. For the ones who joined this journey more recently, I would definitely recommend it. Most likely it will improve your deliverables if you pay attention to the recommendations on your daily job.

If you wanna buy the book, click here :)

Redis for beginners, part 1

João H. Capucho — Sun, 31 Jul 2022 19:30:00 +0000

There's no discussion that Redis is one of the most consolidated caching tools today. Whenever someone wants to introduce a cache layer in their systems, Redis will be one of the names on the table.

For a while, I only associated Redis with caching solutions. However, I was missing a lot that I could get from this tool. About three and a half years ago was the first time I saw a solution that used Redis as a queue mechanism. As someone that had recently come from Java, I was having a hard time finding a solution where I could schedule jobs in Node.js. Talking about that with a few other teams with more experience in Node.js, I've come across "BullJS", a Redis-based queue for Node. This library introduced me to a few solutions based on Redis capabilities, like rate limiting, job priority, job delay & scheduler, among a few other things.

By seeing the capabilities offered by this library, I started to wonder what other things I could get from Redis. I will likely write two (or three) articles on this to save you the trouble of reading them all at once. First, let's start with some concepts on Redis and a few relevant data structures.

Redis

Do you need a caching tool? Redis. Do you need a message broker? Redis. Do you need a database? Redis. Do you need a stream engine? Still, Redis.
I'm not saying Redis is the perfect tool for all of the above, but depending on your use case, it might be the right fit. It includes transactions, pub/sub, failover mechanisms, and more.

Redis Serialization Protocol (RESP) is what powers Redis behind the scenes. According to the docs, a simple to implement, fast to parse, and human-readable protocol, usually running over TCP (but not tied to it). It's aimed to be a request/response model. However, it contains two main exceptions:

It allows clients to batch multiple commands. Which Redis refers to as Pipelining.
When using a pub/sub subscription, the protocol will change its semantics to behave as a push protocol. Meaning that the server will push messages to the client as they arrive.

One curiosity on the protocol is that Redis keys are binary-safe. This means you can use strings, numbers, and binary values. Just keep in mind that they're also case-sensitive. One thing that struck me is that keys are allowed to be up to 512MB. I honestly have a hard time finding use cases for that.

Basic Commands & Data Structures

Deleting data

SCAN
Iterates over a collection of elements. You must be careful with this command since it can return too many elements in production and degrade your performance.

Use with caution.

KEYS
Returns all keys matching a giving pattern.

EXISTS
Returns a given key exists.

OBJECT ENCODING
Returns the internal encoding for the Redis object in a given key. This is used a lot internally to check if an operation is valid. For instance, Redis checking if the value is a number before doing a INCRBY or DECRBY operation.

RENAME
Rename a key. An error is returned if key doesn't exist.

UNLINK
It follows the same idea as a delete command, however, it's a non-blocking operation, unlinking the key & freeing the memory associated with it in a separated thread.

DEL
Removes key in a blocking operation

Expiring data / Time to Live (TTL)

When setting expiration periods in your keys, you can mainly use three functions: PEXPIRE, EXPIRE, EXPIREAT. They will set the time to live in milliseconds, seconds, and Unix timestamps, respectively.

If you're curious about the TTL for your key, you can check that using the TTL or PTTL commands.

And if at some point you decided that the key/value pair you have are really important, you can make them persistent using the PERSIST command.

Hashes

Command list

Mostly used to store objects, typically used for rate limiting and session stores. Data manipulation commands are as follows:

HGET - retrieves the hash field value. If the value is also passed, it returns a boolean informing whether the value is present or not
HEXISTS - checks if a field exists in the hash
HINCRBY - increments key value;
HINCRBYFLOAT - increments field float value
HSET - set key value
HSETNX - only set field value if the key doesn't exist, if the key already exists, this command has no effect;
HKEYS - return all keys in the hash;
HVALS - return all values in the hash;
HDEL - delete a field stored at a key;

Lists

Command list

An ordered collection of strings that enables you to add elements to the left & right, it can be used for stacks and queues. In lists, there's no nesting.
Good use cases are activity streams and queues (producers, consumers).

Data manipulation

LLEN - list length
LPUSH - add an element to the left
RPUSH - add an element to the right
LPOP - remove an element from the left
RPOP - remove an element from the right
LRANGE - retrieves elements from a list. It's used as (key, start, stop), where the key is the list, and start and stop are the indexes. In case you want to retrieve all elements, one can use a -1 as the stop index.

If you want to use the list as a stack, you can limit yourself to using the LPUSH and LPOP commands. If you want to use it as a queue, you can use RPUSH and LPOP commands.

Sets

Command list

Sets are an unordered collection of strings, in which you can do difference, intersection, and union operations. The data cannot be nested.
This is interesting for use cases like tag clouds, and unique visitors.

Data manipulation:

SADD - add an element to set
SMEMBERS - returns all the members of a given set
SISMEMBER - returns whether the element is present in the set
SREM - removes an element from a given set
SPOP - removes a random element from the given set
SDIFF - returns the difference between two sets
SDIFFSTORE - stores the diff between two sets into a new set
SINTER - returns the intersection between two sets
SINTERSTORE - stores the intersection between two sets into a new set
SUNION - returns the union between two sets
SUNIONSTORE - stores the union between two sets into a new set

Sorted sets

Command list
Sorted sets contain a field called score, which will be used to sort your data. The elements in your sorted set cannot repeat, the score is not considered for the uniqueness of an element.

For instance, let's say you want your elements sorted by time so you can have a time-series, you can ensure that by having the minute the element was added as the score, and this will enforce that your elements are always sorted by time.

Your elements will always be sorted, with efficient inserts.

Data manipulation:

ZRANGE - returns the elements from start to stop, enabling you to ask in reverse order, also returning the scores.
ZRANK - returns the rank of an element in the sorted set
ZADD - adds the elements with the score to the sorted set
ZCOUNT - how many elements the set has with scores between min and max parameters
You will have lots of commands that look like the set commands, and a few extra ones due to the nature of the sorted sets.

References

Post image from: By Carlos Prioglio created the logo for the copyright owner Salvatore Sanfilippo, lead developer of Redis. - http://redis.io/images/redis-logo.svg, Fair use, https://en.wikipedia.org/w/index.php?curid=40127986
Redis docs
Notes taken a long time ago from Redis for Javascript Developers
Notes taken a long time ago from Introduction to Redis Data Structures

The zen of focus

João H. Capucho — Thu, 30 Jun 2022 19:35:38 +0000

After a long time without writing, I wrote an article in Portuguese regarding what changes I introduced in my life and the many attempts to improve it. In a quick summary of the article, I spoke about changes I've made in my routine, like waking up earlier, having a proper sleep routine, and whatnot. This article is somewhat a continuation of that, where I will talk about something I've spent quite some time and enjoyed a lot: meditation.

The start of the journey

Two years ago, I started my first attempts with it, using an app called Headspace, I've tried for a few months, and the experience was great.

I struggled in the beginning. I couldn't find a comfortable position to sit down without feeling pain in my back. Ten minutes felt like an eternity to me.

I've always been a bit too anxious, and sitting down for ten minutes without doing anything? Impossible, I said to myself. But would I be beaten by a ten minutes process in my day? Not a chance. I decided to trace a plan, one week doing 5 minutes, next week I should be able to do ten minutes without worries. And to my great surprise, everything went great! I could keep up with those 10 minutes a day for a long time.

However, I wasn't finding the value for the cost I was having. I was paying around 12 euros per month, which was a bit too much. I tried a few playlists on Spotify, and even Headspace has some good guided meditations there. After a while, I gave up. But I got lucky, and the company I work for decided to start giving it to us as a benefit during the pandemic to help us cope with the lockdown.

I picked up the routine again, going from 5 to 20 minutes a day of meditation. And I will be honest with you. Meditating helps me a lot to cope with reality. It helped me through the covid pandemic (two people living in a 30m2 apartment for one year and a half) and keeps helping me nowadays. When I have a streak without meditating, I can feel the difference: my patience and focus are jeopardized after just a couple days without practicing. It also reduces the quality of how I can process events that happen to me in a daily basis (according to my therapist).

But the main question is, why I'm telling all these things? After around 100 hours of meditation, I was intrigued by what it was doing to me. Even my psychologist said I could reflect better and be more reasonable. So, I decided to study a bit about it.

I picked an author I like a lot: Daniel Goleman. I picked him because he had a book on a subject that I felt that was directly impacted by meditating, my focus. So, I've dug up his "Focus - The hidden driver of excellence" book, which has some interesting explanations on mindfulness and meditation.

Learnings

The so-called "mindfulness" practice can offer different benefits according to the author, which states: that it "boosts the classic attention network in the brain's frontoparietal system that works together to allocate attention. These circuits are fundamental in the basic movement of attention: disengaging your focus from one thing, moving it to another, and staying with that new object of attention".

That's how it feels after a while. After practicing for some time, you start to be able to focus on your breath, isolating undesired thinking patterns, and avoiding scattered thoughts. Whenever I struggle with my brain being 100km/h, I know where I have to go: inhale, exhale.

This is one of the things I've heard the most in my guided meditations. Whenever you feel your thoughts wandering, focus on your breathing. It's the key factor to calming down and going back to a place of attention.

Yet, another benefit I'd never heard of before reading his book was "meta-awareness, attention to attention itself, as in the ability to notice that you are not noticing what you should and correcting your focus.". After reading this passage, I've reflected upon my own experience reading the book. How many pages did I "read" when I was thinking about something else? Maybe something that happened at work or a bill I had to pay?

When I brought this subject to my therapist, she actually asked me to be mindful of doing basic things. For instance, paying attention to the flavor of whatever I ate, or when I was simply taking a shower. "Take your moment when chewing your food, appreciate it. When taking a shower, enjoy the water, the feeling of being clean after it.".

The realization I had was that paying close attention to these small things will bring even more joy to life and will improve your overall happiness significantly.

Of course, this article doesn't intend to state that you should be focused all day long. A wandering mind is a place of creativity, while it also "operates in the service of solving problems that matter for our lives.".

This was a mere invitation for you to give it a shot. Five minutes the next couple of weeks before work or during your lunch break. See how it goes. The worst it could happen is feeling like you didn't watch one or two episodes of your favorite series. You can start with this one, if you want. Here's a free playlist as well.

And always remember, focus on your breath :)

References

Goleman, Daniel. Focus: The Hidden Driver of Excellence. Bloomsbury, 2013. Print.

"Zen" by Hipnosapo Peres is licensed under CC BY-NC-ND 2.0. (Cover)

Rotina e sanidade, parte 1

João H. Capucho — Tue, 31 May 2022 19:20:49 +0000

Em primeiro lugar, que fique claro que esse texto é uma exposição de ideias sobre o processo que venho fazendo, me
utilizando de cobaia, para melhorar o que eu acredito ser essencial em minha vida. Logo, esse artigo não tem a
intenção de te convencer de que o que eu faço é a melhor forma de se fazer. Quero trazer apenas opções e razões do que
fiz e que acredito ter melhorado minha vida. Espero que a partir disso você possa fazer suas reflexões e identificar o
que é ou não aplicável ao seu processo. :)

Acontecimentos recentes na minha vida me fizeram ter vontade de escrever sobre algo que vivo há pelo menos dois
anos e meio. Um processo contínuo de transformar hábitos e processos que façam do meu dia-a-dia algo mais
saudável. Esse processo teve início em meados de janeiro de 2020. Escreverei sobre isso em alguns posts, acredito eu,
pois é algo que demandará um pouco de tempo para falar sobre o que fiz e os impactos que tenho percebido.

Para eu conseguir explicar o que mudou, eu preciso explicar do que a minha vida consistia antes desse processo. Se
você está lendo isso, provavelmente sabe que sou desenvolvedor de software. Assim como a maioria das pessoas que vivem
nesse meio, minha vida poderia ser resumida em doses cavalares de café, dormir tarde, acordar o mais tarde possível de
forma que eu ainda conseguisse chegar no escritório antes da Daily (reunião diária de alinhamento do time, grosseiramente).
Além disso, eu sempre tive o hábito de estudar após o trabalho, pelo menos 45 minutos, durante os dias de
semana. O que acontecia frequentemente era estar muito cansado após o dia de trabalho (que frequentemente envolvia hora
extra) para conseguir fazer isso com qualidade.

No início da minha carreira profissional de software, priorizei muitas vezes as necessidades das empresas que
trabalhei ao invés das minhas, e não posso dizer que isso foi de todo negativo, tive benefícios claros na minha
trajetória graças a isso. Acredito que no início da sua carreira, dependendo das suas ambições e objetivos, diversos
sacrifícios serão necessários. Entretanto, isso não é sustentável no longo prazo, e percebi nesse momento que eu havia
chegado num ponto de inflexão, onde priorizar a minha vida e carreira não era mais opcional e sim uma necessidade (um bom
livro sobre o assunto emprego vs. carreira, é o livro do
Luciano Santos, "Seja egoísta com sua carreira").

O curso Learning how to Learn abriu a minha mente sobre como o meu
processo de aprendizado poderia ser melhorado, e cheguei a conclusão que eu precisava usar o meu melhor momento (pela manhã)
para fazer tudo que fosse voltado para o meu crescimento pessoal e de carreira. E foi aí que veio o pivô da mudança, onde
num belo dia em janeiro, olhando o dia frio lá fora, tomei a difícil decisão de
passar a acordar cedo (6am). Usaria esse horário para estudar por uma hora, e ter a minha noite livre. Morando no Brasil
talvez 6am nem seja tão cedo assim. Porém, morando na Suécia, o sol só aparece entre 8 e 9 da manhã em janeiro e o clima
frio faz a cama ser muito mais convidativa, o que fez a decisão ser bem mais difícil para ser sincero.

E qual foi o maior impacto dessa decisão? Percebi, com o tempo, que o nível de compreensão das coisas que eu fazia nesse
período eram muito maiores e que eu conseguia fazer conexões mais sólidas do que eu aprendia. Porém, honestamente, esse
não foi nem de longe o melhor ganho. O melhor ganho eu ainda estava para entender.

Ao longo das semanas que se passavam, cada vez menos o meu primeiro pensamento do dia era sobre trabalho. Não acordava
pensando no que eu tinha que entregar, com quem eu precisava marcar reunião, quem eu precisava cobrar ou o que eu estava
esquecendo. Como eu não estava correndo para me arrumar para trabalhar, tampouco eu me via pensando sobre trabalho.
O meu foco passou a ser totalmente direcionado na minha vida, nos meus afazeres (nesse meio tempo passei a usar metodologias para organizar
a minha vida que falarei no futuro, tentei algumas, como GTD,
bullet journal, onde eu falhei diversas vezes e voltei mais vezes ainda para elas).

Que tranquilidade foi passar a viver assim. Sou muito grato ao meu eu do passado que tomou essa decisão. Com a pandemia
eu passei a trabalhar de casa, e por começar às 9 am, eu passei a ter aproximadamente 3 horas pela manhã para realizar as
minhas coisas. Foi aí que o trabalho remoto ganhou meu coração.

Conforme fui entendendo o tempo que tinha disponível para mim, comecei a acrescentar coisas nessas horas que faziam o meu
dia melhor. Por exemplo, nunca fui de tomar café da manhã, sempre tive que sair correndo e não consigo sentir fome logo
ao acordar. Ao passar a estudar e ficar mais de uma hora entretido estudando, meu apetite aparecia e passei a tomar café
da manhã, o que fez com o que o resto da minha manhã fosse muito mais produtiva.

Foi em janeiro também que voltei a malhar, deixava para malhar a noite, pois era quando eu não precisava mais de um
raciocínio fino e poderia usar para descompressão do trabalho. Infelizmente, com o advento da pandemia, a
academia teve um fim forçado que só teve retorno após as 3 doses da vacina (final de 2021), e tenho conseguido me manter
consistente (+ de 4 vezes por semana) até então. Acredito fortemente que esse início e todas as mudanças inseridas durante
o processo, foi um dos fatores fundamentais para eu conseguir viver o lockdown sem entrar em
colapso, dado que eu passei mais de um ano trancafiado num apartamento de 30m2.

As mudanças que enxergamos como positivas tendem a ter um efeito cascata. Quando você passa a enxergar os efeitos positivos de algo
que fez, começa a procurar quais são as outras coisas que talvez façam sentido mudar que podem trazer-lhe algum
benefício. Um subproduto da decisão de acordar cedo, foi construir uma rotina que fizesse com que eu dormisse bem, para
conseguir levantar no horário.

Decidi que as 10pm seria meu horário para dormir, e que a partir das 9pm eu não mexeria mais em telas. Uma das coisas
mais úteis que fizemos foi comprar lâmpadas que você pode selecionar para ficar amarela, e durante as noites usávamos
esse modo para estimular o nosso sono. O que fazer entre as 9pm e as 10pm, me perguntava. Bom, um hábito que havia
ficado de lado desde o advento da Netflix na minha vida virou a melhor opção: ler.

Passei então a trocar a tela da televisão pelos meus livros. Sempre foquei muito tempo nos meus estudos em tecnologia, porém
com esse novo espaço, passei a dedicar parte do meu tempo com coisas que eu enxergava como essenciais para o meu futuro:
a minha saúde emocional, a gestão da minha vida e das atividades que preciso fazer, minha carreira e eventuais livros
de literatura. Além disso, nesse período também descobri que a cafeína tem um efeito estimulante de cerca de oito horas no seu corpo, o que
me fez tomar com muito prazer o meu último cafezinho às 14h.

Nas próximas vezes que eu me aventurar por essas terras quero falar mais a fundo sobre tudo que disse acima e sobre
outras coisas que acrescentei e removi da minha vida durante esse período. Acredito que o mais importante que aprendi
nesse processo é que uma rotina bem estabelecida me trouxe muita paz, mesmo em meio a momentos bastante delicados. Para muita gente ter uma vida assim pode parecer entediante, porém, eu descobri
uma relação muito saudável com o famigerado “tédio da rotina”. Ter um direcionamento claro e palpável no meu dia-a-dia
faz que eu consiga encontrar o meu norte quando a bússola começa a falhar. Evito questionamentos corriqueiros, tenho respostas simples
e tangíveis para a maioria dos momentos no meu dia.

Why are you using cronjobs?

João H. Capucho — Wed, 12 Aug 2020 18:48:21 +0000

There's an addendum in the end with a "fix" :)

Context

You work for an e-commerce company. In this company, users will create their accounts. What is the most likely your feature will do when they sign up? That's right. Send an email with a confirmation token so you can be sure that you're creating the account for the right user.

Ok, but what does this have to do with cronjobs? I will explain further down the road.

Most likely, this token will have an expiration. It can be 3 hours, one day, it doesn't matter how long it will take, at some point in time in the future, it needs to be canceled. A common way to handle these scenarios where you need to perform something in the future is using cronjobs.

The easy way? A cronjob that goes through the database table where you store the tokens, check the ones with more than 3 hours (if that's the expiration time for your company) and update or delete the tokens that are not valid anymore.

You might ask me, what is the issue with this approach? Well, the first thing is that this approach is time and resource consuming. By using this approach, every hour or so, you will need to go through the whole table to find the elements you need and do an operation on them. If there's nothing to be canceled, the cronjob will still use the resources.

Let's look through this with another set of lenses: scalability. The server that triggers the cronjob cannot scale up, otherwise, you could end up running the same process concurrently. For instance, if you're processing payments within a cronjob: two instances of the server triggering the same thing at the same time could end up paying a user twice if you're not careful.

What can we do to improve it? The nature of the process is event-based. Why can't we use that in our favor?

Instead of taking a time-based approach, let's think about how we can deal with this thinking asynchronously. First, when the user signs up, we process the event triggered by the "Sign up" button. This event, let's call it "CREATE_ACCOUNT". When you do so, another event is also triggered, which is "REVOKE_SIGN_UP_TOKEN". However, this event is to be processed in the future, how can we deal with that?

Asynchronous queues to the rescue

Imagine you have a queue. Now imagine that in this queue you can say: "Hey, process this thing for me in 3 hours, please?". The queue properly stores the information you sent and later, will execute the code you want for the events stored in it.

One may say: "Ok, this could be awesome. But how is it achievable? I have no idea on how to delay something for 3 hours".

No worries, Redis is here to rescue you from the deep cronjob forest. If you are a happy developer that works with Node.js, I will show an amazing library that can help you on this journey.

Bull is a queue tool for Node.js, based on Redis. By using it, you can implement asynchronous job processing, with delays, retries, priorities on your jobs, and much more. You may also use a dashboard to visualize in a nice way how your jobs are being handled. Remember, asynchronous processing can fail like any other piece of code. Keep in mind that you need to handle your failures.

The libraries provide good examples of how to use it and what are the features they contain. I will leave to you to find what's the best approach for you using it. Adding examples here, it would only increase the time to read and would be really close to the documentation to keep it simple.

Conclusion

I hope that this article brought a new perspective on how you can handle time-constrained events (or just events). I hope that the next time you see a cronjob handling an event-based process, you can think if it wouldn't be better to use an asynchronous process to do so.

Stay safe, use a mask!

Adendum

After posting, a friend brought to my attention that we could use a DB that has a TTL feature. In this case, since we use Redis, it contains an EXPIRE command which you could indeed implement this specific feature more efficiently. Using this command you could tell the DB to remove the key after a given time window. Unfortunately, the example wasn't complex enough, but what I wanted to bring to attention are the scenarios where you have a complex operation that you must execute after a given time an event occurred. Hopefully, that can bring some ideas for you future projects =]

Thanks @hkupty for bringing that to my attention and helping me improving the article :)

Image

"My vintage watches" by penmanila is licensed under CC BY-NC-ND 2.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-nd/2.0/

A quick guide to fix your Git mistakes :)

João H. Capucho — Sun, 02 Aug 2020 14:35:13 +0000

Introduction

Git is a wonderful tool. It's the tool that is usually mentioned as the one every developer should have under its belt. However, git is not easy to understand as well. Most of us frequently face a really simple flow, doing commands like pull, add, commit, push, merge and in some company flows you will use rebase. However, once in a while we screw up.

We know that sometimes ~~shit~~ things happen in your commit history and you might find it easier to create a new branch, doing a revert, or creating a commit like Fix typo. Making mistakes in Git is not so uncommon. However, knowing how to fix them is what makes me not bother when things like that happen. If you don't know the tools git provides to you, fixing mistakes can be painful. But if you do know the tools that allow you to fix your mistakes, it's like having a nice cup of coffee on a raining morning. Just delightful.

This post will go through the interactive rebase tool to enable you to fix your commits.

Basic rebase concepts

If you never rebased a branch, I will go through a quick explanation on it and I'll leave this article so you can dig deeper into the subject if suitable for you. The goal here is not to go through the rebase itself, but without a basic understanding, it won't be possible to explain the interactive options. So, let's start with a quick explanation of rebasing.

First, let's assume we have a main branch with the following commit history:

From this main branch, you create a new one called button-feature and started your work while your colleagues kept merging their PR's into the main branch. The times where a file was locked until your work is finished are long gone, thankfully. So your branches history will be a beautiful thing, looking like this now:

At some point, you will need to get the updated code from the main, right? Usually, people merge the main branch into their development branch. However, you can also rebase your branch from the one you created yours from.

To do so, you will update the contents from main by pulling the new contents and then, within your button-feature branch you would run: git rebase main. The result would be close to:

When doing so, the rebase command will add your commits on top of the latest commit from main. Even though they still have the same content, they're not the same commit anymore. The history of those commits is rewritten now, new commit hashes are assigned to each one of them. That's the reason I've changed the colors of it.

The main advantage here is that it allows the git history to keep a linear structure, making it easier to use operations like bisect to find issues within the code.

Rewriting history is a powerful tool, and "with great powers, comes great responsibilities". You know that since you were a kid, right? So, be aware.

Changing commit history can be a problem if you rebase a branch that people branched of from, since the commit hashes won't be the same anymore, not being possible to track down the original changes. A deeper explanation can be seen here

Interactive rebase to the rescue

It's common for us developers to forget something when doing a commit, to name the commit inconsistently, forgetting to add some piece of code or we simply want to delete a commit we've added.

These things among others can be helped if you know how to do an interactive rebase. I've separated three topics from it that at least in my development workflow are the most common to be used.

Renaming a commit

In this scenario, I've added a commit with a name that doesn't follow the guideline that I should use.

If you take a closer look, you will see that the second commit is not consistent with the latest one, so let's fix that.

First, we do a git rebase -i HEAD~2, which means "I want to do the rebase operation on the latest two commits".

After executing that command, you will see the following:

The rebase menu already shows you everything you can do. Read through it to see the possibilities if you don't remember from the top of your head. Let's choose the reword option, or simply r, to rename the commit we want.

Every time you see this screen, to execute the operation, you will need to save and close the file. When doing so, the rebase process starts and the commit prompt is showed to me.

Here I can change the text to whatever I want.

The new commit message is assigned to that commit I had. If I take a look into the log again, we'll see an updated message.

Dropping a commit

This one is for those cases you just want to test something and keep in your change history, but you want to delete later. You could stash, apply the stash later, but you can also simply commit it and later just delete it. I use this a lot when I have to do some tests, especially if I want the change to go through the pipeline. After testing everything I wanted, I can just delete the commit.

Assume that I have a commit called Remove this commit that I will delete it in the future as we can see in the following image.

We can do a git rebase -i HEAD~, which means we want to rebase the last commit.

Now you would just replace the pick word with d for drop, after doing so, your commit should be long gone.

Editing a commit

If you ever had to face a commit where you forgot to add a piece of code, or only after committing you saw a broken test (yeah, we know it's not always we do TDD). This is the tool to help you fix it.

The commit Add message file contains a space that it shouldn't. You can see it in the green line with a console.log(' message').

How to fix this issue without creating a new commit? Let's make the magic happen. First, we will rebase to the point we want. So, let's do a git rebase -i HEAD~2 to rebase the latest two commits.

Now, we will pick the option e to edit the commit.

This is what you will see when saving the file and closing the editor:

Now we can edit the code:

After this change, we can add the fix in the same commit using git commit --amend. This will open the commit prompt for the commit you wanted to edit.

You can simply save it and close the editor. If there's nothing else to be done, after running a git rebase --continue the operation will be done and your commit will be fixed :)

Dealing with remote

If you do a rebase after your contents are already on remote (you pushed the contents), you will have to do force the push, since the history changed.

To do so, the best idea is to use git push --force-with-lease, since this command will check if no other new content was added to the branch by someone else :)

References

You can learn a lot on Git from Atlassian tutorials
Git manuals can be helpful.
The most recommended book I've seen on Git is the Pro Git

Acknowledgments

Thanks again, Martin Fieber. You've helped my growth a lot in the past year with your inputs. Especially reviewing my writings and giving great suggestions. :)

Smooth Undefined - Maybe value, maybe not?

João H. Capucho — Wed, 29 Jul 2020 07:01:33 +0000

Introduction

Functional programming (FP), for the ones used to it, is joyful, lovely, and a slice of heaven on earth. However, for mere mortals like me, it's painful to even try to understand those concepts. Weird symbols, crazy words without any meaning to my knowledge that people frequently talk about. It just feels like we're getting into a labyrinth. With no exit, to fight a minotaur, barehanded.

For a while now, I've been trying to push myself to understand these concepts and applicabilities. However, materials are usually not so beginner-friendly, and from my perspective, you don't need to become a Jedi in FP concepts to get some benefit from it, even if it's with some hacks.

In the end, I hope that you can add some basic concepts into your current codebase without much effort, neither having to change the programming paradigm you're currently using. The goal here is not to make you a functional programmer, but instead, to use a concept from it to improve your daily code.

Maybe `null` or `undefined`?

If you're a JavaScript developer, I'm quite sure you faced or wrote a piece of code that checked if a variable was null or undefined, right?

You probably have seen a lot of code that had this exact behavior:


function parseNumber(string) {
  const number = parseInt(string, 10)
  if(!Number.isNaN(number)) {
    return number
  }
  // won't make it easy to you. Figure it out
  // if it's undefined
}

And somewhere in the code, something like this is happening:

// now this returns undefined
let number = parseNumber('a10') 

console.log(c.toFixed(2))
// this will simply throw
// Uncaught TypeError: Cannot read property 'toFixed' of undefined.

And after someone added this piece of code, alarms start to fire, phones starts to ring. In the end, someone realizes that a simple check was missing. They change the code to the following and life goes back to normal.

let number = parseNumber(string) 

if(number) {
  console.log(c.toFixed(2))
}

Maybe Container

Now imagine the following scenario, instead of handing over to you the value you want or a possible undefined, you will be handed over a gift box. You won't know what's in that gift box until you open it. Now you have the Schrödinger's value.

Let's give this gift box a name. Let's call it Maybe. This is what a really basic value container would look like:

// Maybe.js
'strict mode'

class Maybe {
  constructor(x) {
    this._value = x;
  }

  static of(x) {
    return new Maybe(x);
  }

  static empty() {
    return new Maybe()
  }

  get isNothing() {
    // this._value == undefined is equivalent
    // to this._value === null || this._value === undefined
    return this._value == undefined
  }

  get isPresent() {
    return !this.isNothing;
  }

  get value() {
    return this._value
  }
}


module.exports = Maybe

This code is not following the purist Functional principles since we're exposing the value and not providing some FP operators. I will add at the end of the article some references on how to get deeper into the Functional concepts, so you can explore if you desire to do so.

This simple piece of code can improve a lot your code design. Instead of allowing null and undefined values, now you have a way to enforce your function consumer to handle those weird scenarios.

Let's refactor the code to use it. From now on, your function will return our gift box, a Maybe.

// example.js
'strict mode' 

function maybeParseNumber(string) {
  const number = parseInt(string, 10)

  if(!Number.isNaN(number)) {
    return Maybe.of(number)
  }

  return Maybe.empty()
}

And your code that calls this function will have to open the box before getting the value:

let maybeNumber = maybeParseNumber('9')

if(maybeNumber.isPresent){
  console.log(maybeNumber.value.toFixed(1)) // prints 9.0
}

Maybe is what people call a Monad. However, a monad is something more complex than this, that won't expose the value as we're doing and will provide functions like map, join, chain... so you don't need to retrieve the value from it like when we access maybeNumber.value. If you want to get some deeper context into these concepts, I highly recommend this book.

Still, from my viewpoint, these concepts can be a bit scary in the beginning and harder to introduce in a codebase that is not focused on the functional programming paradigm. Therefore, these small tweaks are a way to start introducing these nice concepts into your code while you get some familiarity with the ideas.

Hopefully, this simple idea can help you out when making some decisions on what to return when you face a situation where you most likely will return null or undefined values. Since null is considered the billion-dollar mistake, let's avoid it as much as we can.

I will write a next post on how to provide a nice interface when handling asynchronous operations, stay tuned.

Have a nice week, stay safe. Use a mask!

Credits

"JavaScript Logo" by b0neskull is licensed under CC BY-NC-SA 2.0. To view a copy of this license, visit https://creativecommons.org/licenses/by-nc-sa/2.0/

TypeScript (patterns?)

João H. Capucho — Wed, 24 Jun 2020 16:01:32 +0000

Throughout my journey with TypeScript, I've been impressed by its features. The v2 documentation is a really good improvement to the learning resources. However, some patterns are not so easy to find documented in the common ground of TS materials. Therefore, I wrote this quick article with 3 interesting things in TS, that from my perspective, can make life a bit easier :).

We shall begin!

Companion Object

I've found this one when I read Programming TypeScript. It provides a simple and easy way to enable your module customers to import the type and a factory for that type in a single import. Hence, the name "Companion Object".

It amazed me by how simple and useful this can be. This is how the module is presented:


// Currency.ts

// Here we will create a type and a variable 
// with the same name
type Currency = {
    unit: "EUR" | "GBP" | "JPY" | "USD";
    value: number;
};

let Currency = {
    DEFAULT: "USD",
    from(value: number, unit = Currency.DEFAULT): Currency {
        return { unit, value };
    }
};

export { Currency };

And this is how the module is consumed:

// index.ts

import { Currency } from "./Currency";

// Use case 1: Used as type
let amountDue: Currency = {
  unit: "JPY",
  value: 83733.1
};

// use case 2: Used as factory object
let otherAmountDue = Currency.from(330, "EUR");

console.log({ amountDue, otherAmountDue });

If we look at the index.ts file, a single import of Currency is declared. Yet, as you can see, it has two use cases. The first one as a type and the later as a factory object. By looking at the Currency.ts file, it's possible to see that we also have a single export, which is valid for both variable and type.

Consequently, with a single export and a single import, you gain a type and an object factory.. This enables you to work with both smoothly. However, things can't be so easy. Since we usually opt-in for the strict option in TS, this code throws an error:

7022: 'Currency' implicitly has type 'any' because it does not have a type annotation and is referenced directly or indirectly in its initializer.

(If you want to understand a bit more on the compiler options, this link provides an explanation to every option TypeScript contains.)

However, whenever we developers like an idea, we ~~smash the code until it works~~ find a proper path to make it reasonable. In this particular case, it is possible to use this with the strict option, by doing a small tweak. Our module will look like the following:

type ValidCurrencies =  "EUR" | "GBP" | "JPY" | "USD";

// "Small tweak": Type that will be used on
// the variable that is used as constructor
type NotExposedCurrency = {
  from: (value: number, unit: ValidCurrencies) => Currency
  DEFAULT: ValidCurrencies
}

// Here we have the things we would like to export

type Currency = {
  unit: ValidCurrencies
  value: number;
};

// Type Constructor
// Here is where we use the tweak
let Currency: NotExposedCurrency = {
  DEFAULT: "USD",
  from(value: number, unit = Currency.DEFAULT): Currency {
    return { unit, value };
  }
};

export { Currency }

By typing the factory with a private type, the variable won't have an implicit any type. This won't impact our usage since we have no interest in the Factory type. Our interest is at the factory function return type.

Applying this change, the index.ts export was still the same and it enabled the feature usage even when the compiler is set to a strict mode.

Exceptions: Java, Go, and TypeScript?

In my career, I've worked mostly as a Java Developer. In Java, you can add the Exceptions that might be thrown to your methods signatures, thus, enforcing the client to properly handle those cases.

Such a simple thing, which I've never imagined how much I would miss. For those who might have never used Java or a language with this feature, here's a piece of code with an example:

public class ThisOneThrows {

    // ThisOneThrows.java:4: error: 
    // unreported exception Exception; 
    // must be caught or declared to be thrown
    // ThisOneThrows.hereWeThrow();
    public static void main(String[] args) {
        ThisOneThrows.hereWeThrow();
    }

    public static void hereWeThrow() throws Exception {
        if(true) {
            throw new Exception();
        }
    }
}

Java compiler will enforce you to declare that the main function throws as well or that you need to wrap the call ThisOneThrows.hereWeThrow() in a try...catch block.

public class ThisOneThrows {

// public static void main(String[] args) throws Exception {
    // ThisOneThrows.hereWeThrow();
// }

// OR

 public static void main(String[] args) {
     try {
         ThisOneThrows.hereWeThrow();
     } catch (Exception e) {
         e.printStackTrace();
     }      
 }


 public static void hereWeThrow() throws Exception {
     if(true) {
         throw new Exception();
     }
 }
}

Knowing this behavior upfront is always resourceful when dealing with error handling, and I've always missed that in TS. It's a really impressive type system, especially considering the environment it runs at, but not being able to know the possible errors I could expect bugged me for a while.

However, the Go community has been dealing with this for a while. In Go, you don't have a throws declaration, the solution? Return an actual error object to the function consumer.

So, why not doing that in TS as well? (This is a controversial pattern for many, but from my perspective, as long as it increases the chances of catching an issue before the client does, it brings value to the table).

Here's one example of how you can do that in TS, again, from the book Programming TypeScript. I think I've done some really small tweaks to it, so I could make use of function return types as Type Guards.

One example of how we could define the errors and Type Guards:

// helpers.ts

// First Part: define the errors
class InvalidDateFormatError extends Error {}
class DateIsInTheFutureError extends Error {}

// Type guard for the errors
function isError(input: unknown): input is Error {
    return input instanceof Error
}

// a helper functions for dates.
function isValid(date: Date) {
    return Object.prototype.toString.call(date) === '[object Date]'
        && !Number.isNaN(date.getTime())
}

This is how we could define our logic to handle the errors:

// birthday.ts
import { 
  InvalidDateFormatError, 
  DateIsInTheFutureError,
  isError,
  isValid
} from './helpers'

function parse(
  birthday: string
): Date | InvalidDateFormatError | DateIsInTheFutureError {
  let date = new Date(birthday)

  if (!isValid(date)) {
    return new InvalidDateFormatError(
      'Enter a date in the form YYYY/MM/DD'
    )
  }

  if (date.getTime() > Date.now()) {
    return new DateIsInTheFutureError('A what?')
  }

  return date
}

function getYear(birthday: string = new Date().toISOString()): number {
  const possibleDate  = parse(birthday) // step 1

  if(isError(possibleDate)) {
      // step 2
      console.log(possibleDate.message)
      return
  }

  // step 3
  return possibleDate.getFullYear()

}

Let's start with the step 1 part, inside the getYear function. Here we call the parse function. This function, as the signature states, tries to parse a string into a Date. The signature also shows that besides the Date we want, 2 errors could be returned from the validation conditions.

By doing so, at the moment parse is executed and returns the value to the possibleDate variable, TypeScript is exactly like this:

Is possibleDate a Date or an Error? Since TypeScript can't figure it out at compile-time, it won't allow us to safely access any value. We will have to check the variable using a Type Guard. Only after that, you will be able to access the value you want.

In step two, we have our Type Guard. If the call to isError returns true, which has a return defined as input is Error, TS knows that inside that if condition, we're dealing with an Error. Consequently, the compiler will allow access to the message attribute.

(To see more on this Type Guards, take a look at this link)

Moreover, by figuring that out on step 2, TypeScript also knows that the only possible type left for the possibleDate, after executing the if condition, is the Date type. This is because both options are a subtype of Error.

Now you're allowed to access the Date functions and attributes. :)

Some Functional Programming concepts can improve this error handling in a quite nice manner, I will write a post regarding that in the next few weeks. If you're curious about it already, I can recommend reading this book.

Mostly Adequate Guide to Functional Programming is pleasant to read, with exercises so you can practice the concepts. I'm quite sure it will be a nice addition to your library.

Mapped Objects

This one I don't recall where I've encountered for the first time, but it's a simple use of generics to improve a lot our safety.

For instance, imagine you have an interface with the event types for a button, containing those events like: click, mouseover, etc.. Furthermore, imagine that you expose a client to that module, where users can subscribe to your events. Have you ever seen something like .on('click', callbackFunction)?

Maybe you also want to tell them the types they might expect in their callback functions. It would be way better if we had a type system that could provide this information, right?

This pattern can help your module users a lot on those questions mentioned above. I can't stop thinking of a younger version of me while learning the JS basics. The countless number of times I had to look up on MDN which events were available on a given type of element.

With this pattern, you can derive types that will improve your IntelliSense and autocomplete features by simply using Generics.

// redis.d.ts

type Events = {
  ready: void
  error: Error
  reconnecting: {attempt: number, delay: number}
}

type RedisClient = {
  // subscriber function
  on<E extends keyof Events>(
     event: E, 
     f: (arg: Events[E]) => void 
  ): void
}

And this is how this would be used:

// redis.ts
import Redis from 'redis'

// Create a new instance of a Redis client
let client: RedisClient = redis.createClient()

// Listen for a few events emitted by the client
client.on('ready', () => console.info('Client is ready'))

If we take a look at our subscriber function inredis.d.ts, TypeScript will realize that the event can only be one of the types 'read' | 'error' | 'reconnecting', since the values are keys from the Event type. It will also add types and dynamically validate the function arguments since it will be based on the event type you select.

This is also resourceful whenever you add a new key into that type, all your clients will be able to see the new addition, and if we didn't break anything, use that new event :)

Time to say goodbye :)

Hopefully, this article can bring some ideas to your future coding, even if it's to avoid using these concepts.

Feel free to comment and bring your ideas to this article.

References and Resources

TypeScript Docs
Programming TypeScript
FrontendMasters - TypeScript; Some material related to this one is presented for free in here
TypeScript Deep Dive
Udemy - Understanding TypeScript

Appreciation

Martin Fieber, that helped me a lot with my TypeScript learnings and had a lot (I mean, a LOT) of patience with me.

Apache Kafka 101 - Introduction

João H. Capucho — Tue, 14 Apr 2020 07:28:23 +0000

Disclaimer: the intention behind this post is to do a brain dump of some understandings I have on Kafka and to make that useful by sharing it. I'm not a Kafka expert whatsoever, and if there's anything wrong in here, please, comment so I can fix it. Anything that can help improve this text is welcome. With that being said, let's start the journey :)

Sync vs Async communication

Before diving into Kafka, let's briefly go through the concepts of synchronous and asynchronous communication for the ones that never heard about it. If you know the difference you can just skip this part.

To start off, let's take an example of synchronous communication that is a client app sending an HTTP request towards a REST API service. In this case, the client opens a connection with the server via the HTTP protocol and sends some data that will be received and processed by the server, which in turn will respond to the client with the result. If the client, by any chance, does not receive a response within an expected time, an error will be thrown (the beloved timeout). This architecture is quite simple and usually suits well the majority of the cases where a client needs to communicate with a server.

However, imagine that you work for an e-commerce company and you have to implement the purchase button of the website. Would a regular synchronous request be enough to handle this case?

If the customer clicks purchase you will probably need to check whether the warehouse has stock of the products, the user's credit card is valid, the delivery address exists, etc, but eventually a response will have to be given to the customer telling them that the purchase has been finished successfully. And even though you told the user so, the purchase process won't stop there. The logistics will need to separate all the items of the purchase and assemble the package, a delivery company will fetch them, and some tracking information will be sent to the user as the progress changes. And these things are naturally asynchronous. The user doesn't need to be waiting for all of them to happen before he knows his purchase is done. Otherwise, online shopping would be a huge pain.

Here is where asynchronous communication enters the game. By using it you are able to leave things such as emails, logistic integrations, tracking, that don't need to be immediately taken care of for later, making sure that the user will receive the purchase feedback as soon as possible.

Through messaging systems, we enable senders and receivers to communicate without the restriction that both must be available listening to the message at the same time. As the following image shows, the sender can produce a message and whenever the receiver is available, it will read and handle it.

In this mentioned scenario, imagine that instead of doing REST API calls to each system informing that an order was placed, you could just send a message to the queue saying "Hey, someone created an order, here's the information you might be interested in". Or even, "hey, the tracking position of that package has changed, you might want to do something about it".

How beautiful would it be if all of your services that rely on this event could do whatever they need (and maybe even post follow-up messages?) whenever they're available to do it? Your email service would be really happy to be able to read this message calmly and send the update to the user when all its dependencies are available, isn't that so? That's what we want to achieve with asynchronous communication.

So, what about this Kafka?

What's Kafka?

Apache Kafka is a distributed streaming platform. It's built to be resilient and fault tolerant.

You can achieve many use cases with Kafka, examples of it's usage could be:

Messaging
Log gathering
Stream processing
etc.

In this article we'll go through an overview of some Kafka core concepts to enable you to have some understanding of its main features with the focus on using it for messaging and distributed systems communication.

Topics
Partitions
Brokers
Producers
Consumers
Consumer Groups

With all these concepts, Kafka's proposition is to deliver a structure that enables horizontal scalability and high performance.

Core Concepts

Topics

A topic is where your messages will be sent to (produced). Whoever wants to read this piece of information will read it (consume) from the topic.
Inside Kafka, you may have as many topics as you want and each topic contains a given number of partitions (defined by you when creating the topic). So, whenever producing a message to a topic it will be written in a given partition of the topic, randomly selected unless a key is provided.

(image from Kafka docs)

Within the topic, data is kept for some time. The default value is 7 days, but it can be extended. There are cases where these messages are kept on the topic for one year. One important thing here is that after a message is written in a given partition, you will no longer be able to change it. Being able to store the data for such a long period of time, enabling you to reprocess the messages, if necessary, is what makes Kafka reliable. After seeing some use cases for it in production, I can safely assure that this will be life-saving in some cases.

Partitions

Within each partition the messages will be ordered, receiving an incremental id called offset in Kafka. A consumer is basically pointing to the last offset it was read, whenever a new message is consumed, this pointer moves forward to the next message. This process is an offset commit, which is always incremental.

(image from Kafka docs)

Brokers

When using Kafka, the infrastructure you will have is a Kafka Cluster. This cluster is basically a bunch of servers, in our case, the brokers. So, what is this broker for?

Remember when we said that a topic consists of many partitions? To be more resilient and handle server failures, each broker will contain a certain topic partition.

When connecting to a broker (Bootstrap server), you're connected to the entire cluster. Every Kafka broker is a bootstrap server which knows about all brokers, topics and partitions. When the Kafka Client connects to a Broker, it will receive a list of all brokers and can eventually connect to one of them, if needed. Brokers are identified by an ID.

Let's imagine a cluster with 3 brokers and one topic (called A). To achieve resilience in the delivery of the messages, we need some replica of the information, right? Just in case something goes wrong in one of the servers.

This is called replication factor in Kafka and topics should have a replication factor > 1, usually between 2 and 3.

In this example, we have a replication factor of 2, therefore, each partition is present in 2 brokers. So, in the case a broker becomes unavailable, you're still able to read the partition from another broker. However, if we had a replication factor of 3, we could afford even losing two brokers and we would still have the three partitions in one broker to work with.

How replication works?

At any given point in time, one broker (and only one) can be a leader for a given partition. The other partitions are called In Sync Replicas (ISR). When writing to a topic, only the leader can provide data for a partition and the brokers will need to sync the data.

Compression

You can enable data compression in the Broker configuration to reduce the size of messages, see more in here. Search for compression.type.

Producers

To enable ourselves to use Kafka we need to be able to read and write messages to our loved topics, isn't that so? Therefore, we need to have an API that makes that possible.

The producers are the ones responsible to write data to the topics (remember that this data will actually be written into a partition). If we want to summarise what a producer is, that would be it. However, if we only had this in the API it wouldn't be resilient.

For that, a producer is able to detect a failure in the broker and recover from that producing to another broker. Like it has been said, keys can also be sent within the messages to enable ordering. If no key is present, the data will be sent to a partition on a round robin selection. However, with the key, a hash will be made out of that and the key will always be sent to the partition of that given hash.

This is to enable order in messages. One thing to consider is that the number of partitions can never change otherwise the hashing mechanism will change and the guarantee won't be there for you anymore.

Acknowledgment

It's possible to choose to receive acks when producing messages.

Ack Value	Result
0	No ack is needed
1	Wait for leader to ack
all	Leader + replicas must ack

Something worth mentioning is that it is possible to configure retries in Kafka, with backoff configuration and timeout. But remember, with retries, Kafka could send messages out of order if you have that configured, be aware.

Something else is that you can achieve idempotent producers:

[...] The idempotent producer strengthens Kafka's delivery semantics from at least once to exactly once delivery. In particular producer retries will no longer introduce duplicates. [...]

Consumers

Now we reached the point where we'll read the messages from a given topic. Consumers know which broker to read from and each consumer will read from a given partition (or multiple partitions as well, more on this later). By not having multiple consumers in the same partitions, Kafka ensures that the message will not be processed by two consumers. Here's the API for reference.

(image from Kafka docs)

Consumer Groups

Consumers are tied to a consumer group. As we can see in the image above, each server contains Partitions (P#, where # is the partition number) and each Consumer belongs to a given Consumer Group. Each consumer from Consumer Group A reads from two topics, but each consumer from Consumer Group B reads from a single partition. When each consumer reads from a single partition and you send messages with keys, you can ensure that consumers will read the message from the partition it reads from in an ordered manner. Important, if you have more consumers than partitions, some consumers will be inactive.

Offsets

Consumer offsets are for a given consumer group and they're stored by Kafka when committed. This commit happens when the data received by the consumer is processed.

Consumers choose when to commit these offsets. Three types are available:

Delivery	Impact
at most once	As soon as the message is received, offset is committed. Process failures means lost messages.
at least once	Commits the offset when message is processed
Exactly once	I'll leave a blog post from Confluence here since it's a bit more complex.

One thing to be aware of is that when using at least once configuration, you may read the message more than once, so be prepared for that.

Libraries

So, I've been working for the past year with Node; therefore, the library we use is KafkaJS. An open-source library, with some really good features and easy to start working with. If you want to get started with the basics, check it out! :)

Also, follow this doc to enable yourself to run Kafka locally (with Docker) and be happy :)

References

https://kafka.apache.org/intro
https://kafka.apache.org/documentation/
https://kafka.apache.org/24/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html
https://kafka.apache.org/24/javadoc/index.html?org/apache/kafka/clients/consumer/KafkaConsumer.html

JS Library: https://kafka.js.org/docs/getting-started

Appreciations

Túlio Ornelas, for having a lot of patience when helping me while I was building some communications using Kafka.
Gabriel Gomes, for doing such a neat review on this text and helping me improve the readability (also fixing my English mistakes hahaha).

DEV Community: João H. Capucho

Thoughts on API Design

Goals

Error handling

Be predictable

Adaptability

Pagination

Documentation

Versioning

Request tracing

Idempotency Key

5 books I'm happy that I've read in 2022

The Passionate Programmer - Chad Fowler

Laziness does not exist - Devon Price

Atomic Habits - James Clear

The sexual politics of meat - Carol Adams

Focus - Daniel Goleman

PostgreSQL - Partitioning & Inheritance

Introduction

Partitioning your data

Using inheritance

Changing parent table

Conclusion

A Philosophy of Software Design - Review

Introduction

What I've liked

What I didn't like as much

Simple takeaways

Conclusion

Redis for beginners, part 1

Redis

Basic Commands & Data Structures

Deleting data

Expiring data / Time to Live (TTL)

Hashes

Lists

Sets

Sorted sets

References

The zen of focus

The start of the journey

Learnings

References

Rotina e sanidade, parte 1

Why are you using cronjobs?

Context

Asynchronous queues to the rescue

Conclusion

Adendum

Image

A quick guide to fix your Git mistakes :)

Introduction

Basic rebase concepts

Interactive rebase to the rescue

Renaming a commit

Dropping a commit

Editing a commit

Dealing with remote

References

Acknowledgments

Smooth Undefined - Maybe value, maybe not?

Introduction

Maybe null or undefined?

Maybe Container

Credits

TypeScript (patterns?)

Companion Object

Exceptions: Java, Go, and TypeScript?

Mapped Objects

Time to say goodbye :)

References and Resources

Appreciation

Apache Kafka 101 - Introduction

Sync vs Async communication

What's Kafka?

Core Concepts

Topics

Partitions

Brokers

How replication works?

Maybe `null` or `undefined`?