DEV Community: Gerardo Sandoval

Backend decisions: Webhooks, File Uploading, and API Versioning

Gerardo Sandoval — Tue, 18 Apr 2023 15:16:28 +0000

As a backend engineer, we face many decisions when building APIs, some of them are easy to make and some others are not that easy because of the several existent solutions for a specific problem, deciding which one is the best fit can become a nightmare if you do not assess it correctly. In this article, we'll cover three common topics: handling webhooks, file uploading, and API versioning, and for each one I'll describe few facts we can identify to help us easily make the correct decision.

Webhooks: One Endpoint or Many?

Webhooks are a way for your API to notify other services or get notified of events that occur. When designing your webhook system, one question that often arises is whether to use a single endpoint to handle all possible events or to create separate endpoints for each event.

The advantage of a single endpoint is that it simplifies the codebase and reduces maintenance overhead. All webhook payloads can be parsed and processed in one place, this is great! However, this can become unwieldy as the number of events grows, leading to complex branching logic, hard to debug or track, and potential performance issues, this is not so great...

On the other hand, having separate endpoints for each type of event allows for more fine-grained control and easier scaling. Nonetheless, this approach can lead to a bloated codebase and increased maintenance overhead, a way to reduce that overhead is to have an adequate separation of concerns in your endpoints. Always aim to separate or group your endpoints by business logic, example:

If you have an integration with a payment gateway, you can separate and group your endpoints by specific models: payments, invoices, subscriptions, etc.

Ultimately, the decision depends on your specific use case, but if we could write a quick rule of thumb, it would be the next:

If you have a small number of events and don't anticipate many more, a single endpoint may be the simpler solution.
If you have many events or anticipate adding more in the future, consider using separate endpoints for each event.

So, basically it just depends on the number of webhook events that you are handling.

File Uploading: Synchronous vs. Asynchronous

When it comes to file uploading, there are different ways to do it, each with its own advantages and disadvantages. One common decision is whether to upload files synchronously or asynchronously, should we upload files before submission or wait until the submit buttons is clicked?

Synchronous file uploading means the file is uploaded when the form is submitted. This approach is simpler to implement, as it doesn't require additional background processing. However, it can lead to slow response times if the file is large, as the user has to wait for the file to finish uploading before getting a response.

Asynchronous file uploading means the file is uploaded in the background, even before the form is submitted. This approach allows for faster response times and a better user experience. The user does not have to wait until the file is uploaded to see the new page. However, it requires additional background processing and can be more complex to implement.

When deciding which approach to use, consider the size of the files being uploaded and the user experience you want to provide.

If you expect large files, asynchronous uploading may be the better choice. You can upload the files even before the form is submitted, this would give the user a great experience.
If you want to provide a simple and straightforward user experience and make a quick implementation, then synchronous uploading may be the better option, just take in mind that the file upload time depends on the bandwidth of the connection and can sometimes take long, some users can try clicking somewhere else or reloading the page because they believe that there is a problem with the webpage, and then mess everything up.

API Versioning: Why and How

API versioning is the practice of creating new versions of your API as it evolves over time. This allows you to add new features, fix bugs, and make breaking changes without affecting existing clients. Even when your API is private and it serves only your own client, these practices may be needed.

Note that is a good practice to follow a versioning schema like semver.

There are different approaches to API versioning, but the most common ones are adding version numbers to the endpoint or using custom headers to indicate the version. Adding version numbers to the endpoint makes it easy for clients to switch between versions, but can lead to cluttered code and bloated APIs. Using custom headers can provide a cleaner API, but may require more work on the client side to switch between versions.

When deciding how to version your API, consider the tradeoffs between simplicity and flexibility.

Adding version numbers to the endpoint is a simpler approach, but can lead to cluttered code. More code, more work.
Using custom headers can provide a cleaner API, but may require more work on the client side.

In conclusion, easy decisions can become great enemies because we tend to overthink them. As a backend engineer, you'll face many decisions when building APIs. Whether it's handling webhooks, file uploading, or API versioning, there are advantages and disadvantages to each approach. By considering your specific use case and the tradeoffs involved, you can make informed decisions that lead to better APIs.

Understanding ActiveRecord Nested Transactions

Gerardo Sandoval — Thu, 02 Jun 2022 19:27:23 +0000

As it reads in rails documentation -transactions are protective blocks where SQL statements are only permanent if they can all succeed as one atomic action.- You can take advantage of this behaviour to ensure that a number of statements are executed altogether or not at all.

There is plenty of resources that talk about Transactions, but this one focuses only where it seems to lack clarification: nested transactions. The official documentation on this topic can be confusing and because of that, here are some insights to fully understand how nested transactions behave and how to use them.

What are nested transactions?

Taking the simplest definition found online: - A nested transaction is used to provide a transactional guarantee for a subset of operations performed within the scope of a bigger transaction. Doing this allows you to commit and abort the subset of operations independently of the larger transaction. -

Most databases don't support true nested transactions and the only database that supports them is MS-SQL.

In order to get around this problem, Rails transaction will emulate the effect of nested transactions, by using savepoints

When are nested transaction required or used?

Nested transactions can exist when we have models pointing to different databases. Each transaction evaluates and can create a single database connection, and rails does not distribute transactions across database connections, therefore, it is best if we nest transaction to imply the connection to different databases.

Another example that I encounter the most is when we are working with external libraries or gems that process specific code inside a transaction which we can not fully control and in consequence we wrap around them with another transaction.

More on distributed transactions

Quoting from rails source code: - A transaction acts on a single database connection. If you have multiple class-specific databases, the transaction will not protect interaction among them. One workaround is to begin a transaction on each class whose models you alter: -

 # https://github.com/rails/rails/blob/main/activerecord/lib/active_record/transactions.rb#L62

  Student.transaction do
    Course.transaction do
      course.enroll(student)
      student.units += course.units
    end
  end

From the rails source code: - This is a poor solution, but fully distributed transactions are beyond the scope of Active Record. -

How nested transactions behave?

Nested transactions are intuitive and represent the normal transaction behaviour, if one statement fails, all fail.

In detail, exceptions inside a transaction block will force a ROLLBACK that returns the database to the state before the transaction began and the exception will then be propagated to the parent transaction.

ActiveRecord::Rollback exception will trigger a database ROLLBACK when raised, but will not be re-raised by the transaction block and therefore will not be catched by the parent transaction. This exception is only meant to be used deliberately in exceptional situations.

Transactions are meant to silently fail if ActiveRecord::Rollback is raised inside the block, but if any other error is raised, the transactions will be rollbacked and the exception will be passed on.

Explaining the documentation examples.

Looking into the rails docs first example:

User.transaction do
  User.create(username: 'Kotori')
  User.transaction do
    User.create(username: 'Nemu')
    raise ActiveRecord::Rollback
  end
end

An ActiveRecord::Rollback exception is being raised in the inner transaction, and therefore it should avoid the creation of the users, but as all database statements in the nested transaction block become part of the parent transaction and the ActiveRecord::Rollback exception in the nested block does not carry up a ROLLBACK action to the parent transaction, both users are created. As I wrote before, ActiveRecord::Rollback exceptions will be intentionally rescued and swallowed without any consequences, and the parent transaction won't detect the exception.

If we take the same example, but we raise a different exception:

User.transaction do
  User.create(username: 'Kotori')
  User.transaction do
    User.create(username: 'Nemu')
    raise ArgumentError
  end
end

This will work as expected. The transactions are nested and joined correctly in only one connection (this is default behaviour), therefore, Nemu and Kotori won't be created. It also doesn't matter where the error is raised, if it is raised in the parent or child transactions it will still rollback all statements.

Creating real nested sub-transactions

We can achieve a different result by creating real sub-transaction by passing requires_new: true to the inner transaction.

User.transaction do
  User.create(username: 'Kotori')
  User.transaction(requires_new: true) do
    User.create(username: 'Nemu')
    raise ActiveRecord::Rollback
  end
end

This would treat each transaction separately and if an exception is raised in the inner transaction the database rolls back to the beginning of the sub-transaction without rolling back the parent transaction. Therefore the example above would only create the Kotori user.

The documentation also gives just a bit of information about the 2 options that we can pass to the transaction method: joinable and requires_new.

This options help us treat nested transactions as individual database connections and therefore avoid dependancy between parent-childs transactions, also when intentionally raising an ActiveRecord::Rollback exception. Each option is intended to be used depending on the nested hierarchy level of the transaction.

joinable: default true. Allows us to tell the outer transaction if we want the inner transaction to be joined within the same connection. If this value is set to false and the inner transaction raises an exception it wont affect the outer transaction.

User.transaction(joinable: false) do
  User.create(username: 'Kotori')
  # child transaction result wont affect the parent transaction
  User.transaction do
    User.create(username: 'Nemu')
  end
end

requires_new: default nil. Allows us to tell the inner transaction if we want it to run in a different connection. If this value is set to true and an exception is raised, it wont affect the parent transaction.

User.transaction do
  User.create(username: 'Kotori')
  # next transaction result wont affect the parent transaction
  User.transaction(requires_new: true) do
    User.create(username: 'Nemu')
  end
end

So, this two options are meant to be used to run transactions in individual database connections depending on the nested hierarchy that you can control.

https://api.rubyonrails.org/classes/ActiveRecord/Transactions/ClassMethods.html
https://dev.mysql.com/doc/refman/en/savepoint.html
https://dotnettutorials.net/lesson/sql-server-savepoints-transaction/

Avoid code oversimplification with RoR

Gerardo Sandoval — Tue, 09 Mar 2021 05:44:23 +0000

Through my years of working with Ruby on Rails, I've seen some common practices that I believe are not required and add complexity to the code base.

This really simple practices look like they do no harm, but can make code less readable, add verbosity and insert unnecessary code.

One expression methods

Methods are supposed to encapsulate a set of processes or expressions, but sometimes they are wrongly used to define a single logic unit. Most of the time, developers create this methods to make the code more readable.

Consider the next example:

class User < ApplicationRecord
  before_save: setup_user

  def setup_user
    update_complexity
  end

  def update_complexity
    self.complex = true
  end
end

After looking at the example above, think about why should we separate the logic in two different methods when we can just move the assignation inside of the setup_user method.

  def setup_user
    self.complex = true
  end

The first example is easier to read for the most of us, but it adds complexity to the model by adding one more method that is just redirecting to another one without doing anything else.

The second example is readable enough to quickly understand the code intentions and keeps the model clean.

Devs should always account for readability but also keep complexity in mind, it is easier to read a one line raw statement instead of dealing with fat models.

Prefixing the model name when naming methods, variables and constants

Naming variables, constants and methods can be such a basic but painful thing to do, the name you choose is not always understandable enough for others. It is not great also when names mean more that they should.

See the next example:

class User < ApplicationRecord
  USER_DEFAULT_ROLE = 'admin'
  ...
end

In the example above we define the user's default role as a constant, we are being very specific in the name so we know that the default role belongs to the user model, but... why?

If we call this constant from outside the scope of the model, we would call it like this User::USER_DEFAULT_ROLE, and as you can see, we are repeating the word "User". It may not look like it can affect the readability of the code, but imagine having several constants in the same model or just having long model and constant names...

class CompanyConciliationFee < ApplicationRecord
  COMPANY_CONCILIATION_FEE_DEVELOPMENT_COST_RATIO = 0.59
  ...
end

Look at how long it would look if we call this invented constant from outside the model scope!
MaintenanceFee::COMPANY_CONCILIATION_FEE_DEVELOPMENT_COST_RATIO

Yeah, the example might look like I wrote the first thing that came to my mind... but that's basically how naming works.

Another reason people do this is because they are thinking to use the constant within the scope of the model, where it wouldn't need to prefix the class name. This may be more straightforward but remember that all constants are always associated to the model they are defined into.

Prefixing the model or class can happen with constants, variables and methods, and even when they feel necessary, keep in mind that they will clutter your code.

Endless class reference

Endless class reference refers to something similar to one expression methods but through classes.

Sometimes we break down the code structure to what we think are the necessary classes to get the job done, I.E. services, presenters, PORO's or whatever type of code encapsulation we can use, but sometimes we lengthen the logic by using more than we need, and therefore increase complexity.

Look at the next example:

class UserGenerator
  def initialize(user)
  ...
  end

  def call
    UserAvatarUploader.new(user, avatar).call
  end

  def avatar
  ...
  end
end

class UserAvatarUploader
  ...

  def call
    AvatarUploader.new(:user, user, avatar)
  end
end

class AvatarUploader
  ...

  def call
    user.update(avatar: avatar)
  end
end

If you follow the process to create a user and upload an avatar, you can observe that we are actually using 3 different services to do one simple thing, upload an avatar.

This example is a very basic and simplified version of what happens in production code bases that are used by thousands of people every day. User's don't notice, but devs suffer when debugging only to find out that they have to follow an endless class reference to figure out where in the code the avatar is uploaded.

We don't need all those classes. Keep it simple... if you only job is to upload an avatar, follow the KISS principle and keep it in only one service, you don't need the others! It may feel to you that you are decoupling service responsibilities, but what you are actually doing is making your jr. teammates life very unpleasant one.

And there it was... a few cases that I often see in Ruby on Rails codebases that can be avoided. What do you think? Do you know about another common simple practice that can be avoided to simplify our code?

Minimal Net/HTTP client template

Gerardo Sandoval — Sun, 17 Jan 2021 21:35:55 +0000

Have you ever wanted to fetch information from an API and realize that there is no existent library or gem for that specific service?

If you find yourself in that situation and are planning to build your own client, you may already know that there are plenty of third party HTTP client libraries to use (Faraday, HTTParty, rest-client, etc.)... And yeah, without issues you can use any of those to build your client, but also you can achieve that with minimal effort using Net::HTTP.

Net::HTTP may not be the first option for most developers, its implementation is not the simplest or the easiest, but it definitely capable of building a robust client without the need of requiring additional code.

Here's an example of a client built with Net::HTTP.

Net::HTTP Client example

require 'net/http'
require 'uri'

class Client
  API_ENDPOINT = "http://api.example.com"

  def initialize(api_key, secret_key)
    # set credentials or token if required
    @api_key = api_key
    @secret_key = secret_key
    @api_uri = URI(API_ENDPOINT)
  end

  def get_request(opts)
    request(
      Net::HTTP::Get.new("/users"),
      opts # additional request options defined by the API I.E. { page: 1, limit: 30 }
    )
  end

  def post_request(opts)
    request(
      Net::HTTP::Post.new(api_uri.merge("/create_user")),
      opts # User options that are required to create the user
    )
  end

  private

  def request(net_request, data = nil)
    # execute and store the request response
    @response = api_client(net_request, data)
    # parse response
    parsed_response = JSON.parse(response.body)
    # return the parsed response, otherwise raise an error
    return parsed_response if response_successful?

    raise CustomError, "#{parsed_response['code']} - #{parsed_response['message']}"
  end

  # Receives a valid Net::HTTP request object and data to set as the request body
  def api_client(net_request, data)
    Net::HTTP.start(api_uri.host, use_ssl: use_ssl) do |http|
      # Set headers
      headers.each { |key, value| net_request[key] = value }
      # Set request body from the data argument
      net_request.body = data.to_json if data 
      # Execute the request 
      http.request(net_request)
    end
  end

  def headers
    # define headers here
    {
      'Content-Type' =>  'application/json',
      'Authorization' => #<auth token or keys>,
      ...
    }  
  end

  def response_successful?
    response.code == '200'
  end
end

Usage

The client usage is very simple, just initialize the client, call the specific request method and pass the required options.

  client = Client.new(*api_credentials*)
  response = client.get_request({ page: 1, limit: 30 })

Adding more request endpoints

You just need to add a new method to the client with the specific Net::HTTP request object and pass the parameters required by the API.

That's it!... a simple Net::HTTP client template for you to use.

Use your brain!

Gerardo Sandoval — Mon, 28 Dec 2020 20:44:00 +0000

The idea of programmers sole job is googling stuff has gained popularity over time, and it actually has a great amount of truth in it, but in order to be a great programmer, you need to absorbe the knowledge you google for, and that may not be as easy as it reads.

Some programmers use google, stack-overflow or any other source to find insights on how to solve a problem, and most of them follow this as the first thing to do after a new issue has been assigned to them. But hey! Guess what? It shouldn't be!

Your plan A or first thing to do should always be using your brain... Think first and search after (if needed).

Plan B will be to google for some insights and/or maybe the answer. More times that not you'll be using the fallback plan (plan B), and thats OK, but always try to use the fallback plan after you apply plan A.

Two reasons of why you are not learning anything new:

If you don't put some effort on thinking solutions... Your brain will never be used to associate and remember this information.
You are a lazy ass that avoids heavy thinking and prefers to google stuff out first.

Tips:

Think before googling!
Read, analyze, implement, and if it works -> document.

Code review culture does matter

Gerardo Sandoval — Mon, 28 Dec 2020 20:32:03 +0000

Code-Review culture does matter.

Some time ago I found myself working for a company with great social and professional culture, working there felt great, but... it didn't last long.

I realized that there were some organizational issues that were impacting greatly on the engineering team performance:

Definition of responsibilities was lacking
Devs were mainly working alone with poor team communication
Only few development standards and practices were set.

Among those issues, I was always struggling with an specific problem: getting my PR's reviewed, approved and merged.

Deliverance was so slow that some PR's were stuck in the in review column for more than a month, those developments were a lost couse, outdated and with several conflicts, more work was needed to update them that the work needed to develop them in the first place.

I found myself working in a startup with a very poor code-review culture.

After reading and investigating a while I found that there are some practices that can be adopted to strengthen your code review culture, boost deliverance and build a better team. The next practices are based on a talk by Derek Prior (link) and some other readings.

¿How to prepare for and improve your code review culture?

As a team, decide your process and what to expect of reviews and make responsibilities clear:
- Number of reviews before merging
- Who can merge?
- Can I merge my own PR?
- How should I ask for review?
- Who and why should be reviewing my code?
Take in mind that code review does not include QA
Adopt a styleguide and outsource it

¿What are the benefits of a strong code review culture?

Better code - quality discussions improve code quality
Better developers - Conflict are good for learning, in a healthy way
Team ownership - Remove modular dependencies and win developer's versatility
Healthy debate - More meaningful technical discussions in a proactive culture
A better place to work

Rules of Engagement

¿What to do as an author?

Provide sufficient context. Explain your changes and avoid context hunting. If content is king, then context is God (Gary Vaynerchuk).
Explain why you chose that solutions, not everyone thinks like you.
¿You learned something new and useful? Document it! It can help others.

¿What to do as a reviewer?

Ask, don't tell. Write your observations as a question instead of demanding a change. Asking the right questions the right way results in a better technical discussion.
Offer compliments and be positive. Comment the code that you find useful and of quality with a compliment.

In practice

¿How to handle disagreements?

Agree to disagree. There's different ways to get things done.
Create a discussion and be open to new alternatives. If you don’t know which is better, ask.
Ask for a tie breaker opinion.

¿What should I be reviewing anyway?

Single responsibility principle
Naming, easier to understand and discuss
Complexity
Test Coverage
Look for bugs involving your area of expertise

Interested? Read these articles for more tips and information:

https://www.youtube.com/watch?v=PJjmw9TRB7s
https://thoughtbot.com/blog/five-tips-for-more-helpful-code-reviews
https://chelseatroy.com/2019/12/18/reviewing-pull-requests/