DEV Community: Jean-Michel Gigault

Direct File Upload With Paperclip and S3-compatible Object Storage

Jean-Michel Gigault — Wed, 21 Apr 2021 08:47:35 +0000

As usual, I will take the opportunity of describing technical solutions to go further with the topic and talk about the why, the how and the whom, with the widest inclusive example I was able to cover. If you are only looking for the technical guidelines, go directly to the third section, or even quicker: open attached source code ;-)

That being said, my name is Jean-Michel and I am a committed engineer at Per Angusta. We are building softwares for big companies to solve their need of Procurement Performance Management. We are coding with Ruby on Rails.

Direct File Upload principles

Implementing a file upload functionality is not an obvious challenge due to the multiple security concerns (summarised by OWASP) and the technical implementation, the later being the sole topic of this blog post.

Modern applications running on services like Heroku often have a connection to an Object Storage system behind the scene, in order to store files such as images and files uploads. Those applications therefore can face into multiple issues when they implement a simple upload interface that require data to transit through its layer. A user trying to upload a 100Mb file through such mechanism would lock a thread of the application during the file upload process, resulting in a reduced threads pool for other users to request the application (increased queue latency) and request timeouts due to hard limits of the hosting provider (the well-known H12 error on Heroku).

Be even more naive because request timeouts do not only occur with huge file uploads or downloads. Since the COVID-19 pandemic started, working from home has become an even bigger concern for application editors providing features based on data transfer, because of the bandwidth being more and more based on users’ personal Internet access. At Per Angusta, we have been able to monitor an increased latency due to Internet bandwidth, especially with US-based users who are usually less well equipped with Internet access than Europeans.

Direct File Upload consists in isolating the expensive data transfer between the Client and the Object Storage, bypassing the layer of the Application. The latter does only pilot the flow, handles the validations and handles expensive operations only in its controlled network. By “controlled network”, I mean both application and Object Storage are efficiently connected together with a stable bandwidth (for instance when both are hosted in the same data center).

This is exactly the same purpose of pre-signed expiry URLs commonly used to display images on, or download files from, websites that use modern Object Storage connectors (such as Rails’ ActiveStorage component).

The benefits of implementing Direct File Upload bring you the opportunity to increase the file size limits, which can be considered as a feature and not only a performance improvement.

Why ActiveStorage does not fit with our needs

And why to write this blog post today meanwhile Paperclip is deprecated since 2018 by the maintainers in favour of ActiveStorage?

To start with, it’s worth noticing that Paperclip is not dead thanks to new active maintainers working on a fork named kt-paperclip. However, you would be well advised to start using, or to migrate to, ActiveStorage if you can. As part of Rails, this component has a guaranteed roadmap and qualified maintainers.

Thus, you may have good reasons for not choosing to migrate, as we decided until now:

The lack of customisable Blob#key configuration, referred as “path” in Paperclip, and which my colleague Frantisek is by the way tackling in PR #41004 (please upvote!). Our use case is a base specification: As per our multi-tenancy strategy, we need to isolate files of clients in a clear manner , prefixing all keys with clients identifiers. Relying on ActiveStorage would force us to be dependent on the database stored keys to determine the files owners, resulting in painful (or even risky) maintenance operations such as isolated backups, archiving and reversibility procedure.
ActiveStorage has a great design but stands totally different from Paperclip’s. As well as with ActionText, data is stored in separate and dedicated tables active_storage_blobs and active_storage_attachments, whereas Paperclip invites the domain-level models to store the attributes (there are alternatives, as chosen above in my example of implementation). Having not to rework the models and their interactions can be considered as a pragmatic decision. Especially if you deeply embraced Paperclip’s design, there is no absolute obligation, whenever the association “Paperclip + deprecation” is scary.
Some other developers reported us a few edge cases or painful implementation of blob manipulation, such as post-processing. Because we are not concerned, this may be speculating :-) Those issues are not specific to ActiveStorage’s pattern, but another reason why keeping a working solution in place may remain a good choice until you face into a real issue with it.

Implementing Direct File Upload with Paperclip

Prerequisites:

Paperclip (forked or even legacy)
Fog::Aws and AWS S3 SDK
S3-compatible Object Storage
jQuery, jQuery-UI, jQuery File Upload
PostgreSQL (if the primary key is part of the path builder, which is the default behaviour through :id_partition key)
Multipart upload is not supported

Specifications:

A model called Attachment, on which is implemented Paperclip with has_attached_file :upload.
Domain-level models have many attachments through a polymorphic association named “attachable”.

class Attachment < ApplicationRecord
  belongs_to :attachable, polymorphic: true, optional: false

  has_attached_file :upload
end

To have the most inclusive example, this post and its related code source make the implementation of Direct File Upload compatible with Post Processing, Fingerprint, URI Obfuscating, :path setting that includes the primary key, and also multiple file uploads.

has_attached_file :upload, 
  styles: { medium: '300x300>', thumb: '100x100>' },
  path: ':tenant/:class/:attachment/:hash/:style/:filename',
  hash_data: ':id/:extension/:fingerprint/:updated_at'

1. Request pre-signed URL

Let’s start with the assumption that your front page has a form with a file input, to which you have attached an event listener on submit:

function directFileUpload() {
  const form = $(this)
  const input = form.find('input[type="file"]')
  const files = input.prop("files")

  $.each(files, function (index, file) {
    // start upload process for each single file
    obtainPresignedURL(file)
  })
}

$(document).ready(function() {
  $('form[data-direct-file-upload="true"]').on('submit', directFileUpload)
});

For each single file, we start the sequence of requests by obtaining a pre-signed URL from an endpoint of the application with the following specifications:

Method: POST
URL: /direct_file_uploads
Request parameters: The file name and file type, as provided by the browser from the file input
Response: A pre-signed URL (upload_url) and a unique identifier (upload_key)

function requestPresignedURL(file) {
  $.ajax({
    url: "/direct_file_uploads",
    method: "POST",
    dataType: "json",
    data: {
      authenticity_token: <VALID_AUTHENTICITY_TOKEN>,
      file: {
        name: file.name,
        type: file.type
      }
    },
    success: function (response) {
      // go to step 2
      uploadFileUsingPresignedURL(file, response.upload_url, response.upload_key)
    }
  })
}

A controller handles the request and prepares a pre-signed URL with the following specifications:

The upload_key is partially random to prevent from duplicates and to obfuscate the path to prevent clients from hacking other uploaded files.
The upload_key retains the file name to strengthen the step 4, and also to facilitate debug.
The actual path where the file will be uploaded (upload_path) must be partially built with a specific and hard-coded prefix (understand forced by the backend) in order to prevent clients from hacking and accessing other locations available in your bucket (there is no way to access parent directories using a path with S3, as you would expect on Unix using “..”).
The file type is appended to headers in the field “Content-Type” to retain the typology of the file.
The expiration of the link depends on your use case: The greater the size limit is accepted, the longer the expiry time must be. Think about choosing a short duration until you actually need to extend it.
The upload_url is obtained using put_object_url on storage connection. A storage connection can be instantiated using Fog::Storage.new (please refer to the code source for an example).

class DirectFileUploadsController < ApplicationController
  # POST /direct\_file\_uploads
  def create
    upload_key = "#{SecureRandom.uuid}/#{params[:file][:name]}"
    upload_path = "direct_file_uploads/#{upload_key}"

    headers = { 'Content-Type' => params[:file][:type] }
    expires = Time.current + 60.minutes
    bucket = Paperclip::Attachment.default_options[:bucket]

    upload_url = STORAGE.put_object_url(bucket, upload_path, expires, headers)

    render json: { upload_url: upload_url, upload_key: upload_key }, 
           status: :created
  end
end

2. Upload file using pre-signed URL

Once pre-signed URL is obtained, the browser si expected to start upload to it. In this example, I have been using the jQuery plugin called jQuery File Upload for compatibility, but you can adapt with your other libraries:

“fileupload” capability is attached to a cloned form from the original form, to manage each single file individually to prevent conflicts of concurrent uploads.
“fileInput” and “autoUpload” are disabled because we start the upload manually using “send” at the end of the function.
“multipart” is disabled because I assume not to have found a working solution that does not break the file integrity ¯_(ツ)_/¯.
The header “Content-Type” is filled with the file type as of pre-signed URL generation, otherwise S3 would reject the request.
“paramName” is an arbitrary string to enforce a non-nested parameter name, which would not be correctly handled by S3.
“dataType” is more or less arbitrarily set to XML, which is the format of S3 responses.

function uploadFileUsingPresignedURL(file, upload_url, upload_key) {
  const tmp_form = form.clone();

  tmp_form.fileupload({
    fileInput: null,
    url: upload_url ,
    type: "PUT",
    autoUpload: false,
    multipart: false,
    paramName: "file",
    dataType: "XML",
    beforeSend: function(xhr) {
      xhr.setRequestHeader("Content-Type", file.type)
    },
    done: function() {
      // go to step 3
      requestRecordCreation(upload_key)
    }
  })

  // sending one file at a time
  tmp_form.fileupload('send', { files:[file] })
}

In order to allow file uploads from browsers to S3, you have to update CORS settings of your bucket:

[
    {
        "AllowedHeaders": ["*"],
        "AllowedMethods": ["PUT"],
        "AllowedOrigins": [
            "http://127.0.0.1:3000", // local development
            "https://your-domain.com", // production
            "https://*.your-domain.com" // production
        ],
        "ExposeHeaders": []
    }
]

3. Request record creation

Once upload is done, the browser can notify the application that the file is ready for being referenced into an Attachment record. This is more or less a standard “create” request on Attachment’s controller, to which we simply inject the upload_key obtained at step 1:

function requestRecordCreation(upload_key) {
  $.ajax({
    url: "/attachments",
    method: "POST",
    dataType: "json",
    data: {
      authenticity_token: <VALID_AUTHENTICITY_TOKEN>,  
      upload_key: upload_key,  
      attachment: {  
        attachable_type: <ATTACHABLE_TYPE>,
        attachable_id: <ATTACHABLE_ID>,  
        // put here other attributes if required  
      }
    },
    success: function () {
      // operation completed (step 6)
    }
  })
)

On the backend, the controller would start looking like to that crazy standard implementation:

class AttachmentsController < ApplicationController
  # POST /attachments
  def create
    @attachment = Attachment.new(attachment_params)

    upload_key = params[:upload_key]

    ### STEPS 4-5-6 ###
  end

  private
  def attachment_params
    params.require(:attachment).permit(
      :attachable_type,
      :attachable_id,
      # put here others attributes' form if required
    )
  end
end

4. Request file metadata or content

Here is the tricky part of the implementation, and its easiness depends on the use cases you need to handle. You may also want to handle both proposed approaches depending on attachment types, when you have multiple ones in your application.

Let’s start with the easiest one, and to some extent the safest one, which consists in letting Paperclip doing the job as if it was a simple file upload. The idea is to download the file back on the application layer and assign it to the record, so that all features that come with Paperclip work in a standard way. That also means that the application triggers an extra upload of the file to the object storage on record save, which sounds expensive, but we can rely on the fact that we “control” this part of the network, as mentioned in the blog post introduction.

Adjusted sequential diagram for the first approach

If the cost of doing so is too expensive for your use case, you may be interested in the second proposed approach, which consists in faking the IO part of Paperclip by manually assembling all required information. This approach is cost-free because it just needs a few simple API requests, but it has a few limitations:

Post Processing are not triggered and you need to implement them in a different way, for instance in a background job (which sounds good by the way).
Fingerprint must be set with the default algorithm (Digest::MD5), based on what S3 is able to provide through ETag header and given that S3 encryption is set to SSE-S3 or plaintext.
You have added hard validations on the file type or content, that require the file is physically present on the record while saving it.
The path builder requires the primary key (by default :id_partition or commonly :id) and your database does not provide safe ways to predict the next value. If this is the only one limitation, a workaround consists in switching steps 5 and 6 together (save the record first, and then move the file to the final location), but this is not covered in the present example.
Any other limitation I have not been able to cover while writing this blog post :-)

Let’s go for the implementation of both solutions, differentiated by the boolean REQUIRES_DOWNLOAD, starting by building back the upload_path and retrieving bucket name accordingly with step 1:

class AttachmentsController < ApplicationController
  # POST /attachments
  def create
    @attachment = Attachment.new(attachment_params)

    upload_key = params[:upload_key]
    upload_path = "direct_file_uploads/#{upload_key}"
    bucket = Paperclip::Attachment.default_options[:bucket]

    if REQUIRES_DOWNLOAD
      # [...]
    else
      # [...]
    end

    ### STEP 6 ###
  end

  # [...]
end

The first approach is simple:

Obtain a pre-signed URL to download the file. Note that it does not trigger any real API request.
Ensure to have registered Paperclip::UriAdapter, and do it only once from an initialiser file such as config/initializers/paperclip.rb. I have simply put it here to facilitate the reading.
Assign the URL to the attachment using URI, and let’s Paperclip’s internal IO adapter processing the file.

if REQUIRES_DOWNLOAD
  download_url = STORAGE.directories.new(key: bucket)
                        .files.new(key: upload_path)
                        .url(Time.current + 10.seconds)

  Paperclip::UriAdapter.register
  @attachment.upload = URI(download_url)
else
  # [...]
end

The second approach is custom because we need to fake Paperclip’s internals so that it builds valid paths. Paperclip uses interpolations based on :path setting. Each interpolation is based on a record’s attribute or attached file metadata, so that we have to ensure that all of them are pre-filled before calling the path builder:

“file_name”, “file_size”, “content_type” and “updated_at” are standard attributes brought by Paperclip when you follow the installation guidelines, whereas “fingerprint” is only applicable if you have enabled it explicitly.
The sequence related to the primary key is only applicable if it is part of :path or :hash_data settings.
“file_name” is inferred directly from upload_path. We have made it possible at step 1.
“file_size” and “content_type” are obtained from object’s headers.
“updated_at” is an arbitrary timestamp, expected to be the current time.
“fingerprint” is obtained from ETag header (MD5 digest of the content)
The value of the primary key of the record (which is not yet saved at this stage) is obtained using “nextval()” on the given sequence (PostgreSQL only). If you are using UUIDs, you would have certainly implemented a custom interpolation and so you are required to generate it manually at this stage. If you are using MySQL, I suggest to follow workarounds such as this one.

if REQUIRES_DOWNLOAD
  [...]
else
  headers = STORAGE.head_object(bucket, upload_path)[:headers]
  @attachment.upload_file_name = upload_path.split('/').last
  @attachment.upload_file_size = headers['Content-Length']
  @attachment.upload_content_type = headers['Content-Type']
  @attachment.upload_updated_at = Time.current
  @attachment.upload_fingerprint = headers['ETag'].gsub(/"/,'')

  primary_key = @attachment.class.primary_key
  primary_key_sequence =
    ActiveRecord::Base.connection.execute("
      SELECT nextval(pg_get_serial_sequence(
        '#{@attachment.class.table_name}', '#{primary_key}'
      ))
    ")
  primary_key_nextval = primary_key_sequence.first['nextval']
  @attachment.public_send("#{primary_key}=", primary_key_nextval)

  ### STEP 5 ###
end

5. Move uploaded file to the right location

When the first approach is chosen (REQUIRES_DOWNLOAD), the file is automatically uploaded to the right location by Paperclip, among with the post processed images, while saving the file at step 6.

When the second approach is chosen (!REQUIRES_DOWNLOAD), this operation must be done manually. We have previously assigned all required attributes for PaperclipAttachment#path to return the correct path of the file, so the operation simply consists in moving the file to that final destination. Thus, S3-compatible storage does not support “moving” files, therefore we have to make a “copy” using copy_object. This request is a simple API request that does not require any data transfer between the application and the storage, the copy of the file is processed on S3 side.

Finally, both approaches result in a duplicate file (the original one uploaded by the client at step 2) that remains on S3 with no use case except debug or history. I suggest to delete it using delete_object, so that your bucket size does not grow at double speed:

if REQUIRES_DOWNLOAD
  # [...]
else
  # [...]

  final_path = @attachment.upload.path

  STORAGE.copy_object(bucket, upload_path, bucket, final_path)
end

STORAGE.delete_object(bucket, upload_path)

Going a bit further is still required if you have enabled the versioning on your storage, and deletion is indeed a simple delete marker. To permanently delete the file, we have to perform a specific request using get_bucket_object_versions and retrieve the version identifier of the object to append it to the delete_object request:

options = nil
if STORAGE_VERSIONING_ENABLED
  versions = STORAGE.get_bucket_object_versions(bucket, prefix: upload_path)
  version = versions.body['Versions'].first['Version']
  options = { 'versionId' => version['VersionId'] }
end
STORAGE.delete_object(bucket, upload_path, options)

6. Save record and return status

We are finally ready to save the record and return the status to the client, as we would do with any kind of ActiveRecord objects or if we were using simple file upload:

def create
  # [...]

  if @attachment.save
    render json: @attachment, status: :created
  else
    render json: @attachment.errors, status: :unprocessable_entity
  end
end

Code source

You will find a working example in this sample app that covers all mentioned use cases, with a bit more logic.

Interested in joining an enthusiastic engineering team? We are recruiting talented people! Learn more about the team and our open positions here

Common Mistakes With Guard Clause Pattern

Jean-Michel Gigault — Mon, 11 Jan 2021 10:47:05 +0000

My name is Jean-Michel and I am a committed engineer at Per Angusta. I have tried to summarise my thoughts and experience about good and bad practices of the Guard Clause pattern, starting by common assumptions to applicable examples with code samples.

Basically, a common mistake with Guard Clauses is to think that they are simple substitute for “if…else” statements. Of course they consist in “if…else” statements, but it’s not a simple musical game of chairs and it actually involves a design pattern with special meanings and consequences on code quality.

Photo by Kayla Speid on Unsplash

What is a Guard Clause?

A Guard Clause, also known as Early Return or Bouncer Pattern, is a common practice in programming, consisting in an early exit of a function based on preconditions check.

It is nothing other than an “if…else” statement, but it provides with two benefits: It makes the code flatter by reducing number of indentation levels, and it allows exceptions handling to be placed at the beginning, making subroutines clearer to read.

It is not limited to a specific programming language, however practices are more or less common across communities. For instance, in C programming, their usage is very common due to the basic need to handle null-pointers, but also due to strong Unix conventions that involves restriction of functions’ lengths in width and height (to facilitate edition within a terminal) and in order to optimise compilation time.

When to use a Guard Clause?

I would summarise the answer into 3 fundamental assumptions:

It helps with readability
It has a strong and obvious link with the scope of the method
It solves a significant problem (performance, stability, business issue)

If a guard clause does not verify at least 2 of those 3 assumptions, I would consider there probably is a bad design somewhere, and you should consider to split your algorithm into multiple subroutines (or simply not to use Guard Clause).

I would add a tip: If a guard clause is not present at the beginning of a method, there are serious chances that you have a bad design somewhere. But, be aware it’s not systematic.

1. It helps with readability

Implementing Guard Clauses while losing readability is almost a non-sense. Their existence is an answer to the lack of readability of nested “if…else” statements and high levels of indentation.

I would say a Guard Clause that reduces the readability must have strong other reasons, such as stability or performance benefit. And if you feel not comfortable with a convention that forces you to use a Guard Clause rather than something more readable, ask your team if it would better be to adjust the convention instead of the code.

(Note that in this article, I chose the Ruby language to give examples, because it is my natural programming language. But each example applies to all languages)

❌ The following example shows a method that includes two conditional actions, but the Guard Clause in there results in different indentations for both conditions’ and actions’ contents, reading is made difficult:

def do_multiple_things
  if valid_context_for_first_action?
    # action 1
  end
  return if !valid_context_for_second_action?
  # action 2
end

2. It has a strong and obvious link with the scope of the method

The ideal use case of a Guard Clause is when it highlights a reason for not having called the given context at all. I mean, when a given context as a whole does not even make sense with current subject.

This point is particularly involved by inheritance and shared logic. For instance, a set of shared validations can require exceptions with certain models. Or a class inheriting of parent’s methods can require additional conditions of applicability.

In other words: A guard clause should allow a developer to quickly understand that even the method’s name is a non-sense for the current subject.

✅ The following example shows a method that early exits due to a missing piece of critical information for the purpose:

def send_welcome_email
  return if @user.email.blank 

  UserMailer.with(user: @user).welcome_email
end

3. It solves a significant problem

Third point is about pragmatism and efficiency. In certain circumstances, you would think about implementing a Guard Clause to solve an issue in relationship with the global context, whose scope is larger than the local context.

Exiting a method, or skipping an element in a loop, due to external factors or attempted limits, is a good use case for Guard Clauses because they actually highlight a major concern of applicability.

✅ The following example shows a method that exits from a nested level due to a major limit:

def notify_many_third_parties_with_many_records
  @notifiable_records.each do |record|
    return if API.daily_limit_reached?

    API.notify_third_parties_with_record  
  end
end

Photo by Jesse Martini on Unsplash

Don’t be dogmatic, don’t forget to think

Every language has its own set of coding style and conventions. Your team may also have its own: max number of lines per file, naming conventions, order of inclusion, etc.

Static code analysers such as Rubocop, for Ruby programming, or norminette for 42 School’s exercices, provide packaged rules to help developers to follow conventions by highlighting where in the code base some are not respected. Rules are usually configurable to comply with the needs, and there are often default settings issued by the community.

As a result, a community with such popular tools like Ruby’s can see a lot of projects and libraries following similar code styles, whoever has coded them, whatever their level of coding skill is, whatever the maturity of the project is. But not to follow a convention can make sense in many cases, such as:

This method is too largely inspired by an open source algorithm: prefer to keep the spirit of its origin so that it is easier to compare when updates or bugs occur.
This file is legacy, well tested, and not supposed to change so many times: no need to invest time in rewriting it.
This project is small and not intended to grow: keep code readable as it deserves and that’s fine!

Therefore, there is a lot of situations where developers should better beside a static convention. Especially when pros and cons are not mastered, when intentions behind rules are not explained and cristal clear. As an example of that, the following static rule often brings me some headaches in code reviews:

Style/GuardClause: Use a guard clause instead of wrapping the code inside a conditional expression.

Our static code analyser does obviously not trigger this warning on every «if» statement of the application, but does warn all «if» statements placed at the end of methods when they have no «else» statement. In most of the cases, fixing that warning by writing a Guard Clause is counter-productive because of a root cause largely unrelated with the concern of the pattern: a lack of separation of concerns. Often, the solution I request is to disable warning, not to rearrange the code. Below examples may help you understand which solutions to choose.

Examples with code samples

Methods with single concern

✅ The following example looks fine because it is clearly designed: One method with single role:

def validate_dates_consistency
  return if start_on.blank? || end_on.blank?
  return if start_on <= end_on

  errors.add :end_on, 'must be greater than Start Date'
end

Methods with multiple concerns

❌ The following example is ambiguous, it is a mix of conditional statement and Guard Clause, without any hierarchy or special meaning:

def validate_complex_situation
  if something_wrong?
    errors.add :base, 'You must correct something'
  end

  return if !something_else_wrong?

  errors.add :base, 'You must correct something else'
end

✅ Think readability and solve it by using either simple “if” statements…

def validate_complex_situation
  if something_wrong?
    errors.add :base, 'You must correct something'
  end
  if something_else_wrong?
    errors.add :base, 'You must correct something else'
  end
end

…or split into multiple components:

def validate_complex_situation
  handle_errors_about_something
  handle_errors_about_something_else
end

def handle_errors_about_something
  return if everything_valid?

  errors.add :base, 'You must correct something'
end

def handle_errors_about_something_else
  return if everything_else_valid?

  errors.add :base, 'You must correct something else'
end

Loops with multiple transformations

✅ The following example involves a sequential algorithm, where the role of guard clauses is directly linked to the context of the method (a sequential list of similar actions to be performed on each record), and readability is also present:

tasks.each do |task|
  task.mark_as_read!
  next if task.completed?
  task.mark_as_pending!
  next if task.almost_completed?
  task.mark_as_quickwin!
end

❌ The following example falls into a previous one, where a loop has multiple roles and the guard clause’s role is to handle a specific case. It is hard to read, it has a distant link with the context and it does not solve any problem:

very_important_tasks = []

tasks.each do |task|
  task.mark_as_read!

  if !task.completed?  
    task.mark_as_pending!  
  end

  if task.owner != current_user  
    task.notify_owner_of_read_event_by_current_user
  end  

  next if !task.involves_administrator?
  very_important_tasks << task
end

Significant problem solved

✅ The following example involves an expensive call to a huge service called “SyncTaskWithThirdParties”, and a performance profiling of the application has highlighted that it was the root cause of significant useless memory allocations with particular conditions. Readability is ensured through comments and test suite, whereas the link between the role of the method and the content of the guard clause can be hard to understand:

def sync_with_third_parties
  # avoid an expensive call to the sync service (15% of use cases)
  # please refer to the documented specs in task_spec.rb
  return if start_on.blank? && owner.is_a?(AdminUser)

  ::Services::SyncTaskWithThirdParties.new(self).call
end

✅ The following example involves a global context of maintenance window, which is making the role of the method not efficient or even not applicable, so that exiting the method is made necessary and understandable:

def invalid_cache!
 # we have some issues with cache during maintenance windows 
 # we prefer to save performance of display with obsolete data
  return if CacheProvider.maintenance_window?

  update_columns(cached_at: Time.zone.now)
end

Inherited methods

✅ The following example involves an overridden method with a subroutine that does not make sense in a subclass’ context and should be skipped:

def clean_parameters
  # skip whole process with action not requiring parameters at all
  return if action_name == 'toggle_visibility'

  super
end

❌ The following example is ambiguous and malformed because the Guard Clause does not comply with any rule: The method “clean_parameters” is still applicable, it does still make sense with children class’ subject. We only want to handle a specific case, which is not a significant problem. Plus, we lost ease of maintainability:

def clean_parameters
  super
  return if !@params.key?(:send_notification)

  @send_notification = @params.delete(:send_notification) == 'true'
end

✅ As a solution, forget Guard Clause pattern. Keep the ability to implement additional logic after the conditional statement…

def clean_parameters
  super
  if @params.key?(:send_notification)  
    @send_notification = @params.delete(:send_notification) == 'true'
  end
  # additional logic here
end

…or extract the specific case into a separate context, where a Guard Clause actually makes sense. Plus, additional logic can later be easily appended:

def clean_parameters
  super
  clean_parameter_send_notification
  # additional logic here
end

def clean_parameter_send_notification
  return if !@params.key?(:send_notification)

  @send_notification = @params.delete(:send_notification) == 'true'
end

Conclusion

Guard Clause is a design pattern, with meanings and consequences on a code base for the long-term. It is not a code style rule as you would follow with indentation (“2 spaces” or “1 tab”, applicable everywhere, mechanically).

Guard Clauses are related to code structuring and separation of concerns. Be pragmatic with their usage!

Interested in joining an enthusiastic engineering team? We are recruiting talented people! Learn more about the team and our open positions here

What’s Happening Behind The Scene of Rails Applications

Jean-Michel Gigault — Wed, 21 Oct 2020 09:47:15 +0000

What’s Happening Behind The Scene of a Rails Application

My name is Jean-Michel and I am a committed engineer at Per Angusta. I have a continuous challenge in my job: put developers in the right conditions to understand what they are building and where their realisations stand in the Product. First and foremost the technical side.

Being an actual Web developer requires to be conscious at all times about where the code stands in the components’ architecture. Indeed, Rails is made for building final products without having to care about that and developers usually keep confused about how user’s requests are actually handled.

Diagram of a common Rails Application’s architecture and components’ stack.

Main components of the architecture

Let’s have a look at all levels of the components’ stack and define their role. At first, let’s forget the Rails framework and focus on the surrounding environment.

Web server

The role of a Web server is to orchestrate the Unix-side of your Web Application. It keeps alive until you stop it and it cares about how to scale in terms of memory and CPU usage. It is the front office of all HTTP requests before they actually interact with Rails.

Its configuration answers those kinds of question:

How many visitors are supposed to be handled at the same time by a single instance of your application?
How the application is supposed to handle signals of the host machine (start, stop, rage quit, etc)?
Which HTTP ports the application is supposed to listen to?

Puma is the default Web server of a Rails app. It’s standing behind the command «rails server» until you change the configuration. Therefore it is one of the most flexible Web servers for Ruby and Rack applications because it scales in two different manners called “threads pool” and “clustered mode”, allowing you to get the advantage of both available memory and CPU cores of the environment.

It’s worth noticing that when you deploy your Web Application, what we call here a “Web server” may in fact takes the role of an Application server. For instance, Heroku handles requests through its own Web servers and then forwards them to your application. This is best explained by such article.

Background jobs

Because the Web is asynchronous, background jobs are optional but usually required to build a scalable Product. Their role is to handle all processes that are not supposed to render End-User’s content such as HTML pages. For instance: export data into Excel files, send mails or synchronise data with external services.

In Rails, background jobs are handled through Active Job framework, and depending on the use cases you would like to plug a dedicated backend like Sidekiq or Resque. The choice of a backend mainly depends on the running environment (available databases, memory usage, etc.) and the way you want to handle queuing and concurrency.

I suggest you to read this brief article written by my colleague: Benchmarking Excel Generation In Ruby And How To Avoid Background Jobs Memory Explosion!

Databases

Except if your Web Application is a simple Landing Page that displays static information, the role of databases is to build advanced stories. It allows you to store user’s input and create something else with it that can later be rendered to the same user or even other users. Indeed, databases are made for all features that need to go further than the scope of a single user’s session: Keeping a shopping cart into memory for multiple days or sharing user’s data with others.

PostgreSQL is an advanced open-source relational database with a large set of SQL-compliant capabilities and extensions built by the community. You would build applications on top of a relational database like PostgreSQL when you need to store multiple scheme of structured data and build features based on top of their relationships. For instance linking articles, prices, clients and invoices within an e-commerce solution.

Redis is more likely a “dictionary” or a “hash table” based on the key-value storage paradigm. It allows you to store a set of data structures with an optional durability. It fits perfectly with caching purposes because of its expiry capabilities and fast data retrieval performance. In Rails, you would also use this kind of database to handle advanced features based on user’s session like a history of navigation, filtering ans search options, different modes of navigation.

Web services

Today, a common way of building applications is to rely on a Cloud environment. The role of Web services is to help you building final products by focusing on your added value. It brings your Web Application the components that you are not efficient to deliver by yourself.

For instance, when you build an e-commerce you would choose an external Web services to handle the payment means. Because you’re not a banking service. When you deploy on PaaS solutions like Heroku, you would choose to forward your access logs to an addon like Sumo-Logic. Because you’re not efficient in digesting a log trace.

Most of the popular Web services provide you with libraries packaged in Gems, in order to quickly and easily interact with their systems.

Rack Middleware

It’s important to understand that what makes Rails a Web framework is Rack. Indeed, Rails is built on top of the Rack framework.

The role of Rack is to handle the HTTP protocol on which Web Applications are based. HTTP describes how to share octets between two machines and Rack translates them into comprehensive structured Ruby objects for Rails: The request and the response, headers and parameters, settings of cookies and session, etc.

Photo by Julia Kadel on Unsplash

A Rack middleware is a component that handles one specific part of the user’s request and/or Rails’ response.

They are stacked in a given order, determined at the initialisation step of the application and they are linked together like Russian dolls. Each middleware is supposed to do one single thing, call the next middleware of the stack and then return its result to its parent.

class MyMiddleware
  def initialize(app)
    @app = app
  end

  def call(env)
    # do something before calling the next middleware  
    puts "My name is Rack middleware"

    # then call the next middleware and return its response
    @app.call(env)
  end  
end

At the end of the stack comes the router of the Web Application (explained in the next section), as you can see by running the command “rails middleware”:

$ rails middleware

use Webpacker::DevServerProxy
use Rack::MiniProfiler
use Rack::Cors
use Rack::Sendfile
use ActionDispatch::Static
use ActionDispatch::Executor
use Rack::Runtime
use Rack::MethodOverride
# [...]
use Rack::ConditionalGet
use Rack::ETag
use Rack::TempfileReaper
use Rack::Attack
use Warden::Manager
run MyRailsApplication.routes

One of a great middleware is Rack::Attack. Its role is to block abusive requests to avoid your application from being overloaded during a DoS attack, for instance. When the IP address of a request is recognised as being abusing (based on number of requests per period of time), it decides not to call the next middleware but to render immediately an error response. It breaks the chain and consequently increases the performance to handle such abusive requests.

Other examples of use cases:

Rack::Cors handles Cross-Origin Resource Sharing settings in the response to tell the browser about the authorized AJAX calls on cross domains.
Apartment’s Elevators handle the strategy to determine which tenant is concerned by the user’s request within a multi-tenancy application.
A custom middleware that blocks requests and renders a proper message when the application is put in maintenance mode.

Some of you may ask the difference between Rack middleware and Rails controllers’ callbacks like “before_action” because both allow you to build logic around an action. The main difference stands in their position in the components’ stack regarding the router: Rack middleware takes place before the route is determined, so that it is always involved whenever the requested route is not supposed to be handled by an internal controller. For instance, when routes are handled by Rails Engines like Devise, Grape or Sidekiq Web UI. The scope of Rack middleware is always “greater” than Rails’ controllers.

The router

The role of the router is to declare which URLs and HTTP request methods are available in your Web Application, and map them with controllers’ actions.

Photo by John Barkiple on Unsplash

For instance, it determines which action is involved when making a GET request to the URL “/users”. And even if the convention implies “UsersController#index”, you must explicitly declare your routes in the configuration file “router.rb”, either by using helpers or manual declarations:

# using helper
resources :users, only: [:index]

# manual declaration
get '/users', to: 'users#index', as: 'users'

This is also the place where to declare the Rails Engines. A Rails Engine is an embedded Rack application that extends the capabilities of your application. One of the most popular one comes with Devise as a complete MVC solution to handle user’s registration and sessions:

# mounting Devise's Engine in router.rb
devise_for :users

Controllers and MVC pattern

It’s well known, Rails framework is based on the architectural pattern called model-view-controller.

Controllers handle user’s requests (for instance “get the list of active users”), interact with Models (for instance “retrieve User records with an active flag from database”) and respond to the user by rendering Views (for instance “build an HTML content with a title and a table of users”).

Contrary to what’s shown in the diagram at the beginning of the article, views are not the final point before going back to the Rack middleware’s stack. They stand in the scope of controllers, and controllers can still perform a lot of things after rendering them. For instance, when you declare “after” and “around” action callbacks.

A great use case comes with Pundit’s verification callbacks that aim at adding a security layer regarding the permissions of the current user. Those verification run after the view is rendered. Also think about controllers’ actions that do not even interact with proper views from the folder “app/views/”. For instance when you render a JSON object in API contexts or even when you render No Content HTTP request.

Therefore, the role of controllers is to build and return a response Object that can then be handled by Rack middleware’s stack, in the reverse order (think Russian dolls). At the end of the stack, when each middleware has returned the response Object to its parent caller, the response is formatted in raw HTTP-compliant octets by Rack and sent back to the user through the Web server.

Service Objects, libraries and more

There are many different design patterns to orchestrate what’s happening inside controllers and help you organise your code into maintainable sub-components, in opposition to writing the whole logic right into controllers’ actions.

A popular approach is to use the Service Objects design pattern, consisting in embedding the logic of a given action into a dedicated Ruby object. By building a collection of Service Objects with their own responsibilities and interactions, you can then build advanced features shared across the application through multiple controllers or even other layers (as shown in the diagram with the API Server).

Service Objects or what else you use to embed your logic may also interact with libraries such as Slack Ruby Client whose you don’t own the code. A good approach while using external libraries is to recreate your own local interface that embeds the specific pieces of code. For instance having a generic object called “Notifier” that embeds Slack-specifics will help you migrating your Web Application if you plan to change Slack by MS Teams using MS Teams Ruby Client.

API Server

The Rails framework is made for building Web Applications and API servers in a reversible way. Controllers have builtin features to handle multiple formats at once in the same place and you can render HTML content for real users or JSON content for machines within the same controller’s action.

But in some circumstances, you may choose to extract the API context into a separate layer of the application such as in a Rails Engine like Grape. The router is therefore in charge of forwarding requests to the relevant layer. For instance, you can mount a Grape API in a namespace “/api” and consider all sub-URLs to be in handled by it.

Depending on the scale of your Product, you would better consider having dedicated Web servers for each Web and API context, so that you can adapt hardware and software components to best comply with their specific usage.

Formatters and presenters

As referred above, your Web Application may have to handle multiple formats of requests and responses depending on the caller. This is the role of formatters, also known as “presenters” in some design patterns.

Depending on the Accept header of a request, your Web Application can format the response’s content using different formats like JSON, JSONAPI, CSV, XML, and of course HTML.

API Client

API Client is the layer of your application that aims at requesting external Web services. For instance to retrieve the weather forecast for a tourism website or to send scheduled notifications through a mailer service.

An API Client is not supposed to know about your internal database schema, your models or even any business rule, its single role is to know how to authenticate and communicate with external web services and fetch/pull data for your service objects.

A good approach is to wrap them all into a common interface object, so that you can implement shared logic such as logging, monitoring, switch on/off during maintenance periods, etc…

Interested in joining an enthusiastic engineering team? We are recruiting talented people! Learn more about the team and our open positions here

Why I Don’t Use Any Aliases

Jean-Michel Gigault — Tue, 28 Apr 2020 09:01:00 +0000

My name is Jean-Michel and I am a committed engineer at Per Angusta. Here is my point of view about using shortened words to describe complex intentions.

I am used to challenge the benefits of aliases in a Shell environment with my peers. I rarely hear mastered reasons but rather a pretended gain of efficiency. I personally chose to unset them all and keep being explicit with my machine.

Figure 1: Artist view of an alias deploying application to production

The Shell environment

A shell is a command line interpreter for Unix-like operating systems. A place where developers (users) get control of their machine.

It has wide capabilities (depending on the OS and the version of the distribution) such as browsing directories, managing files, running scripts and installing dependencies. Its usage is a must-known for all developers. Like knives for cooks and saws for carpenters, it shapes the main toolkit for developers.

That’s why best programming training programs always start with Unix learning and practising sessions. In the best case they invite students to create their own Shell program to deeper understand what’s going on behind the scene.

Once the learning step is reached, a developer is much more autonomous in its daily issues and he starts not to depend on each others. His machine stays forever his only one working environment (until coding with LEGOs is made possible!) and mastering it is a must-have skill. That’s the purpose of a command line interpreter.

Are aliases a standard practice over the world?

They are to an extent, but their usage is not. Like Web and API protocols/frameworks, the way they work is common but there is no guarantee it is implemented the same way.

An alias is a string to be substituted for a word when it is used as the first argument of a command. It is usually a shorter word than what it is supposed to be substituted for, and is commonly composed of the first letters of the command and its first arguments. For instance:

# set an alias for staging files (git add) using patch option
alias gapa="git add --patch"

Now user can type “gapa” instead of the original command, and it’s now pretty easy to understand the first benefit of aliases: less typing, quicker control of the machine!

Aliases are set at the user session level in a Shell when the profile is loaded (for example on Bash from file “~/.bashrc”). It is then clearly identified as a User setup not related to the environment. That’s why there is no guarantee that an other user, including one on the same machine, decides that “gapa” means something else, for instance:

# set an alias to grant all permissions to all
alias gapa="chmod 777"

OF COURSE DON’T! 😱

However in this case, user decides not to use the first letters of the command and its first argument (that would have resulted in “c7”), but the ones from its human meaning, which make more sense to him because he doesn’t care about versioning files with Git but is an Access Control List manager.

Not a good approach for abstract

I can hear from my peers that aliases are one of the large set of programming abstracts: objects, methods, variables, and so on, including DRY principle (“Don’t repeat yourself”). However, the common usage of aliases sounds more like short-linking, as if you would create shortcuts on your Desktop folder to quicker access your favourites.

To me, a justified approach of abstract would be to use them as a human readable substitute for technical commands, for instances:

# deploying current branch on staging remote server
alias x-deploy-staging="git push staging $(git rev-parse --abbrev-ref HEAD):master"

# archiving directory recursively
alias x-archive-directory="tar -zcvf"

In this case, user decides to abstract its common technical commands into human readable links. A prefix “x-” allows to differentiate when typing custom commands in favour of builtins and binaries. Now it is not only shortcuts for repeatable commands but it actually becomes an abstract tool.

Explicit over implicit

The more explicit a code or a library is, the easiest it is used and maintained. That’s one of the main principle developers apply: choosing appropriated variable, method and class names, especially when they work on a product that is supposed to be maintained by others.

Every developer has experienced at least once in his life (if not once a day) a peer challenging the naming of a variable after a code review. For example, given a model named “ThirdPartyClient” and a code that iterates on each of its records:

ThirdPartyClient.all.each do |tpc|
  tpc.deactivate!
end

As a matter of fact at Per Angusta, a code reviewer would invite the requester to rename “tpc” by “third_party_client”. Wouldn’t you? No one would invite the opposite by asking to type less characters to gain efficiency. The readability always comes first.

For the same reason and when available, I personally prefer to use the longer version of builtins or binaries’ options. When I am running commands against an “application”, I choose to use “ — app” over “-a”. It makes more sense to me and that’s how I would write it in a Shell script before asking for a script review to my peers.

Let’s go further with the famous “git push” command, which is commonly used without any arguments because it perfectly fallbacks to defaults. When my intention is to push my work from my current branch onto its remote counterpart, I am explicitly typing the following command:

# telling exactly on which server, what to push in which branch
git push origin HEAD:feature-branch

The remote: origin is the default remote, until it is configured another way
The source: HEAD is the current ref, which is by default the last commit of the branch until you checkout a different one
The destination: feature-branch is the default, until the remote one is named differently than the local branch

This way, before pushing my work I am asking myself what I exactly want my machine to do for me, giving myself more chances to not do mistakes. It invites me to ensure I know what remote, source and destination are in my current context.

Thus, the explicit version of a command is the only one I expect to find in a documentation (like articles, posts or wiki pages). Why losing those benefits while controlling my machine? I get the control of it while documenting my intention, and my Shell history is tracking it.

Let your peers know what you are really doing

When you’re riding a bike on a road and you reach a crossroad, you probably know what to do: tell the environment what your intention is. As a matter of fact, you’re aware of your own intention and your local machine (the bike) doesn’t care about it because it is controlled by you. So, why to explain your intention?

Bicycle Safety (https://www.azdps.gov/safety/bicycle)

Of course you want to prevent the other users of the road from hitting you, even if your intention seems obvious. More generally you would try to make things happen in a smooth way and try to minimise the doubt. So what about applying this purpose to your Shell environment: your teammates would probably be more comfortable if you told them what your intention exactly was before running command lines in public.

It is relevant with the history: browsing a Shell session history is smoother when command lines are explicit, especially in case of a mistake analysis. As a rider, you would be more comfortable to remind a driver that you told him you would turn on the right before he hit you.

When you are working with peers with different skills level and setup, using aliases is disturbing and inefficient at all. Beginners won’t follow you and you probably will pass more time to retrieve what your aliased command actually does rather than typing its original. That is similar circumstance as if you had to explain what the variable “tpc” is whereas you could have directly named it “third_party_client”.

Efficiency gain is not obvious

Aliases are nothing more than embedded Shell scripts, except they are stored in the user session instead of script files. It can help you running repeated operations without typing them again and again. That’s why it is, and must be, considered as a good manner to gain efficiency and doing less mistakes. It is especially the case when the substituted commands require a lot of obfuscated options like “tar -zcsv”.

But, assuming you are working in a given environment (with tools and workflows in place), if you, or your team, want to gain efficiency in a Shell environment, my opinion is that you should better first ensure to:

Ask yourself if you’re really controlling the purpose of your personal set of aliases, especially when they come from external tools (like Oh My ZSH!).
Ask yourself if your peers can understand the rules you’ve setup in your own session, their purpose and their impact.
Ask yourself if you would better try to setup common guidelines instead of standing apart. Bring your tools to the team and ensure every one adopt them and is trained for. For example: share a versioned and documented set of aliases or scripts through a Git repository.
Being conscious of the impacts an obscure command line can have, starting by yourself, like issues with history analysis or pedagogical purposes while peering. It does not only apply to aliases.
Being conscious that a Shell environment is not a programming interface. You need to know more than ever what you are actually doing and what your context is. Killing processes afterward with “Ctrl+C” has no guarantee to be reversible, whereas coding in a versioned code base is.
Not to ask for this blog post’s author if aliases have more benefits than issues until they are implemented as a team workflow and become an actual standard 😜

Interested in joining an enthusiastic engineering team? We are recruiting talented people! Learn more about the team and our open positions here