DEV Community: Aurel

multipart/formdata: why is it recommended

Aurel — Sat, 17 Feb 2024 11:02:28 +0000

Multipart/formdata is a way of transferring form data from a client to a server.

It was first described in 1995 with the RFC 1867 witch makes it one of the oldest specification about transferring huge data between computer. Let's see why is it the recommended approach to transfer some data in a client/server architecture.

TLDR

Here are some reasons why it is the recommended approach to implement file upload:

Possibility to transfer different kind of data inside the same request
Efficiency compared to bare text sending
Supported on all internet browsers

Let’s dive into each reason:

Possibility to transfer different kind of data inside the same request

The mulltipart/formdata specification allow sending of multiple types of content along the same request. It means that you can transfer simple text, JSON data, binary data in the payload of the request. Each part need to be separated by a boundary, identified by a name and a content-type.

In the following example, you can see that we are sending :

form-data
text/plain
and image/gif

Content-Type: multipart/form-data; boundary=AaB03x

   --AaB03x
   Content-Disposition: form-data; name="submit-name"

   Larry
   --AaB03x
   Content-Disposition: form-data; name="files"
   Content-Type: multipart/mixed; boundary=BbC04y

   --BbC04y
   Content-Disposition: file; filename="file1.txt"
   Content-Type: text/plain

   ... contents of file1.txt ...
   --BbC04y
   Content-Disposition: file; filename="file2.gif"
   Content-Type: image/gif
   Content-Transfer-Encoding: binary

   ...contents of file2.gif...
   --BbC04y--
   --AaB03x--

Efficiency compared to bare text sending

https://www.w3.org/TR/html401/interact/forms.html#h-17.13.4

The content type "application/x-www-form-urlencoded" is inefficient for sending large quantities of binary data or text containing non-ASCII characters. The content type "multipart/form-data" should be used for submitting forms that contain files, non-ASCII data, and binary data.

application/x-www-form-urlencoded and application/json are designed to handle text data and not binary data. If you want to send binary data, you need to encode it in base64, which is inefficient regarding the output result size.
Browser handle content negotiation: The approach of sending file using JSON is not standard, and you may need to implement it yourself both on your client and the server before benefiting from it. Knowing that your browser and the server negotiate to handle the content transfer, your implementation.

Transferring a content to the browser by default use

Supported on all internet browsers

https://www.rfc-editor.org/rfc/rfc1867

multipart/formdata has been around for decades. It has been the norm for transferring file to a backend, and all the internet relies on it.

multipart/formdata is the way of transferring file, and form-data from a client to a server in an efficient way. They are some alternative, but they would be an overhead for implementation and may not be as benefiting for your development.

In the next post, we will dig deeper into how multipart/formdata is structured to send different kind of data to your server.

How to deploy poetry based Fast-Api apps on Heroku

Aurel — Wed, 21 Apr 2021 17:16:14 +0000

Introduction

Fast-Api is a python framework which intended to help developers to build a modern API easily and quickly. It heavily relies on python standard typing module and Mypy for static type checking. Fast-Api is based on Starlette framework which is ASGI framework and allow to made use of python asyncio module.

Heroku is a Paas which help developers to easily deploy their app. Their offer a free plan which is enough to host an MVP or your hobby project. It can also be integrated to GitHub for automatic deployment on commit.

Poetry is a dependency and virtual env management program for python like Pipenv or many others.

The aims of this article is to show how you can deploy a Fast-Api project which use poetry as dependency management on Heroku. Many tutorials cover the deployment of Fast-Api on Heroku but fewer mention the particularity of poetry deployment on the Heroku platform.

Initializing the app

Create the app on Heroku

Creating the app with is a quite easy process. You just need to log in on https://www.heroku.com/ and create a new app. Make sure to select the language you are mainly using in your application.

Here we are focusing on Python.

Add the Procfile to your local project

Heroku read the instructions to start your app from a file named Procfile, located at the root of your project.

You need to set a web command in your Procfile which define how to start your app. For a Fast-Api project, it is basically:

web: uvicorn app.main:app --host=0.0.0.0 --port=${PORT}

Poetry build pack for Heroku

To be able to install your Poetry dependencies, you need to add a poetry build pack to your Heroku app. The poetry build pack helps Heroku to figure out how to install those dependencies defined in the pyproject.toml as Heroku do not support Poetry by default as dependency manager.

If you already install the heroku-cli app, you could easily add the build pack by issuing the following instructions in your shell:

heroku buildpacks:clear
heroku buildpacks:add https://github.com/moneymeets/python-poetry-buildpack.git
heroku buildpacks:add heroku/python

moneymeets/python-poetry-buildpack

The build pack help to create a requirement.txt file from your pyproject.lock during the build on Heroku.

Your Heroku app is now configured to run properly. Let me share with you this script that is useful for setting env var on Heroku. If you have a bunch of env vars to set up during your app creation on Heroku, then you will definitely need this script. It simply helps to set env var from a local .env.

I hope this post will helps you in a way.
You can follow me on Twitter if you like the content.

How to Deploy Your FastAPI App on Heroku for Free

Autodeploy FastAPI App to Heroku via Git in these 5 Easy Steps

Clean architecture use in real world

Aurel — Sun, 26 Jan 2020 06:42:58 +0000

Software architecture lectures brought me to clean architecture and onions architecture notions,
I would like to know if those architectural styles are really used in enterprises or if people really adopt them. I'm asking this because there is no concrete resources to use as example on GitHub.

Setting up distributed database architecture with postgresql

Aurel — Sun, 20 Oct 2019 11:41:04 +0000

What is a distributed database

Distributed databases are a set of databases split across different locations which communicate and provide services through a network.

A well designed distributed database should provide:

a network transparency: the end users should not know that the database is split across different locations, they should run queries as they do in a normal database architecture
a architecture transparency: the users don't know the architecture behind the database

Pros and cons of distributed databases

/!\ you do not make a distributed database architecture because you want it, but because your project need it. Distributed databases provide (when they are well designed)

easily scale-out of you entire production
easy to set up replication to maintain your data's integrity
high availability
fail over

some cons are:

not easy to maintain
not easy to set up

In this post, we will use postgresql as DBMS to set up our architecture:.

Case study

Let consider this relational model.

It describes a scenario where students can lend book sample from a library. Let assume that the library is split into two cities: (Paris, and Lagos). In the generated queries of our application, we notice that many queries are of this kind:

select booksample.state, booksample.lendable, booksample.location from booksample where booksample.location = 'lagos';
select booksample.state, booksample.lendable, booksample.location from booksample where booksample.location = 'paris';

To speed up queries execution, we decide to set up a distributed databases in out two sites, Paris and Lagos.

Partitionning

Many methods of tables partitioning exists and are use in distributed databases.

Horizontal partitioning: its about splitting the rows according to the values of the attributes of the table.
Vertical partitioning: its about splitting the columns of the table into different servers.

In this post, we will do a horizontal partitioning.

From the above queries , we can:

partition booksample table into two tables:
- booksample_paris
- booksample_lagos
partition lend table into two tables:
- lend_paris
- lend_lagos

/!\

the intersection of the partitions of a table should be null
the union of the partitions of a table should contain all the rows of the initial table

On the master server

Let create the different table on the master database.

The master database is the one at the front of the user, we will create the tables as normal table to start. Here we would use the Paris's server as the master.

create the database:

create database dd_test;

create the tables student and book:

--table student
drop table if exists student cascade;
create table student (id serial primary key, name varchar);
--table book
drop table if exists book cascade;
create table book(id serial primary key, title varchar);

table booksample:

drop table if exists booksample cascade;
create table booksample(id serial primary key, state varchar, lendable varchar, location varchar, book_id int references book(id));

table lend:

drop table if exists lend cascade;
create table lend(student_id int references student(id), booksample_id int references booksample(id), at date, returned_at date);

On Paris's server

In this example, as we are using the Paris's site as master, this query should be run on the same postgres server as the previous one. Let create the tables booksample_paris and lend_paris

-- paris site
drop table if exists booksample_paris cascade;
create table booksample_paris(check(location='paris')) inherits(booksample);

drop table if exists lend_paris cascade;
create table if not exists lend_paris() inherits(lend);

The check(location='paris') will help postgres to fetch the row in a "smart" way.

I use the inherits keyword to tell postgres that the table booksample_paris is actually a part of the table booksample. Thus, when we issue a:

explain select * from booksample;

we get

dd_test=> explain select * from booksample;
                                  QUERY PLAN                                  
------------------------------------------------------------------------------
 Append  (cost=0.00..151.56 rows=1251 width=104)
   ->  Seq Scan on booksample  (cost=0.00..0.00 rows=1 width=104)
   ->  Seq Scan on booksample_paris  (cost=0.00..16.10 rows=610 width=104)
(3 rows)

It means that postgres will try to grab the rows of the table booksample_paris on a select.

On Lagos's server

Let's create the database

create database dd_test;

Here we are going to set up the postgres server to listen to the other network interfaces. This is done by modifying the configuration file located at (on most Linux os) /etc/postgresql/9.5/main/postgresql.conf

#------------------------------------------------------------------------------
# CONNECTIONS AND AUTHENTICATION
#------------------------------------------------------------------------------

# - Connection Settings -

listen_addresses = '*'      # what IP address(es) to listen on;
                    # comma-separated list of addresses;
                    # defaults to 'localhost'; use '*' for all
                    # (change requires restart)

The next step is to allow a user to connect through the network interfaces by modifying /etc/postgresql/9.5/main/pg_hba.conf

# TYPE  DATABASE        USER            ADDRESS                 METHOD

# IPv4 local connections:
host    all             test_user       all                     md5

Let's create now the partitions tables:

-- booksample_lagos
drop table if exists booksample_lagos cascade;
create table booksample_lagos(id int, state varchar, lendable bool default false, location varchar, book_id int);

-- lend_lagos
drop table if exists lend_lagos cascade;
create table lend_lagos(student_id int, booksample_id int, at date, returned_at date);

On the master server

We will use the foreign table ** feature of postgres to be able to access the **Lagos's database tables remotely from the master server . To be able to do this, we should create the postgres_fdw extension in our database. This action should be done only by an administrator, so let's connect to the database as the administrator postgres user and do:

/!\ There are many alternatives to foreign table use, such as the use of materialized views + triggers or postgres partitioning feature.

create extension postgres_fdw;

create server master_server foreign data wrapper postgres_fdw options (host '{ip address of the lagos postgres server}', port '5432' , dbname 'dd_test');

create user mapping for master_user server master_server options (user '{our username on lagos server}', password '{our password on lagos server}');

alter server master_server owner to master_user;

First, we create the extension postgres_fdw and after a "foreign data server" on the master postgres server. We now create a user mapping to be able to query the Lagos server.

Now, let's create the foreign tables located on the master server which map to the shard on the Lagos servers.

drop foreign table if exists booksample_lagos cascade;
create foreign table booksample_lagos (check(location='lagos')) inherits(booksample) server master_server;
drop foreign table if exists lend_lagos cascade;
create foreign table lend_lagos () inherits(lend) server master_server;

In its actual state, the master server will fill the the table booksample and lend when a query like this is executed.

insert into booksample values(1, 'new','paris',1)

This is not a good behavior as the new partitions we created will not hold any data. To fix this situation, we will use "triggers" to redirect the row into their normal destination.

The trigger bellows is to redirect the booksample insertion into the correct partition: either booksample_lagos or booksample_paris based on the value of attribute location:

-- trigger on insert booksample
create or replace function booksample_trigger_fn() returns trigger as
$$
begin

    if new.location = 'paris' then
        insert into booksample_paris values(new.*);
    elsif new.location = 'lagos' then
        insert into booksample_lagos values(new.*);
    end if;

    return null;
end
$$
language plpgsql;

drop trigger if exists booksample_trigger on booksample;  
create trigger booksample_trigger before insert on booksample for each row execute procedure booksample_trigger_fn();

Now, we would like to redirect the queries on the table lend to the correct partition. Here we store the row into the site where the booksample belongs to.


create or replace function lend_trigger_fn() returns trigger as 
$$
declare
    vbooksample booksample%rowtype;
begin
    -- select the booksample referenced by the booksample_id
    select * into vbooksample from booksample where id=new.booksample_id;

    -- get the location to use and save the row
    if vbooksample.location = 'paris' then
        insert into lend_paris values(new.*);
    elsif vbooksample.location = 'lagos' then
        insert into lend_lagos values(new.*);
    end if;

    return null;
endtut
$$
language plpgsql;

drop trigger if exists lend_trigger on lend;  
create trigger lend_trigger before insert on lend for each row execute procedure lend_trigger_fn();

Our database is now functional.

Test

Let's run some query to get an overview of our database capabilities:

insert into book(title) values('book#1');
insert into student(name) values('std#1'),('std#2'), ('std#3');

-- thanks to the trigger we wrote, the insertion will be executed in the right server 
insert into booksample(state, lendable, location, book_id) values('old', true, 'lagos', 1), ('old', true, 'paris', 1), ('new', true 'lagos', 1);

insert into lend(student_id , booksample_id ,at) values(1,1, now()), (2,2, now()), (3,3, now());

Let see what happen when we try to select all the row.

explain select * from booksample;

-- thanks to the check(location='{}') constraint, the select query is executed in a "smart" way 
explain select * from booksample where location='lagos';
                                 QUERY PLAN                                

----------------------------------------------------------------------------
 Append  (cost=0.00..118.08 rows=4 width=104)
   ->  Seq Scan on booksample  (cost=0.00..0.00 rows=1 width=104)
         Filter: ((location)::text = 'lagos'::text)
   ->  Foreign Scan on booksample_lagos  (cost=100.00..118.06 rows=3 width=104)
(4 rows)

explain select * from booksample where location='paris';
                                 QUERY PLAN                                

----------------------------------------------------------------------------
 Append  (cost=0.00..118.08 rows=4 width=104)
   ->  Seq Scan on booksample  (cost=0.00..0.00 rows=1 width=104)
         Filter: ((location)::text = 'paris'::text)
   ->  Foreign Scan on booksample_paris  (cost=100.00..118.06 rows=3 width=104)
(4 rows)

Thanks to the check(location='{}') constraint, the select query is executed in a "smart" way. The DBMS scan only the table on Paris's server or Lagos's server.

Thank you for your attention.

DEV Community: Aurel

multipart/formdata: why is it recommended

Possibility to transfer different kind of data inside the same request

Efficiency compared to bare text sending

Supported on all internet browsers

How to deploy poetry based Fast-Api apps on Heroku

Introduction

Initializing the app

Create the app on Heroku

Add the Procfile to your local project

Poetry build pack for Heroku

Related posts

Clean architecture use in real world

Setting up distributed database architecture with postgresql

What is a distributed database

Pros and cons of distributed databases

Case study

Partitionning

On the master server

On Paris's server

On Lagos's server

On the master server

Test