API File Upload Done Right - FastAPI

In this article, I document my research on HTTP limitation for file upload and show an implementation to build an API for file upload using fastapi.

Lets assume you want to update your social profile to hold your work email and a new profile picture. The request therefore is required to contain two data entries: email and profile picture. To ensure this update request succeeds, the request header content type would have to reflect the kind of data being sent

What is Content Type and Why does it matter ?

Content Type is how we communicate the kind of data that's being sent. This allows for an appropriate interpretation by the receiver which can either be the client or server.

• content type multipart/form-data: expresses values submitted through an HTML form. A sample HTML form with multipart/form-data

<form action="/" method="post" enctype="multipart/form-data">
  <input type="email" name="email" value="enter your  email address" />
  <input type="file" name="picture" />
  <button type="submit">Submit</button>
</form>

• content type application/json: expresses values submitted through JSON (JavaScript Object Notation) data. This is basically a mapper of key and values pair.

Because content type allows receiver to understand the kind of data that's being sent, for there to be a successful communication, the rules has to be respected. If the receiver is expecting data to be sent in via multipart/form-data but ends up receiving data in form of application/json then there would be a problem in communication and vice versa.

Below is an example of data which would respect the application/json content type:

{
     "email": "[e-mail]",
     "picture": "myfile.png"
}

While the above data is a valid JSON, there's a misinterpretation going on there. The value given to picture payload would be interpreted as string although our intention was to submit a file.

So how do you solve this ?

ACCEPTABLE JSON PAYLOAD WITH FILE FIELD

While you cannot feed in a raw file data to a JSON object, you can convert the raw file into data that could be understood and allowed by JSON. Did i say conversion ? Yes, I did. This would warrant you to reconvert the data to the initial state at the receiving side too. JSON can understand Base64 and here's some excerpts about it from wikipedia

Common to all binary-to-text encoding schemes, Base64 is designed to carry data stored in binary formats across channels that only reliably support text content. Base64 is particularly prevalent on the World Wide Web where one of its uses is the ability to embed image files or other binary assets inside textual assets such as HTML and CSS files.

The more typical use is to encode binary data (such as an image); the resulting Base64 data will only contain 64 different ASCII characters, all of which can reliably be transferred across systems that may corrupt the raw source bytes.

Here's a summary of the above exerpt:

Base64 is typically used to encode binary data (images, files, e.t.c) to text strings (actually bytes) which JSON understands and can work with.

How do we do this ? you'd ask. I would give you a practical answer in a bit. To clarify all that has been said so far, here are some straight forward questions and quick answers

When should I use content type format multipart/form-data ? Use multipart/form-data when you're handling a request from an HTML form.

When should I use content type format application/json ? When building an API (Application Programming Interface).

Let's answer the big question How do i upload a file data with extra payloads using a practical example

HOW TO HANDLE FILE UPLOAD WITH PYDANTIC MODEL

In fastapi documentation, a section is dedicated to explaining how to request file, for which you must have python-multipart package installed as it is a requirement. The given example expects request from HTML form i.e multipart/form-data content type.

But that's not what we want, we want to send our data as with application/json content type. Following are steps showing the right way to go about this

1. convert the file to a base64 data url string: we'll use an online tool to achieve this. I experimented with base64guru during my research. After converting your file, the generated string should have this kind of format

data:@file/jpeg;base64,/9j/4AAQSkxxxxxxxxxxxxxxxx

copy the response from the generator as is and use it for step 2

1. assign the value to picture payload

2. on the receiver side, reconvert the base64 data url string back to it's initial state

I'll assume you've a fastapi application setup already, so create a new pydantic model with email and picture field

from pydantic import BaseModel, EmailStr

class UpdateProfileSchema(BaseModel):
    """Models updatable field of a profile instance"""
    email:EmailStr
    picture:bytes  

Create an api endpoint to handle a real profile update expecting both email and picture data payload. This endpoint reconverts the received base64 data url string back to a file and save the file in the file system. For other cases, you might want to upload the file to a private s3 bucket.

import base64

@app.post("/upload", response_model=dict)
def update_profile(payload:UpdateProfileSchema):
    data_split = payload.image.split('base64,')
    encoded_data = data_split[1]
    data = base64.b64decode(encoded_data)
    with open("uploaded_image.png", "wb") as writer:
        writer.write(data)

    return {"detail": "Profile update successful"}

Below is a snipped screen of my request payload and the content type used in the request header.

CONCLUSION

We've successfully uploaded a file using application/json content type. The resolution approach taken is not a limitation on FastAPI but of HTTP. Therefore the followed approach is how images/files should be uploaded to an API regardless of the framework used.

Encounter any error ? leave a question on the comment section. If you enjoyed this article, follow me for more.

Email me [e-mail]