loading...
Cover image for Active Storage meets GraphQL: Direct Uploads
Evil Martians

Active Storage meets GraphQL: Direct Uploads

palkan_tula profile image Vladimir Dementyev Updated on ・7 min read

Part 2 is available here

Happy code for happy people

Let me start with a strong statement: I am a happy person.

This happiness is multi-dimensional, with some dimensions bringing more value than others (and I'm not talking about the most valuable ones today, sorry).

For example, I'm happy at work because I finally got an opportunity to build a project from scratch using such "cutting-edge" (if we can say so about 15-years old framework 😉) technologies, as Rails 6.

Believe it or not, I've been working on Rails production projects for 5 years, and haven't even touched Rails 5, only Rails 4!

"Legacy Rails applications" was a kind of my specialization (by the way, that's the topic of my upcoming RailsConf session).

These dark times have ended this January: I ran gem install rails --prerelease && rails new ***.

(Actually, it was rails new *** -d postgresql --skip-action-mailbox --skip-action-text --skip-action-cable --skip-sprockets --skip-spring --skip-test --skip-bundle.)

The project I'm working on is not a 100% new codebase; it has a lot of code initially written for Rails 4.

And it has GraphQL API as the main entry point for clients (web and mobile applications).

As a part of porting the old codebase into the new app, we migrated from CarrierWave to Active Storage. The experience was smooth. And though Active Storage has some missing parts, it has its advantages and a Rails-way simplicity.

NOTE: If you're new to Active Storage, check out the post I wrote a year ago with my colleague: "Rails 5.2: Active Storage and beyond".

So, it's time to move to the most notable advantage of Active Storage: direct uploads implemented out-of-the-box.

Life before Active Storage

First, let me tell you how we dealt with file uploads in Rails 4. Neither GraphQL specification nor graphql Ruby gem specifies a way to properly cook file uploads.

There is an open-source specification, which has implementations in different languages, including Ruby. It "describes" the Upload scalar type, does some Rack middleware magic to pass uploaded files as variables and kinda works transparently.

Sounds like a "plug-n-play." In theory. In practice, it transformed into "plug-n-play-n-fail-n-fix-n-fail-n-fix":

  • Buggy client implementations (especially for React Native)
  • Side-effects due to a non-strict Upload type (which doesn't care about the actual object type)
  • Apollo dependency (yes, we said "Good-bye!" to Apollo in the new version; but that's another story).

No surprises (and no alarms 😉), we decided to get rid of this hack and use a good old REST for uploading files.

And here comes Active Storage with direct uploads.

Directing uploads 🎥

What is "direct upload," by the way?

This term is usually used in conjunction with cloud storage services (e.g., Amazon S3) and means the following: instead of uploading a file using the API server, the client uploads it directly to the cloud storage using credentials generated by the API server.

direct uploads diagram

Direct uploads diagram

Good news–Active Storage provides a server-side API to handle direct uploads and a front-end JS client out-of-the-box.

Another good news–this API is abstract, works with any service supported by Active Storage (i.e., filesystem, S3, GCloud, Azure). And that's great: you can use filesystem locally and S3 in production without all that if-s and else-s.

Good news rarely come without bad news, though. And the bad news is that Active Storage (and Rails in general) does not know anything about GraphQL and relies on its own REST API for retrieving direct upload credentials.

What do we need to make it all happen in GraphQL?

First of all, an ability to get direct upload credentials using GraphQL API (via mutation).

Secondly, it would be great to re-use as much JavaScript code from a framework as possible to avoid re-inventing the wheel.

createDirectUpload mutation...

mutation preview in GraphiQL

mutation preview in GraphiQL

Unfortunately, Rails doesn't have any documentation for the server-side direct uploads implementation.

All we have is the source code for DirectUploadsController:

def create
  blob = ActiveStorage::Blob.create_before_direct_upload!(blob_args)
  render json: direct_upload_json(blob)
end

private

def blob_args
  params.require(:blob).permit(:filename, :byte_size, :checksum, :content_type, :metadata).to_h.symbolize_keys
end

def direct_upload_json(blob)
  blob.as_json(root: false, methods: :signed_id).merge(direct_upload: {
    url: blob.service_url_for_direct_upload,
    headers: blob.service_headers_for_direct_upload
  })
end

Take a look at the checksum parameter: this is one of mane hidden gems of Active Storage–a built-in file contents verification.

When a client requests a direct upload, it can specify the checksum of the file (MD5 hash encoded as Base64), and the service (e.g., Active Storage itself or S3) will use this checksum later to verify the uploaded file contents.

Let's come back to GraphQL.

GraphQL mutations are pretty much like Rails controllers, so transforming the above code into a mutation is a straightforward:

class CreateDirectUpload < GraphQL::Schema::Mutation
  class CreateDirectUploadInput < GraphQL::Schema::InputObject
    description "File information required to prepare a direct upload"

    argument :filename, String, "Original file name", required: true
    argument :byte_size, Int, "File size (bytes)", required: true
    argument :checksum, String, "MD5 file checksum as base64", required: true
    argument :content_type, String, "File content type", required: true
  end

  argument :input, CreateDirectUploadInput, required: true

  class DirectUpload < GraphQL::Schema::Object
    description "Represents direct upload credentials"

    field :url, String, "Upload URL", null: false
    field :headers, String,
          "HTTP request headers (JSON-encoded)",
          null: false
    field :blob_id, ID, "Created blob record ID", null: false
    field :signed_blob_id, ID,
          "Created blob record signed ID",
          null: false
  end

  field :direct_upload, DirectUpload, null: false

  def resolve(input:)
    blob = ActiveStorage::Blob.create_before_direct_upload!(input.to_h)

    {
      direct_upload: {
        url: blob.service_url_for_direct_upload,
        # NOTE: we pass headers as JSON since they have no schema
        headers: blob.service_headers_for_direct_upload.to_json,
        blob_id: blob.id,
        signed_blob_id: blob.signed_id
      }
    }
  end
end


# add this mutation to your Mutation type
field :create_direct_upload, mutation: CreateDirectUpload

Now, to retrieve a direct upload payload from the server, your GraphQL client must perform the following request:

mutation {
  createDirectUpload(input: {
    filename: "dev.to", # file name
    contentType: "image/jpeg", # file content type
    checksum: "Z3Yzc2Q5iA5eXIgeTJn", # checksum
    byteSize: 2019 # size in bytes
  }) {
    directUpload {
      signedBlobId
    }
  }
}

...and some JavaScript

Disclaimer: the JS implementation below is just a sketch and hasn't been tested in reality (since in my project we don't use any of Rails' JS code). All I checked for is that it compiles.

To upload a file, a client must perform the following steps:

  • Obtain the file metadata (filename, size, content type and checksum)
  • Request direct upload credentials and a blob ID via API – createDirectUpload mutation
  • Upload the file using the credentials (no GraphQL involved, HTTP PUT request).

For step 1 and 3 we can re-use some of the code from the JS library that comes along with Rails (don't forget to add "@rails/activestorage" to your package.json).

Let's write a getFileMetadata function:

import { FileChecksum } from "@rails/activestorage/src/file_checksum";

function calculateChecksum(file) {
  return new Promise((resolve, reject) => {
    FileChecksum.create(file, (error, checksum) => {
      if (error) {
        reject(error);
        return;
      }

      resolve(checksum);
    });
  });
}


export const getFileMetadata = (file) => {
  return new Promise((resolve) => {
    calculateChecksum(file).then((checksum) => {
      resolve({
        checksum,
        filename: file.name,
        content_type: file.type,
        byte_size: file.size
      });    
    });
  });
};

FileChecksum class is responsible for calculating the required checksum and is used by Active Storage in DirectUpload class.

Now you can use this function to build your GraphQL query payload:

// pseudo code
getFileMetadata(file).then((input) => {
  return performQuery(
    CREATE_DIRECT_UPLOAD_QUERY,
    variables: { input }
  );
});

Now it is time to write a function to upload files directly to the storage service!

import { BlobUpload } from "@rails/activestorage/src/blob_upload";

export const directUpload = (url, headers, file) => {
  const upload = new BlobUpload({ file, directUploadData: { url, headers } });
  return new Promise((resolve, reject) => {
    upload.create(error => {
      if (error) {
        reject(error);
      } else {
        resolve();
      }
    })
  });
};

Our complete client side code example would be the following:

getFileMetadata(file).then((input) => {
  return performQuery(
    CREATE_DIRECT_UPLOAD_QUERY,
    variables: { input }
  ).then(({ directUpload: { url, headers, signedBlobId }) => {
    return directUpload(url, JSON.parse(headers), file).then(() => {
      // do smth with signedBlobId – our file has been uploaded!
    });
  });
});

Looks like we did it! Hope that helps you to build your awesome new Rails + GraphQL project)

For more realistic example, check out this snippet from our React Native application: https://gist.github.com/Saionaro/7ee0e2c02749e2729dc429c9e9bfa7f3

In Conclusion, or what to do with signedBlobId

Let me provide just a quick example of how we use signed blob IDs in the app–the attachProfileAvatar mutation:

class AttachProfileAvatar < GraphQL::Schema::Mutation
  description <<~DESC
   Update the current user's avatar
   (by attaching a blob via signed ID)
  DESC

  argument :blob_id, String,
            "Signed blob ID generated via `createDirectUpload` mutation",
            required: true

  field :user, Types::User, null: true

  def resolve(blob_id:)
    # Active Storage retrieves the blob data from DB
    # using a signed_id and associates the blob with the attachment (avatar)
    current_user.avatar.attach(blob_id)
    {user: current_user}
  end
end

That's it!


Read more dev articles on https://evilmartians.com/chronicles!

Posted on Apr 2 '19 by:

palkan_tula profile

Vladimir Dementyev

@palkan_tula

A mathematician found his happiness in programming Ruby and Erlang, contributing to open source and being an Evil Martian.

Evil Martians

Evil Martians is a distributed product development consultancy that works with startups and established businesses, and creates open source-based products and services.

Discussion

markdown guide
 

great write up for how to use ActiveStorage with GraphQL, I am using Apollo at the client side, I am wondering how your client works to send the request to GraphQL endpoint to create the direct upload, in my case, I have to wrap it in a component to do the mutation, not sure how you do it

 

We stopped using Apollo a while ago, now we use a little bit enhanced fetch.

Here is the example of direct upload functionality from our React Native client: gist.github.com/Saionaro/7ee0e2c02...

 

cool, thank you for sharing that :)

 

Hi Vladimir! This is very interesting! I've implemented direct upload in the past but using REST and the aws-sdk-s3 Ruby gem. The principle was the same. JS asks to the Rails server the credentials, and then uploads directly to S3. It was sweet (and almost indispensable since the app ran on Heroku).

As a part of porting the old codebase into the new app, we migrated from CarrierWave to Active Storage.

do you have a resource about this?

Another good news–this API is abstract, works with any service supported by Active Storage (i.e., filesystem, S3, GCloud, Azure).

This is one of the greatest parts! Though you still need to migrate the files :D

ActiveStorage sounds promising, I hope they add all the remaining features. Rails frameworks sometimes leave things on the simpler side...

 

do you have a resource about this?

We didn't migrate the data, only the code; so it was just like replacing mount_uploader with has_attached.

The only tricky part was implementing variants, 'cause this functionality hasn't been merged yet. So we had to a small patch.

 

We didn't migrate the data, only the code; so it was just like replacing mount_uploader with has_attached.

What happens if you don't migrate the data? Do you lose a reference to all the previous images?

We started a new version from scratch in terms of users generated data (like Basecamp 3 after Basecamp 2: some code is shared but not data)
So, we didn't lose anything, 'cause you can't lose nothing)