DEV Community: Maarten van Vliet

TIL - Implementing Encoder for NifStructs

Maarten van Vliet — Thu, 29 Dec 2022 12:07:50 +0000

I am currently using Rustler, a tool that allows for Elixir to interact with the Rust ecosystem. As a newcomer to Rust, working on Rust-Elixir crossover libraries helps me understand its quirks.

Today, I was trying to use a SQL parser in Rust and convert the results to Elixir terms. Rustler's NifStructs feature made this process convenient. However, I encountered an issue with a data structure defined as follows:

#[derive(NifStruct)]
#[module = "SqlParser.Select"]
pub struct Select {
    pub selection: Option<Expr>
}

The issue arose when I encountered this enum:

pub enum Expr {
    BinaryOp {
        left: Box<Expr>,
        op: BinaryOperator,
        right: Box<Expr>,
    }
}

Because of the Box<Expr>, this becomes a recursive data structure. To encode this with Rustler, I needed to implement the Encoder trait for Box<Expr>.

impl Encoder for Box<Expr> {
    fn encode<'a>(&self, env: Env<'a>) -> Term<'a>  {
        let data =&** self;
        data.encode(env)
    }
}

However, this caused a chain reaction. The Select struct above has a Decoder trait implemented for it, and it expects Expr to also implement that trait to decode. At first, I implemented the Decoder trait for Expr, but since I only need to use the parser one-way (from SQL through Rust to Elixir) and not vice-versa, this was not necessary.

I had to do a bit of digging but found that I could use:

#[derive(NifStruct)]
#[rustler(encode)]
#[module = "SqlParser.Select"]

and the Select struct would never implement the Decoder trait, resulting in that the Expr also did not need to implement it.

So annotate the code with #[rustler(encode)] or #[rustler(decode)] if you only need one-way
encoding/decoding in Rustler.

Using serverless Ruby on AWS Lambda to resize images

Maarten van Vliet — Fri, 04 Jan 2019 09:08:54 +0000

Republished from (https://maartenvanvliet.nl/2019/01/04/ruby_aws_lambda/)

At the last AWS ReInvent, it was announced that AWS Lambda would support Ruby as a runtime language.
I was eager to try this out, Ruby's powerful syntax and features are a joy to work with and coupling this with AWS Lambda I figured it could be leveraged for some easy image resizing Lambda.

I started off with the serverless framework as this is an easy way provision Lambda functions. The goal is that when an image is uploaded to an S3 bucket, a Lambda is started, it resizes the image, and then uploads it to another bucket.

The first step is to install serverless

npm install -g serverless
serverless create -t aws-ruby -p image-resizer

This creates the basic boilerplate of a serverless framework ruby Lambda function. It will create two files, a handler.rb, the file with the actual function the Lambda will call, and a serverless.yml a file that the serverless framework uses to configure and provision the lambda and affiliated services such as AWS S3 and AWS API Gateway.

We will first change the serverless.yml configuration. We add a function handle_resize that calls the ImageHandler.process function in handler.rb whenever an object is created in the your-images S3 bucket. Furthermore, under the provider key, we grant the function access to S3 and use the ruby2.5 runtime.

We also add an S3 resource resized-your-images, this will be the bucket where we moved the resized images to.

service: image-resizer 

provider:
  name: aws
  runtime: ruby2.5
  iamRoleStatements:
    - Effect: Allow
      Action:
        - s3:*
      Resource: "*"

functions:
   handle_resize:
     handler: handler.ImageHandler.process
     events:
       - s3:
           bucket: your-images
           event: s3:ObjectCreated:*

resources:
  Resources:
    ResizedImages:
      Type: AWS::S3::Bucket
      Properties:
        BucketName: resized-your-images

Now we can implement the ImageHandler.process function.

Writing the Ruby lambda

First we'll add a Gemfile because of two dependencies we need to use. The AWS S3 SDK to retrieve and upload files to S3, and the mini_magick gem, a wrapper for imagemagick.

#Gemfile
source 'https://rubygems.org'

gem 'aws-sdk-s3', '~> 1.30.0'
gem "mini_magick", '~> 4.9.0'

Next we update the handler.rb file.

#handler.rb
require 'uploaded_file'

class ImageHandler

  def self.process(event:, context:)
    event = event["Records"].first
    bucket_name = event["s3"]["bucket"]["name"]
    object_name = event["s3"]["object"]["key"]

    file = UploadedFile.from_s3(bucket_name, object_name)
    file.resize "100x100"
    file.upload_file("resized-your-images", "resized_" + event["s3"]["object"]["key"] )
  end
end

The ImageHandler.process is the function called by the lambda and has two arguments, the event, in our case an event sent by S3 about newly created files, and the context, basically metadata about the function and its environment.

The process function retrieves the bucket and key of the newly created object from the event object. Then it calls into the not yet implemented UploadedFile class. This class retrieves the s3 object, resizes it and then uploads it to another bucket. This class is independent from the lambda, and in general all business logic should be. This makes it easier to use the code in other contexts and also allows easier testing. E.g. I can test the UploadedFile.from_s3(bucket_name, object_name) without having to mock the entire S3 Lambda event.

Now we'll implement the UploadedFile class.

# uploaded_file.rb
require "aws-sdk-s3"
require "mini_magick"

class UploadedFile
    def self.from_s3(bucket_name, object_name)
        s3 = Aws::S3::Resource.new()
        object = s3.bucket(bucket_name).object(object_name)

        tmp_file_name = "/tmp/#{object_name}"
        object.get(response_target: tmp_file_name)

        UploadedFile.new(tmp_file_name)
    end

    def initialize(tmp_file)
        @tmp_file = tmp_file
    end

    def resize(params)
        image = MiniMagick::Image.open(@tmp_file)
        image.resize params
        @resized_tmp_file = "/tmp/resized.jpg"
        image.write @resized_tmp_file
    end

    def upload_file(target_bucket, target_object)
        s3 = Aws::S3::Resource.new()
        object = s3.bucket(target_bucket).object(target_object).upload_file(@resized_tmp_file)
    end
end

The uploaded_file.rb is typical ruby. It retrieves the S3 file, stores it in a tmp-file. We call mini_magick to resize the image, and then uploads the file again.

Deployment

When deploying we need to make sure that the Lambda function has access to the dependencies. Serverless framework will build a zip-file with the code and all dependencies locally and then deploy this to AWS. Therefore it is important that we run the same version of ruby locally as on AWS, so that would be ruby 2.5. If you use rbenv, you can use rbenv install 2.5.0 and rbenv local 2.5.0 to set the ruby version for this project.

Next we need to vendorize the dependencies so they are included in the package. Run bundle install --path vendor/bundle, now the dependencies are in the vendor directory.

Before we can deploy we still need to set the credentials for AWS. See Serverless Framework documentation for more information.

Now that is done you can invoke sls deploy and the lambda function will be deployed and the S3 buckets will be created. Upload an image to the your-images bucket and moments later a resized version will appear in the resized-your-images bucket. If anything goes wrong you can follow that in the log files in AWS Cloudwatch. There will be a log stream for the lambda function. You can also call sls logs -f handle_resize to tail the logs.

Find the code on github

Photo by Paulius Dragunas on Unsplash

Writing your own Absinthe DSL with macros and middleware

Maarten van Vliet — Thu, 03 Jan 2019 18:08:54 +0000

Absinthe is a great library to do graphql in Elixir. However, when writing your resolvers you may find that you are writing some boilerplate multiple times. So, in the spirit of keeping your code DRY, in this post I'll show how we can leverage middleware and a macro to write your own DSL for your Graphql api.

I assume some basic knowledge on how Absinthe works, most of the information can be found on in the guides

A common problem in writing api's is making sure that the queries are only accessible for logged in users. To do so in Absinthe we need to set a current_user on the context of a graphql query, see the guides for an example.

To make sure that our resolvers are only accessible when the current_user is set we can pattern match on its existence.

query do
    field :profile_picture, :string do
      resolve fn
        _, %{context: %{current_user: current_user}} ->
        {:ok, current_user.profile_picture_url}
        _, _ ->
        {:error, "Not logged in"}
      end
    end
end

We match on the presence of current_user and if it is not there we return an error. This can get quite repetitive however if you have a lot of authenticated fields. With some middleware we can clean this up.

# authenticated.ex
defmodule Example.Authenticated do
  @behaviour Absinthe.Middleware

  def call(resolution, _config) do
    case resolution.context do
      %{current_user: %{}} ->
        resolution

      _ ->
        Absinthe.Resolution.put_result(resolution, {:error, "Not logged in"})
    end
  end
end

The middleware implements the Absinthe.Middleware behaviour. This requires a call/2 function that gets passed the current resolution struct. In this struct we can get the context of the current query and find out if the current_user is present. Now we need to call the middleware for our field

query do
    field :profile_picture, :string do
      middleware Example.Authenticated
      resolve fn
        _, %{context: %{current_user: current_user}} ->
        {:ok, current_user.profile_picture_url}
      end
    end
end

We placed the middleware before the resolver, any non-authenticated call will be met with the error and when the current_user is present it is simply passed along to the resolver.

This is already a much cleaner option and we can leave this as is. Nonetheless, to showcase how you can use a macro to make a cleaner DSL for your app I'll give a little example.

Create a new file with your macro definition:

defmodule Example.IsAuthenticated do
  defmacro is_authenticated() do
    quote do
      middleware(Example.Authenticated, %{})
    end
  end
end

This macro does very little, I declare it and the only thing it does is call the middleware/2 we have seen earlier with the Authenticated middleware. When the macro is invoked it will be rewritten into everything in the quote block.

We can rewrite our resolver like this

import Example.IsAuthenticated 
query do
    field :profile_picture, :string do
        is_authenticated()
        resolve fn _, %{context: %{current_user: current_user}} ->
            {:ok, current_user.profile_picture_url}
        end
    end
end

Note that we need to import to macro for it to work. In our field definition the macro is_authenticated is called and rewritten into the call for the middleware. As you can imagine the middleware/2 is also a macro.

This is a very simple example of how you can use macros to do code generation for you. For graphql there are many options regarding e.g. authorization or pagination where macros can keep your code more DRY.

Republished from (https://maartenvanvliet.nl/2018/12/29/absinthe_macro_dsl/)

Photo by gil on Unsplash