DEV Community: Ike Gabriel Yuson

Serverless way of uploading pictures to an S3 Bucket

Ike Gabriel Yuson — Mon, 06 Jun 2022 16:49:40 +0000

In creating full-stack applications, there will come a time when a developer undergoes R&D of how to store pictures and files and dynamically allocate them to a specified user. Cases such as profile pictures, photo uploads and other dynamic files that can be transmitted as binary data that are unique to each user are some of the entry-level challenges a developer will experience. And one of the most efficient way of handling these things are through cloud services such as Amazon S3.

Uploading a file or photo to an Amazon S3 bucket is easy in a server-full architect-ed platform. In the case of hosting a backend service in an EC2 platform, one way to upload a file or photo is to upload the image to the server and then the server pushes the image to the S3 bucket. The client then have to call a get request to the server if it wishes to get a resource from the S3 bucket. Another way of uploading is to immediately upload the the image from the frontend to the S3 bucket. This way, the server does not have to handle the image data being passed on from backend to the S3 bucket.

However, if your choice of architecture is a serverless one and that you want the uploading of an image to an S3 bucket be handled by your lambda functions then there is a way for that too! This article would be a step by step guide of how to upload images through lambda functions!

For this example, we would be using AWS CDK as our IaC of choice.

Overall Process:

Configure the infrastructure by creating a lambda function and an S3 bucket construct.
Create an IAM policy statement for the said lambda function that has get and put access to the S3 bucket.
Edit the lambda function handler where it returns a signed URL from the S3 bucket.
Invoke the lambda function.
Upload image using the signed URL (this is a PUT request) from the client.

First is to create an S3 bucket construct.

import * as s3 from 'aws-cdk-lib/aws-s3'

new s3.Bucket(this, 'bucketID', {
  bucketName: 'bucket name'
  publicReadAccess: true,
  cors: [{
    allowedMethods:[ // Allow methods
      s3.HttpMethods.GET,
      s3.HttpMethods.POST,
      s3.HttpMethods.PUT,
    ],
    allowedOrigins:['*'] // Just for the sake of local and dev development
    allowedHeaders:['*']
  }]
})

Next is to create a lambda function construct. This lambda function returns a signed URL from Amazon S3.

import * as lambda from 'aws-cdk-lib/aws-lambda'

const getsignedurl = new lambda.Function(this, 'function', {
  runtime: lambda.Runtime.NODEJS_14_X,
  code: lambda.Code.fromAsset('path/to/function'),
  handler: 'function.handler',
  environment: {
    'UploadBucket': BUCKET_NAME // 'bucket name'
  },
})

Add a Amazon S3 policy statement to be granted to the newly created lambda function.

// s3Bucket Policy for profileImageUpload lambda function
const s3BucketPolicy = new iam.PolicyStatement({
  effect: iam.Effect.ALLOW,  
  actions: ['s3:getObject','s3:putObjectAcl', 's3:putObject'],  
  resources: [`arn:aws:s3:::${BUCKET_NAME}/*`]
})

// Added S3 policy to image upload lambda function
getsignedurl.role?.attachInlinePolicy(
  new iam.Policy(this, 'uploadPolicy', {
    statements: [s3BucketPolicy]
  })
)

Also, don't forget to integrate the lambda function in its corresponding API gateway to serve as a trigger!

The code below will be the logic inside your newly created lambda function.

const AWS = require('aws-sdk')

const s3 = new AWS.S3()
const URL_EXPIRATION_SECONDS = 300

exports.handler = async function (event: any) {
  return await profileImageUpload(event)
}

export const profileImageUpload = async (event: any) => {
  try{
    const Key = `${userID}.jpg`

    const s3Params = {
      Bucket: process.env.UploadBucket,     
      Key,      
      Expires: URL_EXPIRATION_SECONDS,      
      ContentType: 'image/jpeg',      
      ACL: 'public-read'    
    }

    const uploadURL = await s3.getSignedUrlPromise('putObject', s3Params)

    return {
      body: JSON.stringify({
        uploadURL: uploadURL,        
        Key
      }),      
      headers: { "Content-Type": "application/json" },      
      statusCode: 200    
    }
  }catch (error){
    return error
  }
}

The userID will serve as a unique key for the name of the object uploaded to the S3 bucket. This lambda function will return an upload URL and a Key. This upload URL will be used to send a PUT request with the request body having a binary file which is the picture or object to be uploaded.

The example below is a Postman call of our newly created lambda function which returns an upload URL and a Key. The Key is part of the object URL in the cloud which makes it accessible if you stitch together your own bucket URL and Key!

Now call a PUT request to the returned upload URL and attach the picture in the request body. It shown in the example below.

Click send and go check your newly uploaded photo in the AWS console!

Debugging lambda functions with AWS CDK (using Webstorm)

Ike Gabriel Yuson — Mon, 02 May 2022 12:01:41 +0000

PREREQUISITES

Install AWS CDK and AWS SAM (along with Docker)
Install Webstorm
Install AWS Toolkit Webstorm plugin

INTRODUCTION

Step-debugging lambda functions in AWS CDK applications can be quite tricky since the invocation of lambda functions behave of that of a docker container.

AWS SAM can be used to invoke/run your lambda functions locally with the following commands:

$ sam local start api
$ sam local invoke

Partnered by API clients (e.g Postman, Insomnia) and/or CLI tools to transfer data using various network protocols (e.g curl), local development can be quite robust with AWS CDK.

However, step-debugging is not quite as seamless as locally invoking lambda functions with CDK. Through docker containers, the behavior of invoking local lambda functions mimics that of a lambda function deployed in the cloud.

In the cloud, lambda functions are ephemeral where execution environments only exist for a brief time when the function is invoked. This behavior is exactly the same with that of local development through docker containers. When a lambda is locally invoked, Docker builds an image that of the lambda function and runs the container. After the runtime of the lambda function, the container then stops and gets deleted. Can you see the parallelism with deployed lambda functions in the cloud? Turns on when called and then goes back to sleep after execution.

When locally invoking a lambda function, an API call with the use of an API client will only call the container that has successfully been built. This is the reason why we cannot simply attach a debugger using our IDEs - a containerized application (in this case a lambda function) must be debugged inside the container as well.

In order to debug the lambda function locally, we must attach a debugger as soon as we invoked the lambda function.

WALKTHROUGH

First we need to configure the AWS toolkit plugin by updating your AWS credentials. You will be asked for your AWS ACCESS KEY ID and your AWS SECRET ACCESS KEY. After configuring your AWS credentials, go to Settings → Tools → AWS. Then select the drop down button of the AWS tab then select “Lambda”. Check the “Show gutter icons for all potential AWS Lambda handlers”.

Then go to your lambda function handler. You will now be able to see a lambda function logo in the gutter next to your handler.

Click the lambda function logo and select “Modify Run Configuration”.

Fill-out the necessary information about your lambda function. The input portion is asking for an event that invokes the lambda function. This mimics the actual invocation of a lambda function from the cloud. The event that I used in this example is an API Gateway AWS Proxy where it invokes a lambda function through an API call from AWS API Gateway service.

This is an example of an API Gateway AWS Proxy event. Looks daunting? Don’t worry! AWS Toolkit provides you templates to use for specific events located at the configuration settings of your lambda functions.

Click on the “-- Event Templates --” drop down and AWS Toolkit will provide you with ALL the events that invokes lambda functions in AWS.

Moving forward, go to the SAM CLI tab and select “Build function inside a container”. This is the safest way to avoid NodeJS version errors. An example would be if you are using a different version of Node and the compatible runtimes of your lambda functions are different too. It’s best to containerize the debugging session to an isolated environment to avoid these types of errors.

Apply all your changes and you are all set! To debug your lambda function, setup some breakpoints and click the lambda function logo and run “Debug”!

PROBLEMS

The walkthrough above seems seamless and quite easy right? However, lambda functions invoked from the handler does not take into account lambda layers. You might ask why can’t we just select the “From template” option? But the thing is, with CDK, there is no template.yaml file. You might also suggest to convert the .template.json file to a template.yaml with the command:

$ cdk synth --no-staging > template.yaml

Yes, this converts the template.json file to a template.yaml however, the template.yaml produced is NOT the same as the template.yaml of AWS SAM. This then will lead to errors in compile time when AWS Toolkit runs:

$ sam build

We also don’t want to create another template.yaml file with the correct AWS SAM format. This will just only mimic your current IaC (Infrastructure as Code) with CDK which may lead you to confusion in the future since editing infrastructure will lead you to change both AWS CDK’s stacks and AWS SAM’s template.yaml file. A problem may arise where you deployed your infrastructure through the AWS SAM CLI instead using AWS CDK. Remember that SAM is a tool for CDK - not the other way around. Also, DRY!! - Don’t Repeat Yourself.

WORKAROUND

My workaround is to create a package.json file with all the dependencies inside the lambda functions folder. I know what you’re thinking, aren’t I repeating myself? Yes, I am but this is only for local development and great developer experience for us to use step-debugging in lambda functions with AWS CDK. If we were using AWS SAM this maybe quite easier for us but CDK is quite more robust when it comes to infrastructure. I consider it as the “true” IaC since we can also test the infrastructure with testing libraries such as Jest.

Install all the dependencies and don’t forget to put “node_modules”, “aws-toolkit-ts-output”, and “aws-toolkit-tsconfig.json” in the gitignore since these files were created when AWS Toolkit ran

$ sam build

If you don’t want to use step-debugging, you can just simply invoke the API Gateway with the command

$ sam local start-api

and call you’re API endpoints with an API Client such as Postman or Insomnia.

As said above, after creating a package.json and install the needed dependencies, just setup some breakpoints in your lambda function and click the lambda function logo located at the gutter and run “Debug”!

My Love Story With Tech

Ike Gabriel Yuson — Mon, 31 Jan 2022 16:30:40 +0000

My experience with tech wasn't exactly a bed of roses. It's more of a romantic relationship where there were ups and downs, misunderstandings, and euphoria. Tech can be very complex but is never incomprehensible.

I am Ike Gabriel Yuson, currently a 2nd year student taking up a bachelor’s degree in Computer science at the University of the Philippines - Mindanao. I am also a freelance web developer who is also gunning for internships to add to my experience in the industry. Up until this point in my life, I only realized how big, beautiful, and challenging the tech industry is.

Although it was not until high school when I was exposed to tech. During this time, I had my fair share of competitive programming competitions and various hackathons both on-site and online. It was where I discovered different fields in computer science.

Stage 1: Butterflies

People usually incorporate “love-at-first-sight” to a person but in 8th grade, I also had mine. With tech. This was the time I was introduced to programming and computer science. The exhilaration and the rush of adrenaline I experienced when writing my first Hello World program in Java was on the roof! That day I realized I want to come across this feeling time and time again and until now, I still have the same intense addictive rush when solving problems in tech.

All throughout high school, I was able to discover various fields in computer science such as web development, game development, and data science which made me love the industry even more.

Stage 2: Assimilation

Upon going into college, I was already cognizant of what profession I was going to take - to work in the tech industry. A very broad ambition, right? During this time, there were only two things I was sure about - first is that I love programming, and second, I would be taking up a bachelor’s degree in computer science. Never really did I thought about what specialization I would want to take may it be web development, game development, and data science, etc. I was naïve for thinking that a college diploma was all I needed to jump start my career to be a successful computer scientist.

Just around the 2nd semester of my 1st year in college, as I was coding data structures for a course requirement, a thought then hit me - will I really be able to apply this in a professional corporate level? To be fair, I do appreciate the knowledge and logic imparted when studying data structures. Up until now, I practice coding them in HackerRank for I consider it as physical exercise for a programmer's brain and are even used as interview questions when applying for various tech jobs.

However, as I was browsing different job opportunities in the industry, I then noticed that I never even got to learn all these "alien" technologies in the job requirements section such as React, Nodejs, Django, etc. Never did I saw a job requirement indicating "Professional in implementing binary trees and graphs". Impostor syndrome struck and it was precisely at that moment that I considered myself clueless when it comes to the dirty work and technical skills required for a certain tech job.

I then looked at my degree program curriculum and to my surprise, nothing indicated that I would be learning these technologies. I then realized that this was one of those up-to-date technologies that are not taught in my university. And so, I took matters into my own hands.

I decided that I would want to learn web development first. Mainly because I see the web developer job at least 80% of the time when searching for job opportunities.

Stage 3: Building

Learning React was my first goal in mind and this was the start of my insatiable hunger towards learning new things. Up until that point, I was more of a go-with-the-flow type of guy. I just wait what my teachers and professors to tell the class what to study and do and I then follow. I never considered myself as self-taught up until now.

The knowledge I had then for React was then enough to make my own personal website to be used for internship applications. I was fortunate enough to be accepted to a remote web development internship program overseas. This was where I learned about various architectures and technologies fit for full-stack development. Along with my peers, we made a single-page application that rewards loyal customers of small businesses. This was my first encounter with the MERN stack.

During the first meeting with my mentor, I was extremely nervous to the point that I wanted to back-out from the program as he was spouting terms and technologies I haven't encountered and couldn't understand. Impostor syndrome struck once again, and my hands were basically shaking while I was typing my notes as he talks. After the technical talk, I then asked him if I could learn all these technologies in a short amount of time and the only advice he told me was to take it slow.

I then heed his advice and started to research and develop. First up was everything about the MERN stack - MongoDB, Express, React, and Nodejs. For months, I was never aware of the time. When I wake up in the morning, I immediately read articles and watch tutorial videos and at the same time code our MVP. I never noticed that I was already at it for 12-hours straight and next thing I knew it's already 3am in the morning.

One thing led to another, and I already found myself studying about technologies irrelevant to the internship's tech stack! During this time, I always wondered how to publish a full web application on the internet so I was able to fiddle around with docker, switched from Windows to Linux (because why not? 😂), studied web servers like NGINX, and even read articles about AWS.

When my mentor told me that we would be implementing a Serverless microservice architecture with AWS I was super excited upon hearing the news!

This was the time I utterly fell in love with DevOps.

Stage 4: Stability

My tech journey so far has been exhausting and hopefully far from over. Yes, it was quite tiresome and never was an easy ride however, I never thought about stopping. You see, I can be quite the persistent guy, and that’s what I’m doing with programming, too. May it be developing a web application or automating a certain job in CI/CD pipeline, I would never get tired of it!

I realize that this industry will make you feel like you're a complete beginner every now and then. Technology evolves at an exponential rate and who knows, the tech we have now would no longer be relevant a year or a couple of years later.

Additionally, getting Linkedin messages for job opportunities makes me blush every time for these messages are actually a big deal for me considering that I am still an undergrad. This made me realize that there are more valid credentials than that of a college diploma.

Having experienced full-stack development granted me a taste of what I decided to be my profession for the rest of my life. Although dealing with impostor syndrome every now and then can be a pain in the ass, but reliving how clueless I was before and assessing how far I have come somewhat does the trick. My curiosity led me to where I am now.

As of writing, I have now made a carpool tracker web application that follows a serverfull microservice architecture. This was when I got to be hands-on with docker and other Amazon web services such as ECS, EKS, and ECR. I am currently working on an ElectronJS project and studying container orchestration with Kubernetes.

Upskilling as early as now has been one of the best decisions of my life and I can't wait to learn new things in the future!