DEV Community: giuseppe

Image manipulation Slack slash command with AWS

giuseppe — Sat, 25 Jan 2020 13:10:57 +0000

After being able to successfully send messages back to Slack, another chance to itengrate Slack with AWS came up. The idea was simple, I had an image to which I wanted to add arbitrary text. I envisioned this as creating a call from Slack with Slack Slash Commands and simply answer it with AWS Lambda... but....NO.

The idea

To be able to achieve this I ended up having to use quite a few pieces from the AWS stack:

Setup Slash command to send a call with the desired text as payload
Lambda Function to deal with the request
Amazon SNS messaging system to publish the requests
A lambda function that processes the image and sends back te response to Slack

Let's tackle these steps one by one.

Slack slash command

This is the easy part. All we have to do is setup an Slack App and then setup an slack command.

In the configuration of the Slack command a URL has to be defined, which is were the Slack command would point at. This call, as specified in Slack Slash Commands documentation sends a application/x-www-form-urlencoded content-type header and a body with, among other things, the slash command's payload which is text we will be printing in out image. The header is relevant as we will see when setting up our Lambda function.

There's another important constraint regarding slash commands. This is, the answer to the request made by slack cannot be more than 3000ms. We will later see that the image processing took longer sometimes times. For these cases, Slack sends a parameter response_url where the final response should be sended up to 30 minutes later.

AWS Services

API Gateway + Lambda + SNS + S3

To be able to offer a response and then took our time to process the image and later respond with it, we will need an API Gateway endpoint pointing to a first LAMBDA, we will call this lambda proxyLambda.

To handle long request asynchronously we will make use of AWS SNS, this is a messaging platform from Amazon and it will let us use the first proxyLambda to publish a message into AWS SNS and have antoher lambda subcribed to SNS for reading and processing the requests, we will call this Lambda subscriberLambda.

Now, once our subscriberLambda has finish it needs to store the new image in an S3 bucket, since Slack is expecting just a text message with a link to an image to desplay.

With all this the flow should look something like this:

API Gateway receives request from Slack and proxies it to proxyLambda
proxyLambda publishes a message to SNS with the request data and answers to Slack ASAP
subscriberLambda gets trigger on the new message and processes the message, generating the image

API Gateway configuration

The idea for the Gateway configuration is to get Slack request proxied straight to the lambda. The reason behind this is what we previusoly said about the Slack slash command request type. The standard configuration to attach an AWS API Gateway to a lambda funtion is via a method (say GET or POST), the gateway expects a payload to be in json, if not it crashes and the request never gets to the lambda. I was able to solve this issue by usin the proxy configuration while configuring a new resource.

The configuration to sned the request to the lambda it's also pretty straightforward.

Proxy Lambda function

Now once we have our requests proxied to our lambda, we will use it to handle it and push it to our SNS channel. Doing it like this we can respond back to Slack very fast and handle the actual request later.
The lambda function would look like this.

console.log("Loading function");
var AWS = require("aws-sdk");

exports.handler = function(event, context, callback) {
    var eventText = JSON.stringify(event.body, null, 2);
    console.log("Received event:", eventText);
    var sns = new AWS.SNS();
    var params = {
        Message: eventText, 
        Subject: "Test SNS From Lambda",
        TopicArn: "arn:aws:sns:us-east-1:333333:your-SNS-topic"
    };
    sns.publish(params, context.done);

    callback(null, {
        statusCode: 200,
        body: "temporal response to Slack...",
        headers: {'Content-Type': 'application/json'}
    });
};

This is actually all we need to forward the request to SNS. The important part here is the Topic ARN, this gets created
when we create our Messaging channel.

Subscriber Lambda

Here is the lambda that would do the actual work, this functions would be listening for messages in the SNS channel
and will act accordingly when a new one comes. Lets see how the main part looks like. Bear in mind, there's a lot happening
in here: Reading SNS message and parsing the request parameters, new modified image generation, storing the new image in an
S3 bucket and finally sending the link and response back to Slack.

const engine = require('./ImageGenerator'); // This is a separate file with the image processing code
var uuid = require("uuid"); // Id generator for the new images
const fs = require('fs');
const AWS = require('aws-sdk');
var request = require('request');
var s3 = new AWS.S3();

exports.handler = async function(event, context, callback) {

    // This section reads the SNS message and parses the request parameters
    var message = event.Records[0].Sns.Message;
    let buff = Buffer.from(message, 'base64');
    let text = buff.toString('ascii');
    let rawParams = text.split('&');
    let params = new Object();

    for(var param of rawParams) {
        let paramSplitted = param.split('=');
        params[paramSplitted[0]] = paramSplitted[1];
    }

    // Now we will use some methods from an imported file to process and add the requested text to the image.
    // For some reason the text from the request puts '+' instead of spaces so we get rid of those
    console.log(params);
    let newImage = await engine.generate(decodeURIComponent(params.text).replace(/\+/g,' '));


    // Now we send the request to S3 to save the newly creted image; now here we wait for the response to 
    // be able to generate the slack message with the new url for the file. 
    var paramsS3 = {
        "Body": newImage,
        "Bucket": "YOU-BUCKET-NAME",
        "Key": uuid.v4() + '.jpg',
        "ACL": "public-read",
        "ContentType": "image/jpg" 
    };
    s3.upload(paramsS3, async function(err, data){
        if(err) {
            callback(err, null);
        } else {
            console.log(params.response_url);
            await sendSlack(data.Location, decodeURIComponent(params.response_url));
        };
    });

    // Lambda response is not really relevant, since our objective of sending the slack message has been previously achieved
    callback(null, null);
};

async function sendSlack(data, url) {
    const options = {
        url: url,
        method: 'POST',
        headers: { 
            'Content-Type': 'application/json',
            'Authorization': "Bearer xoxp-r46y54byeby6erny6erny6erny6ern"
        },
        body: JSON.stringify({
            'replace_original': true,
            'response_type': 'in_channel',
            'text': '',
            "attachments": [{
                'text': '',
                'image_url': data
            }],
        })
    };

    request(options, function(err, res, body) {
        let json = JSON.parse(err);
        console.log(json);
    });
}

The full lambda with dependencies can be found in this GitHub Repository
Now take in to account that your code can get a little bit to big and labmda user interface will not let you handle it from there
so you will need something like this to deploy the files from you local environment.

aws lambda update-function-code --function-name your-lambda-name --zip-file fileb://the-zip-with-the-code.zip --region us-east-1

This AWS-CLI command requires you first zip your code, so a tiny script does the both could become handy at this point, and also,
if the intention is to automate the whole deploy for example.

Amazon SNS configuration

Ok, Now lets configure the magic of connecting these to Lambdas, a.k.a. The message broker configuration.
This is very easy when done from the user interface. We go to Amazon SNS configuration in the AWS console and
crete a new Topic. Just fill the Name and Description and leave the rest for now.

Now the important part, once the Topic is created we need to create a New Subscription for this topic, and select our
subscriber lambda here.

This is how we will get our lambda executed every time there's a new request in the queue.

Final thoughts

Altought being a cool approach to test all theese technologies, i'm pretty sure I'll would have gone with a different one when realising my image processing was taking so long for Slack. 3 seconds should be enough in my opinion. In any case I have learned a lot and I hope you do too.

Any thoughts?, just ask!, thank you!

Scheduled REST call with AWS Lambda and Slack feedback

giuseppe — Sun, 04 Aug 2019 18:43:25 +0000

The reason behind this short post came for two reasons. First, one of our recurrent tasks at work was making a call to a REST service to create a web-hook to it. Second, I've been playing around with Amazon Web Services for a while and thought that Lambdas were a perfect service for running simple scripts. I did some research and of course I was not the only one. Once this process is done we usually inform that it's ready via slack message, so I also thought it could be cool to get that done too.

AWS Lambda

Amazon Lambda lets you choose the language for the functions you need, I choose Python just to test it, and because it has cool and well tested modules, as the Requests one, that we'll be using.

Setting up a lambda function is very simple following the official documentation everything to code and test is available straight from the amazon console. So let's see what we need to do next.

Breakdown

These are the three things we want to accomplish

Call out web service.
Call response to Slack.
Set a trigger for once-a-week execution

Service Callout

Assuming you have already have a python script on a lambda in front of you, for calling to these services we will be using the Requests module this library will make easy to deal with all the REST operations, and it's a good choice since it can be taken from the libraries that are provided inside a lambda by default. This rises a very important point, which is any dependencies needed, have to be manually added to the lambda, but there are libraries that are already available inside a lambdas function.

The following code imports the requests module from the available modules and does the needed call to the service. In my case in needed to be a POST call and it had to include a bearer token for authentication, so I created a couple of extra method.

import json
import datetime
from botocore.vendored import requests

api_url = 'https://someservices.com/getSomeStuff'
myToken = 'someBearerToken'

def getHeader():
    return {
        'Content-Type': 'application/json', 'user-agent': 'my-app/0.0.1',
        'Authorization': 'Bearer {0}'.format(myToken)),
        'user-agent': 'my-app/0.0.1'
    }

def informPayload(region):
return {
    'payload1': 'value1',
    'payload2': 'value2',
    'payload3': 'value3'
}

def makeCall():
    headers = getHeader()
    payload = informPayload()    
    response = requests.post(api_url, headers=headers, json=payload)

    #I'm also interested in giving feedback when it fails (code 422) so i also added it as a returned response
    if (response.status_code == 201 or response.status_code == 200 or response.status_code == 422):
        return json.loads(response.content.decode('utf-8'))
    else:
        print(response.content)
        return None 

#This is the default handle for the lambda, it will be the entry poing
def lambda_handler(event, context):
    response = makeCall()

    if response is not None:
        print("Here's your response: ")
        print(response)

    else:
        print('[!] Request Failed')

    return {
        'statusCode': 200,
        'body': json.dumps('Call was successful!')
    }

Feedback through Slack

Now that we have our call code, we can use the same process of calling an external service to call Slack API and send a message to our team channel with the response. To get credentials to do so, you will have to go over the steps on Slack to create a new Salck APP and install it in your team's Slack workspace. You can follow the official slack docs for this.
With this information we are going to modify the lambda_handler method and add a new sender method, here you have the new methods.

#Updated
def lambda_handler(event, context):
    response = makeCall()

    if response is not None:
        slack_msg = '@here is the result of your call: \n \n'
        slack_msg += json.dumps(response)
        sendSlack(slack_msg)
        print("Here's your response: ")
        print(response)

    else:
        print('[!] Request Failed')

    return {
        'statusCode': 200,
        'body': json.dumps('Avinode Hook was executed!')
    }

#New Method
def sendSlack(message):
    slack_post_message_url = 'https://slack.com/api/chat.postMessage'
    slack_bearer_token = 'yourOwnToken'
    headers = {
        'Content-Type': 'application/json',
        'Authorization': 'Bearer {0}'.format(slack_bearer_token)
    }
    body = {
        'channel': 'test2',
        'text': message
    }
    response = requests.post(slack_post_message_url, headers=headers, json=body)

Scheduled Execution

Now, to execute your script on a regular basis Lambdas can be triggered by events, this events can be configured to be CRON expressions, this is the ultimate tool to schedule tasks and is supported natively by lambdas. Following these two tutorials from official Amazon docs we can have our function scheduled in no time.

This one will help us set up an event.
The next one will show us how to make the event a CRON expressions.

Take into account that the event minimum time between executions is 1 minute, so the position in the CRON expression for seconds it's not used.

Final Thoughts

I think this is a very cheap an reliable way to have recurrent scheduled processes done, specially if you don't have the infrastructure or you don't want to waste a machine to have this done. On the other hand it will probably not be suitable for time-consuming tasks since Lambda service is charge on processing time, it could become quite expensive.

Differential deploy to Amazon S3 from BitBucket

giuseppe — Tue, 02 Jul 2019 21:29:40 +0000

This is my first post on DEV.to, very glad to participate and happy to pump up those writing skills.

Under the idea to automate the deployments of our static files and as a step toward a CI/CD environment, and taking adventage from the fact that we recently move all the static files to AWS S3 buckets, I've decided to give it a try automating our process.

Current state

Our deployment process (the integration one) has two big elements to deploy: A Salesforce package. And a folder with all the static files that the Salesforce package uses, including front end code.

When this Salesforce package is created the repository is tagged with a TEST_X name.

Currently the process to deploy statics goes like this:

Go to Source Tree.
Check the files that changed between current and the last TAG.
Copy these files into AWS S3
Invalidate Cloudfire cache (Amazon's CDN... sort of)

Once this first big deploy has been done it can happen also that minor changes that only require static deployment get deployed individually. Without the need of generating another package.

The new apporach

Use Bitbucket pipelines to execute a script with what we just described.

There is already a very good pipeline for AWS S3 ready to use. The only drawback is that the sync commando used to copy the files to S3 uses a combination of timestamp and size of the files to decide wether do the upload or not. Since in BitBucket the pipe executes from a docker container the files that get clones from GIT always have a more recend timestamp than the ones on S3, meaning all the files get copied no matter what.

The only new thing the script will do will be:

Use GIT diff to get the changed files (if we are in a TAG or not changes the behaviour)
AWS CLI to send the specific files
AWS CLI to invalidate cache programatically

Simple Bitbucket Pipe

The first step is to create a pipe from where we will execute the script.
I think the official documentation does quite a good job on this matter. Remember we will use the Simple Pipe to keep files and code at a minimum.

The only part that I missed from the bitbucket docs was that it makes you think it's not possible to access repository defined variables from the simple pipe. This is half true since you can actually pass them when using the pipe in your project's bitbucket-pipelines.yml file.

Dockerfile

The instructions leave you with a three file project. Let's start with the Dockerfile.

FROM atlassian/pipelines-awscli:1.16.185
RUN apk update && apk add ca-certificates && rm -rf /var/cache/apk/*
COPY pipe.sh /
ENTRYPOINT ["/pipe.sh"]

The only difference from the provided is the first line

FROM atlassian/pipelines-awscli:1.16.185

This provides us with the AWS CLI tools pre installed, so we don't have to do it on building time.

Pipe Script

Next, we have the script we will use as a pipe, and it would be the one that contains the GIT commands, the deployment and the cache invalidation.
I based this script heavily in what was exposed in this article. Let's have a look.

#!/usr/bin/env bash
#TODO: Apply colors and errors with commons.sh
set -ex

# mandatory parameters
# path variables should not contain slashes
AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID:?'AWS_ACCESS_KEY_ID variable missing.'}
AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY:?'AWS_SECRET_ACCESS_KEY variable missing.'}
S3_BUCKET=${S3_BUCKET:?'S3_BUCKET variable missing.'}
LOCAL_PATH=${LOCAL_PATH:?'LOCAL_PATH variable missing.'}
TAG_REGEX=${TAG_REGEX:?'TAG_REGEX variable missing.'}
CLOUDFRONT_DISTRIBUTION_ID=${CLOUDFRONT_DISTRIBUTION_ID:?'CLOUDFRONT_DISTRIBUTION_ID variable missing.'}


echo "Starting deployment..."
echo "AWS_ACCESS_KEY_ID is: ${AWS_ACCESS_KEY_ID}"
echo "S3_BUCKET is: ${S3_BUCKET}"
echo "LOCAL_PATH is: ${LOCAL_PATH}"
echo "TAG_REGEX: ${TAG_REGEX}"

# This will give us the last two tags created that comply with the TAG_REGEX parameter
TAGS=()
for i in $( git tag -l --sort=refname "${TAG_REGEX}" | tail -2 ); do
    TAGS+=( "$i" )
done

echo "Checking difference between tags:"
echo "${TAGS[0]}"
echo "${TAGS[1]}"

# We will get the list of files that changed between those two tags but keep only the ones that
# begin with the path we want to deploy
FILES=()
for i in $( git diff ${TAGS[0]} ${TAGS[1]} --name-only | sed -n "s|${LOCAL_PATH}/||p" ); do
    FILES+=( "$i" )
done
echo "Files to be deployed..."
printf '%s\n' "${FILES[@]}"

# Construct the parameters to the AWS CLI for the files to includes
CMDS=()
for i in "${FILES[@]}"; do
    CMDS+=("--include=$i""*")
done
echo "${CMDS[@]}"

# Exclude ALL diles and only includes the previously defined.
# Remove dryrun to deploy to aws s3
# Remember this command won't show any output if the file in the bucket is MORE RECENT than the one in local
echo "${CMDS[@]}" | xargs aws s3 sync ./${LOCAL_PATH}/ s3://${S3_BUCKET}/ --dryrun --delete --exclude "*"

aws cloudfront create-invalidation --distribution-id ${CLOUDFRONT_DISTRIBUTION_ID} --paths /folderToInvalidate/*

The paremeters section at the beggining it's more as a reminder of what are we going to have during the script, these variables will come from the configuration when using the pipe. It is worth remind you that all the paths should not contain slashes, this is critical
since AWS S3 doesn't allow dopuble slashes.

We list the files that have change, exclude all files from the ´´aws sync´´ command amd include only the ones that have changed.
We finally invalidate cache of directory on cloudfire to be able to see the changes if needed.

Last but not least, before pushing this file to the repo execute a

chmod -x pipe.sh

To have execution permission once in docker.

bitbucket-pipelines.yml

This file configures the upload of your pipe to dockerhub, just don't forget to have available in your repo
the docker hub credential variables.

The project with the static files

Now we configure the pipeline file for our project (bitbucket repo) to use the new pipe and pass the necessary variables.
My bitbucket-pipelines.yml file looks like this:

# This is a sample build configuration for Other.
# Check our guides at https://confluence.atlassian.com/x/5Q4SMw for more examples.
# Only use spaces to indent your .yml configuration.
# -----
# You can specify a custom docker image from Docker Hub as your build environment.
image: atlassian/default-image:2

pipelines:
  custom:
    deploy-statics-diff-last-tags:
      - step:
          script:
            - pipe: docker://gmoretti/aws-deploy-diff-last-tag-simple-pipe:latest
              variables:
                AWS_ACCESS_KEY_ID: '$AWS_ACCESS_KEY_ID'
                AWS_SECRET_ACCESS_KEY: '$AWS_SECRET_ACCESS_KEY'
                S3_BUCKET: 'aws-static-deploy-test'
                LOCAL_PATH: 'static'
                TAG_REGEX: 'TEST_*'

Let's go line by line, first thing we declare this as a custom step, so to use it we need to go to bitbucket branch and manually click on it, next line is the name of the step taht will appear in the list of things that can be executed.

custom:
    deploy-statics-diff-last-tags:

Next, the url to your pipe in dockerhub.

            - pipe: docker://gmoretti/aws-deploy-diff-last-tag-simple-pipe:latest

Now, we have our AWS credentials, you could easily hardcode them here, but they are better as repository variables, specially now that bitbucket lets you have different ones per environment.

            AWS_ACCESS_KEY_ID: '$AWS_ACCESS_KEY_ID'
            AWS_SECRET_ACCESS_KEY: '$AWS_SECRET_ACCESS_KEY'

Next, the target bucket without the protocol S3:// nor any slash at the end. You can add the target path to the very specific folder you want the files on.

                S3_BUCKET: 'aws-static-deploy-test'

Since we are in the root of the project we put the path to the directory we want to deploy. Again, no slashes at the start or end.

                LOCAL_PATH: 'static'

The last variable will help us match with the TAGS we want to compare, so they need to be somehow numerically secuential.

                TAG_REGEX: 'TEST_*'

This will match all tags like TEST_1, TEST_3, TEST_6, etc.

Execution

Now, we only need to push all the changes go to branches in bitbucket and execute the pipe.

Final thoughts

Alotought to solve this specific case, this pipe would work great, the thing is that there are several other cases that have not been covered, like, deploying files not covered by a tag or commiting files between several tags. Althought all these things can be done extending the pipe script, the truth is that just tagging and deploying all the static files will eliminate the whole problem and will leave as using the bitbucket AWS S3 mantained pipe.

That's what I would take home today.

References

If you have any thoughts feel free to contact me, thank you!