DEV Community: Hassan Murtaza

Ingest Millions of Messages in the Queue using Serverless

Hassan Murtaza — Sun, 06 Dec 2020 20:54:38 +0000

While working on one of the Serverless projects, We came across a problem where I needed to ingest millions of messages from an external API to Azure Storage Queue. We made a series of changes in my approach to reach the solution which worked for us and I believe it can be helpful to anyone who’s struggling with the same problem.

We started with the approach where we were executing the Serverless function through the HTTP endpoint. We call it an Ingestion Function. The ingestion function would then hit an external API to get the messages and try to ingest it in Queue.

Here is how my initial architecture looked like:

If you see the architecture diagram, the single ingestion function is a bottleneck as it can only scale up-to specific limits. Also, we’re not utilizing the strength of the actual serverless function, which is multiple parallel small invocations. Therefore, we decided to go with the approach where I could scale up the ingestion serverless to multiple invocations so that I can get the scalability as much as needed.

The idea was to divide the total number of messages into multiple chunks (of 5000 in my case) and then pass those messages to the ingestion function so that it can finally ingest those messages to the Queue.

We created another serverless function, we call it Chunking Function, to divide the messages into chunks using this helper function:

def chunking_of_list(data):
"""return the list in 5000 chunks"""
  return [data[x:x+5000] for x in range(0, len(data), 5000)]

And then uploaded each chunk in a separate file into Azure Blob Storage using this code:

def upload_chunks(idLists):
  st = storage.Storage(container_name=constant.CHUNK_CONTAINER)

  for index, singleList in enumerate(idLists):
     logging.info('Uploading file{0} to the {1} blob storage'  .format(index, constant.CHUNK_CONTAINER))
     st.upload_chunk('file' + str(index), singleList)

Finally, we set up the ingestion function to listen to the Azure Blob Storage file upload events. As soon as the file gets uploaded, the ingestion function will download the file, read it, and ingest the messages into Queue. As desired we now have multiple invocations of ingestion functions to work in parallel therefore, we achieved scalability.

Here’s how the final architecture looked like:

We essentially followed a fan-out architecture, where we fan out our workload to multiple Serverless invocations instead of one.

Peace ✌

Overcoming AWS Lambda Limitations with Fargate

Hassan Murtaza — Mon, 08 Jun 2020 18:14:51 +0000

Using Lambda with Fargate helps us to overcome the limitations of AWS Lambda. If you need to run a background process for more than 15 minutes or if you need more storage than 500 MBs, here's how you can do it:

➡️ Lambda function will get triggered based on your desired event source.

➡️ It will ingest the event in the SQS queue, and trigger Fargate to consume the event from the queue.

➡️ Since we'll be doing all the processing on Fargate, we'll overcome some of the hard limits of AWS Lambda.

Example
If you need to downlaod the file up-to 1 GBs of size, do processing and upload it to S3. AWS Fargate will be your go-to choice since it's practically not possible with AWS Lambda's hard limits.

If you have technical questions and suggestions, let's discuss them here, in Github or say hi me in LinkedIn.

Deploy Azure Functions with Jenkins

Hassan Murtaza — Mon, 01 Jun 2020 18:07:08 +0000

There are multiple ways to continuously deploy the latest code to Azure Functions.

Since at OpenAnalytics we mostly use Jenkins for CI/CD purposes, my preference was to stick to Jenkins. I tried multiple options and the one which worked perfectly on Azure Functions with Linux platform was to do zip deploy using restful API.

The idea is to first create the zip out of source code and then deploy it:

curl -X POST https://<user>:<password>@<function-app>.scm.azurewebsites.net/api/zipdeploy -T <zipfile>

Here's how you can do it:

Go to your function app on the azure portal then open properties from the sidebar. You should find the Deployment Trigger URL there and that's what you need. it should look something like this:

https://$<your-functionapp>:<your-functionapp-password><your-functionapp>@.scm.azurewebsites.net/deploy

Go to Jenkins > Credentials > Global Credentials and create global credentials.

Note: we need to escape the dollar($) sign that's why using backlash.

Finally, this is how your Jenkinsfile should look like:

pipeline {

        agent any

        options {
                buildDiscarder(logRotator(numToKeepStr: '3'))
        }

        environment {
            USER_CREDENTIALS = credentials('<your_azure_function_credentials>')
            FUNCTION_URL     = "<function_name>.scm.azurewebsites.net/api/zipdeploy"
            ARTIFACT         = "functionapp.zip"
        }

        stages {

            stage('Build') {
                steps {
                        sh "zip -r $ARTIFACT *"
                        script {  
                            if (fileExists(env.ARTIFACT)) {
                                sh "echo Artifact: $ARTIFACT created successfuflly"
                            } else {
                                error('Failed to create artifact: ' + env.ARTIFACT)
                            }
                        }
                }
            }

            stage('Deploy') {
                steps {
                    script {
                        int status = sh(script: "curl -sLI -w '%{http_code}' -X POST https://$USER_CREDENTIALS_USR:$USER_CREDENTIALS_PSW@$FUNCTION_URL -T $ARTIFACT  -o /dev/null", returnStdout: true)

                        if (status != 200 && status != 201) {
                            error("Returned status code = $status when calling $FUNCTION_URL")
                        }
                    }
                }
            }
        }

        post {
            success {
                sh 'echo Function deployed successfully.'
            }
        }
}

Note: zip deploy by default assumes that your code is ready to deploy. To enable build process (like installing packages etc.) while deployment. Add this to application settings:
SCM_DO_BUILD_DURING_DEPLOYMENT=true

Thank you for you time!

If you have technical questions and suggestions, let's discuss them here, in Github or say hi me in LinkedIn.

Wrapper over AWS Session Manager to SSH using Public/Private IP

Hassan Murtaza — Tue, 17 Mar 2020 11:50:06 +0000

Previously on AWS, we had two problems with establishing SSH sessions to EC2 instances.

It's hard to manage SSH keys, especially if when we have a lot of servers.
It was hard to open sessions to private instances as you need the bastion host to do so.

AWS Session Manager solved both of these issues for us. Moreover, it is an AWS-centric solution so we preferred it.

Moreover, we needed a way to establish an SSH connection to instances using public IP or private IP as we were doing ssh using IPs before. AWS CLI doesn't support it out of the box (it supports only instance-id). I wrote this wrapper script over AWS CLI to achieve the same.

Repository link: https://github.com/MrHassanMurtaza/ec2-ssh/