DEV Community: Jay Parekh

History of Devops

Jay Parekh — Wed, 22 Jun 2022 20:31:42 +0000

Why this post ?

This article is about how term Devops came to place, who were the pioneers or the maker of whole movement of Devops. Which are key events that helped companies through Devops.
During my engineering days I always use to wonder how the techies came up with such cool tech ideas and its origin/history, out of which many of turned into giants today.
All companies are now moving towards Agile-Devops. So this post is dedicated to pioneers of Devops. (Learned this history during IBM introductory course)
Note: No tools mentioned.

So it begins : Once upon a time ..

Development and Operations two sides of any software development process use to communicate only via tickets in separate teams.

2007 : Patrick Debois got frustrated with Development and Operations - and thought how can they collab.

2008: In Agile Conference 2008, Andrew Clay Shaefer was speaker of a session “Birds of a feather”, which was attended by a single member - Patrick Debois. Where Andrew and Patrick discussed about the Dev and Ops problem and he showed this famous Slide “Wall of confusion” - Dev needs change and Ops needs stability and they communicate with each other through tickets.

2009: Starting of Devops Days in Ghent for the first time

2009: All were surprised by John Allspaw and Paul Hammond reported that they could do 10 deployments in a day (flickr which was bought by Yahoo eventually).

2013: Jez Humble and David Farley wrote a book called Continuous Delivery . Also I highly recommend and admire the Youtube session by Jez Humble - Continous Delivery

2013: Gene Kim, Kevin Behr and George Spafford worked together to write a novel about IT, Devops and Business in conversational and anecdotes style

2016: Gene Kim ( From The Phoenix Project), Jez Humble (Thoughtworks - Continuous Delivery), Patrick Debois (Coined Devops) and John Willis collaborated and created a Handbook for Devops.

AWS SES, API Gateway - POST and Serverless

Jay Parekh — Sun, 01 May 2022 22:12:09 +0000

Pre-requisites

Require to setup Serverless on workstation and get it configured with AWS as cloud provider and AWS access key - programmatic user

Simple Email Service (SES) setup on AWS

Go to SES → Create identity sender email : blah.aws@gmail.com and receiver email : blah-blah@gmail.com
Verify the email

Simple Email Service (SES) and Serverless

Create a Node.js starter project on serverless workstation
Update the handler.js file which has a createContact lambda function

"use strict";
const AWS = require("aws-sdk");
const ses = new AWS.SES();

module.exports.createContact = async (event, context) => {
  console.log("Received:::", event);
  const { to, from, subject, message } = JSON.parse(event.body);

  if (!to || !from || !subject || !message) {
    return {
      headers: {
        "Content-Type": "application/json",
        "Access-Control-Allow-Methods": "*",
        "Access-Control-Allow-Origin": "*",
      },
      statusCode: 400,
      body: JSON.stringify({ message: " to or from... are not set properly!" }),
    };
  }
  const params = {
    Destination: {
      ToAddresses: [to],
    },
    Message: {
      Body: {
        Text: { Data: message },
      },
      Subject: { Data: subject },
    },
    Source: from,
  };
  try {
    await ses.sendEmail(params).promise();
    return {
      headers: {
        "Content-Type": "application/json",
        "Access-Control-Allow-Methods": "*",
        "Access-Control-Allow-Origin": "*",
      },
      statusCode: 200,
      body: JSON.stringify({
        message: "email sent successfully!",
        success: true,
      }),
    };
  } catch (error) {
    console.error(error);
    return {
      headers: {
        "Content-Type": "application/json",
        "Access-Control-Allow-Methods": "*",
        "Access-Control-Allow-Origin": "*",
      },
      statusCode: 400,
      body: JSON.stringify({
        message: "The email failed to send",
        success: true,
      }),
    };
  }
};

Update the serverless.yml file.
- region: ap-south-1 Mumbai
- functions: createContact
- events: -http
- path: /mail
- method: POST

service: aws-ses-project

frameworkVersion: '3'

provider:
  name: aws
  runtime: nodejs14.x
  region : ap-south-1

functions:
  createContact:
    handler: handler.createContact
    events:
      - http:
          path: /mail
          method: POST

Deploy the changes using serverless deploy which will create
- Lambda function - createContact

createContact: aws-ses-project-dev-createContact (1.7 kB)

Improve API performance – monitor it with the Serverless Dashboard: run "serverless"

API Gateway - /mail POST method - Trigger to invoke createContact - API Gateway POST endpoint: *https://oadnufagsd.execute-api.ap-south-1.amazonaws.com/dev/mail
*
- Testing the API and Lambda function using POSTMAN - POST method
- Checking the POST method url: https://oadnufagsd.execute-api.ap-south-1.amazonaws.com/dev/mail
- body :

{ "to": "<-sender-email->@gmail.com", 
"from": "<-receiver-email->@gmail.com", 
"subject": "test mail", 
"message": "send successfully" }

NOTE The response of the POST method says that API is successfully set and triggered the event but "the email is failed to send"
To diagnose and debug the error go to API POST -> Logs method(Observability Tool - Metrics/Logs/Traces) and choose *Open in CloudWatch * in Logs.
The error message is as follows :

2022-04-25T18:54:54.759Z    1623581b-c472-4002-bc06-519766d1f0be    ERROR   AccessDenied: User `arn:aws:sts::673040272970:assumed-role/aws-ses-project-dev-ap-south-1-lambdaRole/aws-ses-project-dev-createContact' is not authorized to perform `ses:SendEmail' on resource `arn:aws:ses:ap-south-1:673040272970:identity/<-sender-email->@gmail.com'
    at Request.extractError (/var/runtime/node_modules/aws-sdk/lib/protocol/query.js:50:29)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
    at Request.emit (/var/runtime/node_modules/aws-sdk/lib/request.js:686:14)
    at Request.transition (/var/runtime/node_modules/aws-sdk/lib/request.js:22:10)
    at AcceptorStateMachine.runTo (/var/runtime/node_modules/aws-sdk/lib/state_machine.js:14:12)
    at /var/runtime/node_modules/aws-sdk/lib/state_machine.js:26:10
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:38:9)
    at Request.<anonymous> (/var/runtime/node_modules/aws-sdk/lib/request.js:688:12)
    at Request.callListeners (/var/runtime/node_modules/aws-sdk/lib/sequential_executor.js:116:18) {
  code: 'AccessDenied',
  time: 2022-04-25T18:54:54.756Z,
  requestId: 'd2111db9-a75d-4c37-bbee-e4d4738268ea',
  statusCode: 403,
  retryable: false,
  retryDelay: 22.98167156490245
}

What caused the error ? : Simple Email Service (SES) and Lambda are two different AWS managed services which does not have access to each other by default.
Solution: Setup an iam role with policy in a way that lambda can access ses functionalities

Create an IAM Role for Lambda to access SES

Go to IAM → Role → AWS Service
Use case : enable Lambda (radio button or select from drop down list).
Setting up permission - AWSSESFullAccess
Set the name of the role : Lambda-SES-Full-Access

NOTE: Message field says : email sent successfully
Confirming on gmail

Creating a frontend to send email using Next.js

Go to Setup Next.js app in workstation. (To know more Next.js)
Create a Next.js application in aws-ses-project folder (same as above)

npx create-next-app nextjs-blog --use-npm --example "https://github.com/vercel/next-learn/tree/master/basics/learn-starter

cd nextjs-blog and run build npm run dev which will deploy a Welcome Next.js page accessible on http://localhost:3000.
Folder structure of nextjs-blog
Modifying the index.js by creating mail form replacing the welcome index page

import Head from "next/head";

export default function Home() {
  const sendMessage = async (event) => {
    event.preventDefault();

    /*
      "to": "youremail@gmail.com",
    "from": "email@gmail.com",
    "subject": "this needs to work!",
    "message": "Hello there,!@"

    */

    const res = await fetch(
 # Replace this by your api method gateway
      "https://3t66ov5uxg.execute-api.us-east-1.amazonaws.com/dev/contact-us",
      {
        body: JSON.stringify({
          to: event.target.to.value,
          from: event.target.from.value,
          subject: event.target.subject.value,
          message: event.target.message.value,
        }),
        headers: {
          "Content-Type": "application/json",
        },
        method: "POST",
      }
    );

    const result = await res.json();
    console.log(result);
  };
  return (
    <div className='container'>
      <Head>
        <title>Contact us</title>
        <link rel='icon' href='/favicon.ico' />
      </Head>

      <main>
        <h1 className='title'>Contact us</h1>

        <div className='grid'>
          <form onSubmit={sendMessage}>
            <label htmlFor='to'>To: </label>
            <input id='to' name='to' type='text' required />

            <label htmlFor='from'>From: </label>
            <input
              id='from'
              name='from'
              type='text'
              autoComplete='from'
              required
            />

            <label htmlFor='subject'>Subject: </label>
            <input id='subject' name='subject' type='text' required />

            <label htmlFor='message'>Message: </label>
            <input id='message' name='message' type='text' required />
            <button type='submit'>Send</button>
          </form>
        </div>
      </main>

      <footer>Powered by Awesomness</footer>

      <style jsx>{`
        /* Style inputs */
        input[type="text"],
        select {
          width: 100%;
          padding: 12px 20px;
          margin: 8px 0;
          display: inline-block;
          border: 1px solid #ccc;
          border-radius: 4px;
          box-sizing: border-box;
        }

        /* Style the submit button */
        input[type="submit"] {
          width: 100%;
          background-color: #04aa6d;
          color: white;
          padding: 14px 20px;
          margin: 8px 0;
          border: none;
          border-radius: 4px;
          cursor: pointer;
        }

        /* Add a background color to the submit button on mouse-over */
        input[type="submit"]:hover {
          background-color: #45a049;
        }
        .container {
          min-height: 100vh;
          padding: 0 0.5rem;
          display: flex;
          flex-direction: column;
          justify-content: center;
          align-items: center;
        }

        main {
          padding: 5rem 0;
          flex: 1;
          display: flex;
          flex-direction: column;
          justify-content: center;
          align-items: center;
        }

        footer {
          width: 100%;
          height: 100px;
          border-top: 1px solid #eaeaea;
          display: flex;
          justify-content: center;
          align-items: center;
        }

        footer img {
          margin-left: 0.5rem;
        }

        footer a {
          display: flex;
          justify-content: center;
          align-items: center;
        }

        a {
          color: inherit;
          text-decoration: none;
        }

        .title a {
          color: #0070f3;
          text-decoration: none;
        }

        .title a:hover,
        .title a:focus,
        .title a:active {
          text-decoration: underline;
        }

        .title {
          margin: 0;
          line-height: 1.15;
          font-size: 4rem;
        }

        .title,
        .description {
          text-align: center;
        }

        .description {
          line-height: 1.5;
          font-size: 1.5rem;
        }

        code {
          background: #fafafa;
          border-radius: 5px;
          padding: 0.75rem;
          font-size: 1.1rem;
          font-family: Menlo, Monaco, Lucida Console, Liberation Mono,
            DejaVu Sans Mono, Bitstream Vera Sans Mono, Courier New, monospace;
        }

        .grid {
          display: flex;
          align-items: center;
          justify-content: center;
          flex-wrap: wrap;

          max-width: 800px;
          margin-top: 3rem;
        }

        .card {
          margin: 1rem;
          flex-basis: 45%;
          padding: 1.5rem;
          text-align: left;
          color: inherit;
          text-decoration: none;
          border: 1px solid #eaeaea;
          border-radius: 10px;
          transition: color 0.15s ease, border-color 0.15s ease;
        }

        .card:hover,
        .card:focus,
        .card:active {
          color: #0070f3;
          border-color: #0070f3;
        }

        .card h3 {
          margin: 0 0 1rem 0;
          font-size: 1.5rem;
        }

        .card p {
          margin: 0;
          font-size: 1.25rem;
          line-height: 1.5;
        }

        .logo {
          height: 1em;
        }

        @media (max-width: 600px) {
          .grid {
            width: 100%;
            flex-direction: column;
          }
        }
      `}</style>

      <style jsx global>{`
        html,
        body {
          padding: 0;
          margin: 0;
          font-family: -apple-system, BlinkMacSystemFont, Segoe UI, Roboto,
            Oxygen, Ubuntu, Cantarell, Fira Sans, Droid Sans, Helvetica Neue,
            sans-serif;
        }

        * {
          box-sizing: border-box;
        }
      `}</style>
    </div>
  );
}

Replace the POST method URL in index.js
Cross-Origin Resource Sharing (CORS) Error while sending the POST request. Go to inspect element → console to know the error.

Access to fetch at 'https://oadnufagsd.execute-api.ap-south-1.amazonaws.com/dev' from origin 'http://localhost:3000' has been blocked by CORS policy: Response to preflight request doesn't pass access control check: No 'Access-Control-Allow-Origin' header is present on the requested resource. If an opaque response serves your needs, set the request's mode to 'no-cors' to fetch the resource with CORS disabled.

What caused the error ?: localhost which a different IP-address is trying to fetch https://oadnufagsd.execute-api.ap-south-1.amazonaws.com/dev which is at a different IP address. By default due to security concerns CORS is disabled which prevents information to share with other IPs or resources.
Solution: CORS policy has to enabled in order to share resource or in other words access a different IP address in our case.
- Method-1: Through AWS Console
- Enabling CORS via AWS Console by going to API Gateway → aws-dev-ses-project → /mail/post → Actions → Select Enable CORS from dropdown
- Method-2: Serverless
  - Through serverless.yml add cors: true under API method

functions:
  createContact:
    handler: handler.createContact
    events:
      - http:
          path: /mail
          method: POST
          cors: true

Deploy the changes sls deploy and run cd nextjs-blog && npm run dev

Confirming through gmail

Note: I dont know Node.js was helped by online session

Getting Started with Serverless and AWS

Jay Parekh — Sun, 01 May 2022 17:07:26 +0000

This is my first ever blog. I hope you find it valuable.

Setting up the Workstation

Install Node.js - Download Node.js in my case - Windows Installer .msi 64-bit on workstation
Install Serverless Download Serverless Getting Started on workstation - npm install -g serverless
Root user or root equivalent user access in AWS

Create an IAM Role - AWS Service

Access key will enable serverless to connect with AWS

Go to IAM -> Create users -> Add a user name: jayp1_serverless -> Select AWS access type -> Select AWS credential type -> Access key - Programmatic access

Description: > Access Key - Programmatic Access - Enables an access key ID and secret access key for the AWS API, CLI, SDK, and other development tools.

Setup serverless configuration to connect with AWS serverless config credentials --provider aws --key <access-key> --secret <secret-key>
Create a directory: mkdir aws_2022 && cd aws_2022 and initialize Serverless project serverless
Choose AWS-Python-Starter
Name it default
Deploy - Yes

C:\Users\jp\Documents\aws_2022>serverless

Creating a new serverless project

? What do you want to make? AWS - Python - Starter
? What do you want to call this project? aws-python-project

✔ Project successfully created in aws-python-project folder

? Do you want to login/register to Serverless Dashboard? No

? Do you want to deploy now? (Y/n) y
? Do you want to deploy now? Yes

Deploying aws-python-project to stage dev (us-east-1)

× Stack aws-python-project-dev failed to deploy (78s)
Environment: win32, node 16.14.2, framework 3.12.0, plugin 6.2.1, SDK 4.3.2
Docs:        docs.serverless.com
Support:     forum.serverless.com
Bugs:        github.com/serverless/serverless/issues

Error:
CREATE_FAILED: IamRoleLambdaExecution (AWS::IAM::Role)
API: iam:GetRole User: arn:aws:iam::673040272970:user/jayp1_serverless is not authorized to perform: iam:GetRole on resource: role aws-python-project-dev-us-east-1-lambdaRole because no identity-based policy allows the iam:GetRole action

View the full error: https://us-east-1.console.aws.amazon.com/cloudformation/home?region=us-east-1#/stack/detail?stackId=arn%3Aaws%3Acloudformation%3Aus-east-1%3A673040272970%3Astack%2Faws-python-project-dev%2F6669ede0-c3c3-11ec-9584-0a8914e80a47

Note the IamRole error , to fix is to update the role permissions by giving the access key user - jayp1_serverless.
Go to IAM -> Users -> jayp1_serverless -> Add Permissions -> Choose Attach existing policy directly

Creating a new serverless project

? What do you want to make? AWS - Python - Starter
? What do you want to call this project? aws-python-project-2022

✔ Project successfully created in aws-python-project-2022 folder

? Do you want to login/register to Serverless Dashboard? No

? Do you want to deploy now? (Y/n) Y
? Do you want to deploy now? Yes

Deploying aws-python-project-2022 to stage dev (us-east-1)

✔ Service deployed to stack aws-python-project-2022-dev (111s)

functions:
  hello: aws-python-project-2022-dev-hello (1.7 kB)

What next?
Run these commands in the project directory:

serverless deploy    Deploy changes
serverless info      View deployed endpoints and resources
serverless invoke    Invoke deployed functions
serverless --help    Discover more commands

Go to Lambda -> Functions -> aws-python-project-2022-dev-hello (filename-stage-function) - try to check in default region - N.Virginia

Invoke lambda function on local workstation serverless invoke --function hello

C:\Users\jp\Documents\aws_2022>cd aws-python-project-2022

C:\Users\jp\Documents\aws_2022\aws-python-project-2022>serverless invoke --function hello
{
    "statusCode": 200,
    "body": "{\"message\": \"Go Serverless v3.0! Your function executed successfully!\", \"input\": {}}"
}

Digging deep into folder stack of serverless project
There are two concerned files for developer - handler.py and serverless.yml
- handler.py contains a function hello which returns a json message.

import json


def hello(event, context):
    body = {
        "message": "Go Serverless v3.0! Your function executed successfully!",
        "input": event,
    }

    return {"statusCode": 200, "body": json.dumps(body)}

serverless.yml is generated or created automatically while we create a serverless project. To understand serverless YML keywords Serverless Framework - AWS Lambda Guide

service: aws-python-project-2022

frameworkVersion: '3'

provider:
  name: aws
  runtime: python3.8

functions:
  hello:
    handler: handler.hello

service - project configuration file
provider : which cloud provider in our case its AWS
- name : aws
- runtime : python3.8

Deploying a new lambda function in desired region

Create a new lambda function getting_started in new handler file application.py which will be deployed in Mumbai region: ap-south-1

Create a function - getting started in application.py

import json


def getting_started(event, context):
    body = {
        "message": "Go Serverless v3.0! Getting Started with Serverless \
             execution!",
        "event": event,
    }

    return {"statusCode": 200, "body": json.dumps(body)}

Modify serverless.yml

service: aws-python-project-2022

frameworkVersion: '3'

provider:
  name: aws
  runtime: python3.8
  region: ap-south-1

functions:
  hello:
    handler: handler.hello
  getting_started:
    handler: application.getting_started

Deploy changes on AWS serverless deploy

Invoke getting_started on workstation serverless invoke --function getting_started

C:\Users\jp\Documents\aws_2022\aws-python-project-2022>serverless invoke --function getting_started
{
    "statusCode": 200,
    "body": "{\"message\": \"Go Serverless v3.0! Getting Started with Serverless              execution!\", \"event\": {}}"
}

API Gateway and AWS Lambda through Serverless Framework

Event - Http - GET* Method will invoke getting_started function
Setting up the APIs in API Gateway events under function and listing one such event -http with path: /data and method: GET

functions:
  hello:
    handler: handler.hello
  getting_started:
    handler: application.getting_started
    events:
      - http:
          path: /data
          method: GET

Deploy the changes on AWS serverless deploy notice the endpoint


C:\Users\jp\Documents\aws_2022\aws-python-project-2022>serverless deploy

Deploying aws-python-project-2022 to stage dev (ap-south-1)

✔ Service deployed to stack aws-python-project-2022-dev (48s)

endpoint: GET - https://ni5xsk9f08.execute-api.ap-south-1.amazonaws.com/dev/data
functions:
  hello: aws-python-project-2022-dev-hello (2 kB)
  getting_started: aws-python-project-2022-dev-getting_started (2 kB)

Improve API performance – monitor it with the Serverless Dashboard: run "serverless"

Checking the endpoint: https://ni5xsk9f08.execute-api.ap-south-1.amazonaws.com/dev/data

Go to API Gateway -> Select Mumbai region -> APIs

Check the invocation counts, error logs etc. in Monitor in Lambda console.