DEV Community: Zahidul Islam

24-hour coding interview prep challenge

Zahidul Islam — Sat, 04 Jan 2020 05:18:47 +0000

I am not a big believer in measuring a developer's skill based on 60 minutes of coding challenges. Most of the time those problems are designed in such a way that if you know the algorithm it will take 15 minutes to solve. I can’t remember a time when I solved a difficult real-life problem in such a short time. Every complex problem takes time to solve.

However, I have two coding interviews coming up this Monday, 6 January. So instead of complaining about the interview process, I will commit to spending 24 hours starting tomorrow morning January 4, 2020, at 8:00 AM till January 5 at 8:00 PM. During that time I will study data structures and algorithms.

I will update my progress periodically, publish notes and codes. Here are the topics I have in mind:

Array and String
Linked List
Stack and Queue
Trees
Graphs
Bit Manipulation
Recursion
Dynamic Programming
Searching and Sorting

I am planning to read Cracking the Coding Interview by Gayle Laakmann McDowell and Data Structures and Algorithms with JavaScript: Bringing Classic Computing Approaches to the Web by Michael McMillan.

I will read other blog posts and solve as many coding problems as I can and share them with the DEV community.

Please do share with me any good interview prep resources you know. I want this post and conversation to be a great place where anyone can refer to when they have very little time to prepare for the coding interviews.

Day 1 (January 4, 2020)

I started my 24-hour coding Interview prep challenge at 10:00 AM. I have a perfect cup of coffee and I am ready to crack the coding interview prep. Here are the topics I will go through today.

Arrays
String
Linked Lists
Stacks
Queues
Trees
Graphs
Search

I will study the topics, document all useful resources, solve 5/10 problems in each topic and update the community every 2 hours. I believe the community can be a great accountability partner.

Please share your best ideas on how master data structure quickly. I appreciate any suggestions.

The next update will be at 12:00 PM.

12: 00 PM Update

I read the first chapter of Cracking the Coding Interview book.
Here are the programming problems I worked on:

Arrays and Strings

1. Is Unique: Implement an algorithm to determine if a string has all unique characters. What if you cannot use additional data structures?

function uniqueCharacters(str) {
  const ASCII_CHARS = 256;

  if(str.length > ASCII_CHARS) {
    return false;
  }

  const chars = Array(ASCII_CHARS).fill(false);

  for(let char of str){
    const index = char.charCodeAt(0);
    if (chars[index] === true) {
      return false;
    }
    chars[index] = true;
  }

  return true;
}

console.log(uniqueCharacters('abcd10jk')); // true
console.log(uniqueCharacters('hutg9mnd!nk9')); // false

2. Check Permutation: Given two strings, write a method to decide if one is a permutation of the other.

function isPermutation(str1, str2) {
  if (str1.length !== str2.length) {
    return false;
  }

  const map = new Map();

  for(const char of str1) {
    if(map.has(char)) {
      map.set(char, map.get(char) + 1);
    } else {
      map.set(char, 1);
    }
  }

  for(const char of str2) {
      if(map.has(char)) {
        const value = map.get(char);
        if(value === 0) {
          return false;
        } else {
          map.set(char, map.get(char) - 1);
        }
      }
  }

  for(const value of map.values()) {
    if(value !== 0) {
      return false;
    }
  }
  return true;

}

console.log(isPermutation("god", "dog")); // true

3. URLify: Write a method to replace all spaces in a string with '%20'.

function replaceSpaces(str, trueLength) {
  let output = "";
  let chars = 0;

  for(let char of str) {
    if(char !== " ") {
      chars++;
    }
  }

  let spaces = trueLength - chars;

  for(let char of str) {
    if(char === " " && spaces > 0) {
      output += "%20";
      spaces--;
    } else if(char !== " ") {
      output += char;
    }
  }

  while(spaces > 0) {
    output += "%20";
    spaces--;
  }

  return output;
}

console.log(replaceSpaces("Mr 3ohn Smith", 13)); // "Mr%203ohn%20Smith"

3. Palindrome Permutation: Given a string, write a function to check if it is a permutation of a palindrome

function isPermutationOfPalindrome(str) {
  let charCount = 0;
  let map = new Map();

  for(let char of str) {
    if (char === " ") {
      continue;
    }

    if(map.has(char)) {
      map.delete(char);
    } else {
      map.set(char, true);
    }

    charCount++;
  }

  if(charCount % 2 === 0) {
    return map.size === 0;
  } else {
    return map.size === 1;
  }
}

console.log(
  isPermutationOfPalindrome("tacoa cat") === true,
  isPermutationOfPalindrome("atco cat") === true,
  isPermutationOfPalindrome(" rac ecar rara ") === true
);

4. String Compression: Implement a method to perform basic string compression using the counts of repeated characters.

function compress (str) {
  const map = new Map();
  let result = '';

  for(const char of str) {
    if(map.has(char)) {
      map.set(char, map.get(char) + 1);
    } else {
      map.set(char, 1);
    }
  }

  for(let [key, value] of map) {
    result += key + value;
  }

  return str.lenght >= result.lenght ? str : result;
}

console.log(compress('aabcccccaaa')); // "a5b1c5"

5. Rotate Matrix: Given an image represented by an NxN matrix, where each pixel in the image is 4 bytes, write a method to rotate the image by 90 degrees.

function rotateMatrix(arr) {
    let n = arr.length - 1;

    for (let i = 0; i < n; i++) {
        for (let j = 0; j < n - i; j++) {
            let temp = arr[i][j];

            arr[i][j] = arr[n - j][n - i]; // top row
            arr[n - j][n - i] = temp; // right column
        }
    }

    return arr;
}


console.log(rotateMatrix([
  [1,2,3],
  [4,5,6],
  [7,8,9]
])); 

/* 
[
    [9, 6, 3], 
    [8, 5, 2], 
    [7, 4, 1]
] 
*/

6. String Rotation; Assume you have a method isSubstrin g which checks if one word is a substring of another. Given two strings, s1 and s2, write code to check if s2 is a rotation of s1 using only one call to isSubstring [e.g. "waterbottle " is a rotation oP'erbottlewat")

const isSubstring = (str1, str2) => str1.includes(str2);

function isRotation(str1, str2) {
  if(!str1 || !str2) {
    return false;
  }

  if(str1.length !== str2.length) {
    return false;
  }

  return isSubstring(str1 + str2, str2);
}

console.log(isRotation("waterbottle", "erbottlewat")); // true
console.log(isRotation("aacdf", "acda")); // false

7. Find the first unique character in a given string or an array

function findFirstUniqueCharacter(str) {
  if(!str) return;

  const map = new Map();

  for(let char of str) {
      if(map.has(char)) {
        map.set(char, map.get(char) + 1);
      } else {
        map.set(char, 1);
      }
  }

  for(let [key, value] of map) {
    if(value === 1) {
      return key;
    }
  }
  return "None";
}

console.log(findFirstUniqueCharacter('foobar')); // f
console.log(findFirstUniqueCharacter('aabbccdef')); // d
console.log(findFirstUniqueCharacter('aabbcc')); // 'No Unique Character Found'

3: 30 PM Update

Linked Lists

Linked List `Node` class

class Node {
  constructor(value, next = null) {
    this.value = value;
    this.next = next;
  }
}

Linked list `print` function

function print(node) {
  while(node !== null) {
      console.log('->' + node.value);
      node = node.next;
  }
}

1. Remove Dups: Write code to remove duplicates from an unsorted linked list.

function removeDups(node) {
  if(node === null) return null;

  let nodes = new Map();
  let current = node.next;
  let prev = node;
  nodes.set(node.value, 1);

  while(current !== null) {
    if(nodes.get(current.value) === 1) {
      prev.next = current.next;
    } else {
      nodes.set(current.value, 1);
      prev = current;
    }
    current = current.next;
  }
  return node;
}

let node1 = new Node(1);
let node2 = new Node(2);
let node3 = new Node(3);
let node4 = new Node(2);
let node5 = new Node(1);

node1.next = node2;
node2.next = node3;
node3.next = node4;
node4.next = node5;
node5.next = null;

print(removeDups(node1));

/* 
"->1"
"->2"
"->3"
*/

2. Return Kth to Last: Implement an algorithm to find the kth to last element of a singly linked list.

function KthToLast(root, n) {
  if(!root) return null;
  let current = root;
  let follower = root;

  for(let i = 0; i < n; i++) {
    current = current.next;
  }

  while(current.next !== null) {
    current = current.next;
    follower = follower.next;
  }

  return follower;
}


let node1 = new Node(1);
let node2 = new Node(2);
let node3 = new Node(3);
let node4 = new Node(4);
let node5 = new Node(5);

node1.next = node2;
node2.next = node3;
node3.next = node4;
node4.next = node5;
node5.next = null;

console.log(KthToLast(node1, 2).value);  // 3

3. Delete middle of linked list

function removeDups(head) {
  if(head === null) return null;

  let fast = head;
  let slow = head;
  let prev = null;

  while(fast !== null && fast.next !== null) {
    fast = fast.next.next;
    prev = slow;
    slow = slow.next;
  }

  prev.next = slow.next;

  return head;
}

let nodeA = new Node('a');
let nodeB = new Node('b');
let nodeC = new Node('c');
let nodeD = new Node('d');
let nodeE = new Node('e');
let nodeF = new Node('f');

nodeA.next = nodeB;
nodeB.next = nodeC;
nodeC.next = nodeD;
nodeD.next = nodeE;
nodeE.next = nodeF;
nodeF.next = null;

print(removeDups(nodeA)); 
/*
"->a"
"->b"
"->c"
"->e"
"->f"
*/

4. Partitioning a linked list around a given value and If we don’t care about making the elements of the list `stable`

function partition(head, x) {
  let tail = head;
  let current = head;

  while(current !== null) {
    let next = current.next;
    if(current.value < x) {
      current.next = head;
      head = current;
    } else {
      tail.next = current;
      tail = current;
    }
    current = next;
  }
  tail.next = null;

  return head;
}

let nodeA = new Node(3);
let nodeB = new Node(5);
let nodeC = new Node(8);
let nodeD = new Node(2);
let nodeE = new Node(10);
let nodeF = new Node(2);
let nodeH = new Node(1);

nodeA.next = nodeB;
nodeB.next = nodeC;
nodeC.next = nodeD;
nodeD.next = nodeE;
nodeE.next = nodeF;
nodeF.next = nodeH;
nodeH.next = null;

print(partition(nodeA, 5));
/*
"->1"
"->2"
"->2"
"->3"
"->5"
"->8"
"->10"
*/

5. Given a linked list where each node has two pointers, one to the next node and one to a random node in the list, clone the linked list.

function clone(node) {
  if(node === null) return node;

  let map = new Map();
  let copy = new Node(node.value);
  let current = node;
  let newHead = copy;

  map.set(current, copy);

  current = current.next;
  while(current !== null) {
    let temp = new Node(current.value);
    map.set(current, temp);
    copy.next = temp;
    copy = temp;
    current = current.next;
  }

  current = node;
  copy = newHead;

  while(current !== null) {
    if(current.randome !== null) {
      copy.random = map.get(current.random);
    } else {
      copy.random = null;
    }

    current = current.next;
    copy = copy.next;
  }

  return newHead;
}

6. Given a linked list, determine whether it contains a cycle.

function hasCycle(node) {
  if(node === null) return false;
  let slow = node;
  let fast = node.next;

  while(fast !== null && fast.next !== null) {
    if(fast === slow) return true;
    slow = slow.next;
    fast = fast.next.next;
  }

  return false;
}

let node1 = new Node(1);
let node2 = new Node(2);
let node3 = new Node(3);
let node4 = new Node(4);
let node5 = new Node(5);

node1.next = node2;
node2.next = node3;
node3.next = node4;
node4.next = node5;
node5.next = node3;

console.log(hasCycle(node1)); // true

7. How to find the middle element of the linked list in one pass?

function findMiddle(node) {
    if(node === null) return null;
    let fast = node.next;
    let slow = node;

    while(fast !== null && fast.next !== null) {
        fast = fast.next.next;
        slow = slow.next;
    }
    console.log(slow.value);
}

let node1 = new Node(5);
let node2 = new Node(6);
let node3 = new Node(7);
let node4 = new Node(1);
let node5 = new Node(2);

node1.next = node2;
node2.next = node3;
node3.next = node4;
node4.next = node5;
node5.next = null;

findMiddle(node1); // 7

8. Add Two Numbers

// Input: (2 -> 4 -> 3) + (5 -> 6 -> 4)
// Output: 7 -> 0 -> 8
// Explanation: 342 + 465 = 807.

const addTwoNumbers = function(l1, l2) {
  let head = new Node(0);
  let carry = 0;
  let current = head;

  while(l1 !== null || l2 !== null ) {
    const x = l1 !== null ? l1.value : 0;
    const y = l2 !== null ? l2.value : 0;

    const sum = carry + x + y;
    carry = Math.floor(sum / 10);

    current.next = new Node(sum % 10);
    current = current.next;

    if(l1 != null) l1 = l1.next;
    if(l2 != null) l2 = l2.next;
  }

  if (carry > 0) {
    current.next = new ListNode(carry);
  }
  return head.next;
};

const node13 = new Node(3);
const node12 = new Node(4);
const node11 = new Node(2);
node11.next = node12;
node12.next = node13;
const l1 = node11; 

const node23 = new Node(4);
const node22 = new Node(6);
const node21 = new Node(5);
node22.next = node23;
node21.next = node22;
const l2 = node21;

print(addTwoNumbers(l1, l2));
/*
"->7"
"->0"
"->8"
*/

7:45 PM Update

Stacks

1. Given a stack sort the elements in the stack using no more than ane additional stack

Array.prototype.peek = function() {
  return this[this.length -1];
}

Array.prototype.isEmpty = function() {
  return this.length <= 0;
}

function sortStack(stack) {
  if(!stack || stack.length ===0) return stack;

  let newStack = [];
  newStack.push(stack.pop());

  while(!stack.isEmpty()) {
    const temp = stack.pop();
    while(!newStack.isEmpty() && temp > newStack.peek()) {
      stack.push(newStack.pop());
    }
    newStack.push(temp);
  }
  return newStack;
}

console.log(sortStack([5, 1, 3, 2, 4])); // [5, 4, 3, 2, 1]

2. Evaluate Reverse Polish Notation in Javascript

function reversePolishNotation(seq) {
  if(seq.length <= 2) {
    return;
  }

  const operands = ['+', '-', '*', '/'];
  const stack = [];
  let i = 0;

  stack.push(seq[i]);
  i++;

  while(i <= seq.length) {
    let item = seq[i];
    let index = operands.indexOf(item);

    if(index < 0) {
      stack.push(item);
    } else {
      const a = parseInt(stack.pop(), 10);
      const b = parseInt(stack.pop(), 10);

      if(index === 0) {
        stack.push(a + b);
      } 
      if(index === 1) {
        stack.push(a - b);
      }
      if(index === 2) {
        stack.push(a * b);
      }
      if(index === 3) {
        stack.push(b/a);
      }
    }
    i++;
  }

  return parseInt(stack[0], 0);
}

console.log(reversePolishNotation(["2", "1", "+", "3", "*"])) // 9
console.log(reversePolishNotation(["4", "13", "5", "/", "+"])) // 6
console.log(reversePolishNotation(["10", "6", "9", "3", "+", "-11", "*", "/", "*", "17", "+", "5", "+"])) // 22
console.log(reversePolishNotation(["2", "1"])) // undefined

3. Implement a stack using javascript

function Stack() {
  this.top = null;
}

Stack.prototype.push = function (val) {
  let node = {
    data : val,
    next : null
  };

  node.next = this.top;
  this.top = node;
};

Stack.prototype.pop = function () {
  if(this.top === null) {
    return null;
  } else {
    const top = this.top;
    this.top = this.top.next;
    return top.data;
  }
};

Stack.prototype.peek = function() {
  if(this.top === null) {
    return null;
  } else {
    return this.top.data;
  }
}

var S1 = new Stack();
S1.push(5);
S1.push(6);
S1.push(7);
S1.push(8);

console.log(S1.pop()); // 8
console.log(S1.pop()); // 7

4. Queue using two stacks javascript

function Queue() {
  this.pushStack = new Stack();
  this.popStack = new Stack();
}

Queue.prototype.enqueue = function(value) {
  this.pushStack.push(value);
}

Queue.prototype.dequeue = function() {
  let popStack = this.popStack;
  let pushStack = this.pushStack;

  if(popStack.peek()) {
    const deq = popStack.pop();
    return deq;
  }

  while(pushStack.peek()) {
    popStack.push(pushStack.pop());
  }
}

const q1 = new Queue();
q1.enqueue(3);
q1.enqueue(4);
q1.enqueue(5);
q1.enqueue(6);
q1.enqueue(7);
q1.dequeue();
q1.dequeue();
q1.dequeue();

console.log(q1);

5.

function stepsToSolveTowerOfHanoi(height, from, to, via) {
  if (height >= 1) {
    // Move a tower of height - 1 to buffer peg using destination peg
    stepsToSolveTowerOfHanoi(height - 1, from, via, to);
    // Move the remaining disk to the destination peg.
    console.log(`Move disk ${height} from Tower ${from} to Tower ${to}`);
    // Move the tower of `height-1` from the `buffer peg` to the `destination peg` using the `source peg`.        
    stepsToSolveTowerOfHanoi(height - 1, via, to, from);
  }

  return;
}

stepsToSolveTowerOfHanoi(3, "A", "C", "B");
/*
"Move disk 1 from Tower A to Tower C"
"Move disk 2 from Tower A to Tower B"
"Move disk 1 from Tower C to Tower B"
"Move disk 3 from Tower A to Tower C"
"Move disk 1 from Tower B to Tower A"
"Move disk 2 from Tower B to Tower C"
"Move disk 1 from Tower A to Tower C"
*/

6. Justify if a string consists of valid parentheses

function isMatchingBrackets(str) {
  if(str.length <= 1) {
    return false;
  }

  let stack = [];
  const map = {
    '{': '}',
    '[': ']',
    '(': ')'
  };

  for(let char of str) {
    if(char === '(' || char === '[' || char === '{') {
      stack.push(char);
    } else {
      const last = stack.pop();
      if (char !== map[last]) {
        return false;
      }
    }
  }
  return stack.length === 0;
}

console.log(isMatchingBrackets("(){}")); // true
console.log(isMatchingBrackets("[{()()}({[]})]({}[({})])((((((()[])){}))[]{{{({({({{{{{{}}}}}})})})}}}))[][][]")); // true
console.log(isMatchingBrackets("({(()))}}"));  // false

Apache Airflow Core Concepts

Zahidul Islam — Fri, 03 Jan 2020 04:10:21 +0000

Apache Airflow is a tool for describing, executing and monitoring workflows.

Here are some core concepts you need to know to become productive in Airflow:

DAG (Directed Acyclic Graph)

In Airflow DAG is a collection of all the tasks you want to run, organized in a way that reflects their relationships and dependencies.

Scope

Airflow will load any DAG object it can import from a DAG file. That means the DAG must appear in globals().

Default Arguments

Default arguments are passed to a DAG as default_args dictionary. This makes it easy to apply a common parameter to many operators without having to type it many times.

Context Manager

DAGs can be used as context managers to automatically assign new operators to that DAG.

Operators

While DAGs describes how to run a workflow, Operators determine what actually gets done.

DAG Assignment

Operators do not have to be assigned to DAGs immediately (previously dag was a required argument). However, once an operator is assigned to a DAG, it can not be transferred or unassigned. DAG assignment can be done explicitly when the operator is created, through deferred assignment, or even inferred from other operators.

Bitshift Composition

Airflow official document recommends that you should setup operator relationships with bitshift operators rather than set_upstream() and set_downstream()

Relationship Helper

chain and cross_downstream function provide easier ways to set relationships between operators in a specific situation.

Tasks

A task is a parameterized instance of an operator

Task Instances

A task that 1) has been assigned to a DAG and 2) has a state associated with a specific run of the DAG

Hooks

Hooks are interfaces to external platforms and databases like Hive, S3, MySQL, Postgres, HDFS, and Pig. Hooks implement a common interface when possible, and act as a building block for operators.

Pools

Some systems can get overwhelmed when too many processes hit them at the same time. Airflow pools can be used to limit the execution of parallelism on arbitrary sets of tasks. The list of pools is managed in the UI (Menu -> Admin -> Pools) by giving the pools a name and assigning it a number of worker slots.

Connections

The connection information to external systems is stored in the Airflow metadata database and managed in the UI (Menu -> Admin -> Connections).

Airflow also has the ability to reference connections via environment variables from the operating system. Then connection parameters must be saved in URI format.

If connections with the same conn_id are defined in both Airflow metadata database and environment variables, only the one in environment variables will be referenced by Airflow.

Queues

When using the CeleryExecutor, the Celery queues that tasks are sent to can be specified.

The default queue for the environment is defined in the airflow.cfg’s celery -> default_queue. This defines the queue that tasks get assigned to when not specified, as well as which queue Airflow workers listen to when started.

Workers can listen to one or multiple queues of tasks. When a worker is started (using the command airflow worker), a set of comma-delimited queue names can be specified (e.g. airflow worker -q spark). This worker will then only pick up tasks wired to the specified queue(s).

XComs

XComs let tasks exchange messages, allowing more nuanced forms of control and shared state. The name is an abbreviation of “cross-communication”. XComs are principally defined by a key, value, and timestamp, but also track attributes like the task/DAG that created the XCom and when it should become visible. Any object that can be pickled can be used as an XCom value, so users should make sure to use objects of appropriate size.

Variables

Variables are a generic way to store and retrieve arbitrary content or settings as a simple key-value store within Airflow. Variables can be listed, created, updated and deleted from the UI (Admin -> Variables), code or CLI.

Branching

Sometimes you need a workflow to branch, or only go down a certain path based on an arbitrary condition which is typically related to something that happened in an upstream task. One way to do this is by using BranchPythonOperator.

SubDAGs

SubDAGs are perfect for repeating patterns. Defining a function that returns a DAG object is a nice design pattern when using Airflow.

This SubDAG can then be referenced in your main DAG file:

Plugins

Airflow has a simple plugin manager built-in that can integrate external features to its core by simply dropping files in your $AIRFLOW_HOME/plugins folder.

References

https://airflow.apache.org/concepts.html

Deep dive: 3 ways to invoke AWS Lambda Function

Zahidul Islam — Thu, 02 Jan 2020 04:38:52 +0000

AWS Lambda functions are everywhere. We are using it for various use cases. Sometimes we want to invoke Lambda functions over HTTP and other times trigger a function based on some external events. It is extremely useful to understand the different ways we can invoke Lambda functions. Here are 3 different ways we can invoke AWS Lambda function:

Synchronous Invokes

This is the simplest lambda invocation model. When we perform a Lambda invoke an API call, we wait for the function to process the event and return a response or times out. The response includes the function response and additional data.

To perform synchronous invocation we can use the CLI or any of the supported SDKs.

Example of synchronous invoke using the CLI:



aws lambda invoke —function-name MyLambdaFunction —invocation-type RequestResponse —payload  '{ "key": "value" }'

Example of synchronous invoke using the Node.js SDK:



const AWS = require("aws-sdk");

const lambda = new AWS.Lambda();

const params = {
  FunctionName: "MyLambdaFunction",
  InvocationType: "RequestResponse",
  Payload: JSON.stringify({ key: "value" })
};

await lambda.invoke(params).promise();

When we specify the Invocation-type flag or InvocationType parameter value as RequestResponse it instructs AWS to execute our function and wait for the function to complete. One of the drawbacks of synchronous invoke is we are responsible for checking the response and determining whether it is an error and we should retry the invoke.

There are many AWS services that can trigger lambda function synchronously. Here are some of them:

ELB (Application Load Balancer)
Cognito
Lex
Alexa
API Gateway
CloudFront (Lambda@Edge)
Kinesis Data Firehose

Asynchronous Invokes

When we invoke a function asynchronously, Lambda sends the event to a queue and returns a success response without additional information. A separate process reads events from the queue and runs our function. To invoke a function asynchronously, we have to set the invocation type parameter to Event.

Example of an asynchronous invokes using the CLI:



aws lambda invoke —function-name MyLambdaFunction —invocation-type Event —payload  '{ "key": "value" }'

Example of an asynchronous invokes using the Node.js SDK:



const AWS = require("aws-sdk");

const lambda = new AWS.Lambda();

const params = {
  FunctionName: "MyLambdaFunction",
  InvokeArgs: JSON.stringify({
    Records: [
      {
        eventVersion: "2.0",
        eventSource: "aws:s3",
        awsRegion: "us-west-2",
        eventTime: "1970-01-01T00:00:00.000Z",
        eventName: "ObjectCreated:Put",
        userIdentity: {
          principalId: "AIDAJDPLRKLG7UEXAMPLE"
        },
        requestParameters: {
          sourceIPAddress: "127.0.0.1"
        },
        responseElements: {
          "x-amz-request-id": "C3D13FE58DE4C810",
          "x-amz-id-2":
            "FMyUVURIY8/IgAtTv8xRjskZQpcIZ9KG4V5Wp6S7S/JRWeUWerMUE5JgHvANOjpD"
        },
        s3: {
          s3SchemaVersion: "1.0",
          configurationId: "testConfigRule",
          bucket: {
            name: "sourcebucket",
            ownerIdentity: {
              principalId: "A3NL1KOZZKExample"
            },
            arn: "arn:aws:s3:::sourcebucket"
          },
          object: {
            key: "HappyFace.jpg",
            size: 1024,
            eTag: "d41d8cd98f00b204e9800998ecf8427e",
            versionId: "096fKKXTRTtl3on89fVO.nfljtsv6qko"
          }
        }
      }
    ]
  })
};

await lambda.invokeAsync(params).promise();

You can use the above code snippet to write integration tests for your lambda function that gets trigger based on the s3 event.

When we specify the Invocation-type flag or InvocationType parameter value as Event if the function failed AWS will automatically retry the invoke twice.

Here is a list of services that invoke Lambda functions asynchronously:

S3
SNS
SES
CloudFormation
CloudWatch Logs
CloudWatch Events
CodeCommit

Pool Based Invokes

Pool-Based invocation model allows us to integrate with AWS Stream and Queue based services. Lambda will poll records from services (Kinesis, SQS, DynamoDB Streams) and invoke our functions. AWS Lambda Event Source Mapping manages the poller and performs Synchronous invokes of our function.

Here is an example mapping of a function (MyLambdaFunction) to a Kinesis stream (arn:aws:kinesis:us-east-2:123456789012:stream/lambda-stream) with a batch size of 500:



$ aws lambda create-event-source-mapping --function-name MyLambdaFunction \
--batch-size 500 --starting-position AT_TIMESTAMP --starting-position-timestamp 1541139109 \
--event-source-arn arn:aws:kinesis:us-east-2:123456789012:stream/lambda-stream
{
    "UUID": "2b733gdc-8ac3-cdf5-af3a-1827b3b11284",
    "BatchSize": 500,
    "MaximumBatchingWindowInSeconds": 0,
    "ParallelizationFactor": 1,
    "EventSourceArn": "arn:aws:kinesis:us-east-2:123456789012:stream/lambda-stream",
    "FunctionArn": "arn:aws:lambda:us-east-2:123456789012:function:MyLambdaFunction",
    "LastModified": 1541139209.351,
    "LastProcessingResult": "No records processed",
    "State": "Creating",
    "StateTransitionReason": "User action",
    "DestinationConfig": {},
    "MaximumRecordAgeInSeconds": 604800,
    "BisectBatchOnFunctionError": false,
    "MaximumRetryAttempts": 10000
}

Here is an example lambda function that receives a Kinesis event input and processes the messages:



exports.handler = async event => {

    event.Records.forEach(record => {

        // Kinesis data is base64 encoded so decode here

        const payload = Buffer.from(record.kinesis.data, 'base64').toString('ascii');

        console.log('Decoded payload:', payload);

    });

};

Retry behavior of different invocation model

Invocation Model	Retry Behavior
Synchronous	None
Asynchronous	Twice
Pool Based	Retry based on data expiration

Now you know 3 different ways to invoke Lambda functions. If you have questions or suggestions please feel free to let me know in the comments.