DEV Community: Mithun Kamath

How did they do it | Control the lights in a room using your hand

Mithun Kamath — Mon, 13 Dec 2021 06:55:17 +0000

A month ago, I came across this tweet from @devdevcharlie where they use hand gestures to control the lights in their room. Check it out:

// Detect dark theme var iframe = document.getElementById('tweet-1460476498775724033-857'); if (document.body.className.includes('dark-theme')) { iframe.src = "https://platform.twitter.com/embed/Tweet.html?id=1460476498775724033&theme=dark" }

Wasn't that cooooool?

So - how did they manage to do that? D-uh! It's in their tweet itself!! Great work Sherlock!

They used tensorflow.js
They specifically made use of the pose detection model named Movenet

Ok ok. But how did they actually pull it off? What could their code look like? Here's my take on how they may have achieved it.

The Smart Bulb

Let's get this out of the way sooner than later. I can't make out much details of the smart bulb / light in play but for this task, I shall abstract it. Instead of obsessing over which device they may have used, since it is not central to this task and it only needs to turn on or off based on hand gestures, let us assume that it is a Light model that has a method state to which you pass either ON or OFF. So, something like this:

// To turn the device on
Light.state("ON")

// To turn the device off
Light.state("OFF")

It could be any smart bulb, but at its very basic, its interface would probably have the above methods that we will make use of. We don't have to worry about the intricacies any further. In fact, for our implementation, we'll be logging the detected hand gesture to the browser console and the resulting light state.

The setup

There's a laptop (with a camera) in front of them - and that's the one that's capturing their pose, not the camera that's recorded the scene that we can see (hey - my dumb brain did not see the laptop initially). So you would need a camera / web cam that you can stream yourself through. If you don't have a webcam, but you possess an Android phone (and a USB cable), check out DroidCam that let's you convert your phone to a webcam.

The code

index.html

We start off by creating a very basic HTML page. Code with explanations follow:

// index.html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>How did they do it? | @devdevcharlie edition</title>
</head>
<body>
  <video id="pose-off"></video>
  <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@3.11.0/dist/tf.min.js"></script>
  <script src="https://cdn.jsdelivr.net/npm/@tensorflow-models/pose-detection@0.0.6/dist/pose-detection.min.js"></script>
  <script src="/script.js"></script>
</body>
</html>

Here, we are creating an index.html file
In this file, we import the tensorflow.js library (@tensorflow/tfjs). We also import the Pose Detection library built on top of tensorflow.js (@tensorflow-models/pose-detection). This requires the tensorflow.js script and hence it is defined after tensorflow.js has loaded.
We have also included our own script.js file, which is where we shall write our script
Lastly, note the presence of the <video> tag. It has an id of #pose-off. It is in this tag that we shall stream our video (and from which we shall analyse the hand gestures)

So far, so good.

We move on to the script.js implementation, which is where we shall have all our logic to control the lights.

script.js

In this file, we start off by defining couple of functions, each of which do a dedicated task.

initVideo()

This function initializes the video tag, so that it plays the video from the camera attached to our computer. It goes something like this:

// script.js

async function initVideo() {
  // Step 1
  const video = document.querySelector("#pose-off");

  // Step 2
  video.width = 640;
  video.height = 480;

  // Step 3
  const mediaStream = await window.navigator.mediaDevices.getUserMedia({
    video: {
      width: 640,
      height: 480,
    },
  });

  // Step 4
  video.srcObject = mediaStream;

  // Step 5
  await new Promise((resolve) => {
    video.onloadedmetadata = () => {
      resolve();
    };
  });

  // Step 6
  video.play();

  // Step 7
  return video;
}

Each code statement has a step associated with it and the explanation of each step is below:

We start off by selecting the video tag in the HTML defined earlier. We are querying by the id of the tag (#pose-off).
We proceed to then set the width and height of the video. In our example, we go with a dimension of 640x480 but you can chose one to your liking. But remember - the value that you set is important. We shall see why further below.
At this step, we are asking the user for permission to access their video stream. The browser should auto detect the camera set up and provide us access to it. We are using the most basic of configuration, where we are setting the video resolution to 640x480 - same as the dimension we set for the video tag in Step 2 above.
Once we get permission to access the video stream, we set that as the source for our video HTML tag.
We then wait until the video metadata loads
Once the video metadata loads, we begin to "play" the video. In our case, since our video source is the camera device, we should start seeing the video feed.
Finally, we return the video object that we have initialised.

initPoseDetector()

This function sets up our "Pose" Detector. Pose here is our body pose / posture. Check out this diagram obtained from the Movenet documentation.

Each number represents part of our body (eye - left/right, wrist - left/right etc). In the referenced link, you can find the identification of each number below the image itself. Reproducing it here for your convenience:

0: nose
1: left_eye
2: right_eye
3: left_ear
4: right_ear
5: left_shoulder
6: right_shoulder
7: left_elbow
8: right_elbow
9: left_wrist
10: right_wrist
11: left_hip
12: right_hip
13: left_knee
14: right_knee
15: left_ankle
16: right_ankle

Isn't that cool? We already have the means to identify the different parts of our body. We just need to make use of it. This is how:

// script.js

async function initPoseDetector() {
  // Step 1
  const model = window.poseDetection.SupportedModels.MoveNet;

  // Step 2
  detector = await window.poseDetection.createDetector(model, {
    modelType: window.poseDetection.movenet.modelType.SINGLEPOSE_THUNDER,
  });

  // Step 3
  return detector;
}

Here's the explanation for each step in the code above:

Tensorflowjs supports multiple "models" for pose detection. Think of models as libraries - there are three prominent ones - MoveNet, BlazePose and PoseNet. We are making use of the MoveNet model. This step is basically configuring the model that we will make use of.
At this step, we are actually initialising our pose detector. We are passing in the model that we'd like to use (MoveNet) and we are further passing the configuration for the MoveNet model. In this case, we are specifying that we'd like to use the SINGLEPOSE_THUNDER variant of the MoveNet model. There are two other variants that we could have chosen but we chose this one because although it is slower, it is more accurate. Also we only intend to detect a single person's pose. (There are models to detect poses of multiple people at the same time).

Think of MoveNet as the Brand of car that you'd like to go with - Tesla. After selecting the Brand, you now need to select which (car) variant you'd like to go with - Tesla Model S, which in our case is the SINGLEPOSE_THUNDER variant of the MoveNet model.
Lastly, we return the pose detector object that we have initialised.

analyzeHandGesture()

Alright. So far we have implemented a function that initialises the webcam based video feed and another function that initialises the MoveNet tensorflow.js model. We now move on to another function that will use the MoveNet model to determine the hand gesture carried out in the video feed. Since this function works on the video feed and makes use of the MoveNet model, we would need to pass as input the video feed and the MoveNet model detector:

// script.js

async function analyzeHandGesture(video, detector) {
  // Step 1
  const poses = await detector.estimatePoses(video, { flipHorizontal: true });

  // Step 2
  recognizeGesture(poses[0].keypoints.find((p) => p.name === "left_wrist"));

  // Step 3
  requestAnimationFrame(async () => {
    await analyzeHandGesture(video, detector);
  });
}

A couple of things are happening in this step. We begin by calling the MoveNet model's estimatePoses() function. To this function we are passing the video feed. Further I have defined a configuration flipHorizontal to flip the video feed, you guessed it, horizontally because the video input from my ghetto camera feed (Recollect that I am using an Android Phone as a webcam) is mirrored. To correct it, I need to flip the feed horizontally.
This function returns the poses identified in the video feed. The structure of the data is an array of objects, where each object has the following structure:
```
  {
    x: // x co-ordinate
    y: // y co-ordinate
    score: // confidence score - how confident
           // the model is about the detected
           // body part
    name: // name of the body part.
          // Ex. right_eye, left_wrist
  }
```
Correction - this is the data structure of one pose. The MoveNet model is capable of detecting multiple humans in a video and for each person, it creates an object that has an attribute of keypoints which is itself again an array of objects. The above is data structure of this keypoint object.
In this step, we are trying to locate the keypoint for the left_wrist body part. Why just the left wrist? We'll find out in a second. After we extract that specific keypoint, we pass it to the recognizeGesture() function. This function identifies the hand gesture and decides the action to carry out based on it. We are yet to define this function - we will do so in the next step.
Lastly, we use requestAnimationFrame() to call the analyzeHandGesture() function again - we essentially end up creating an infinite loop where the analyzeHandGesture() function is called repeatedly thereby analyzing our hand movement forever.

recognizeGesture()

This function receives a keypoint object, with the x and y co-ordinates of a body part and it is expected to recognize the gesture made through that body part.

Bear in mind that detecting a complex movement like a thumbs up or a finger pointing in a direction or a "call me" finger combination requires setting up a neural network to accurately determine the hand pose. That is too cumbersome for our project here. We would like to keep it simple.

In the demonstration by @devdevcharlie we see her lifting her right hand up to turn on the lamp on the right side. And consequently bringing her right hand down to turn it off. Ditto with her left hand movements to control the lamp on the left side of her television.

For our replication, we'll recognize a really simply hand gesture - if our left wrist is on the LEFT SIDE of the video, we'll turn the lights ON. If our left wrist is RIGHT SIDE of the video, we'll turn the lights OFF. We will be dealing with just a single light source, unlike the demonstration where there are two light sources.

So essentially, we are dividing our video area into two parts - since our video width is 640px (see the initVideo() function), this would mean that from 0px to 320px will be our LEFT side of the video while 321px to 640px shall be the RIGHT side of our video.

But hang on - our video feed is flipped. Which would mean that 321px to 640px is our LEFT side while 0px to 320px is our RIGHT side.

Let's translate that to code our recognizeGesture() function:

// script.js

async function recognizeGesture(keypoint) {
  let status;

  if (keypoint.x > 320) {
    status = "ON";
  } else {
    status = "OFF";
  }

  console.log("Light is turned:", status);
}

If the x co-ordinate is greater than 320px, our wrist is on the LEFT side of the video and thus, we turn ON our light. Otherwise we turn it OFF.

That was the penultimate function we implemented.

start()

This is the last function we'll implement. This brings it all together:

// script.js

async function start() {
  const video = await initVideo();

  const detector = await initPoseDetector();

  await analyzeHandGesture(video, detector);
}

// Don't forget to call the function
start();

We initialize the video and store the video object, we then initialize the MoveNet model and store the detector and lastly, we analyze the hand gesture seen in the video.

The full source code for the script.js file looks like:

// script.js

async function recognizeGesture(keypoint) {
  let status;

  if (keypoint.x > 320) {
    status = "ON";
  } else {
    status = "OFF";
  }

  console.log("Light is turned:", status);
}

async function initVideo() {
  const video = document.querySelector("#pose-off");

  video.width = 640;
  video.height = 480;

  const mediaStream = await window.navigator.mediaDevices.getUserMedia({
    video: {
      width: 640,
      height: 480,
    },
  });

  video.srcObject = mediaStream;

  await new Promise((resolve) => {
    video.onloadedmetadata = () => {
      resolve();
    };
  });

  video.play();

  return video;
}

async function initPoseDetector() {
  const model = window.poseDetection.SupportedModels.MoveNet;

  detector = await window.poseDetection.createDetector(model, {
    modelType: window.poseDetection.movenet.modelType.SINGLEPOSE_THUNDER,
  });

  return detector;
}

async function analyzeHandGesture(video, detector) {
  const poses = await detector.estimatePoses(video, { flipHorizontal: true });
  recognizeGesture(poses[0].keypoints.find((p) => p.name === "left_wrist"));

  requestAnimationFrame(async () => {
    await analyzeHandGesture(video, detector);
  });
}

async function start() {
  const video = await initVideo();

  const detector = await initPoseDetector();

  await analyzeHandGesture(video, detector);
}

start();

Finale

When we launch our app using a simple http server, our demonstration will look something like:

Remember - our version detects the left wrist's location - and not the entire arm's movements.

Be(a)ware - AWS Amplify does not have migrations

Mithun Kamath — Mon, 15 Nov 2021 15:16:24 +0000

This post is about the tool / framework AWS Amplify - provided by Amazon itself to work with most of the popular AWS services - when working on serverless web applications.

A word of caution if you are planning to use this tool for your new project - While this tool has many problems, one issue that I face recurrently is that this does not have "migrations" in it - yeah like the "migrations" on a database. Let's understand what that means in the context of AWS Amplify.

Firstly, know that amplify has support for teams. One can create new "environments" specific to their use case and work exclusively in their environment before they decide to merge their changes into their team's "main" environment - much like how git works.

However, this has its limitations - and a very big one at that too in my opinion. It does not keep track of the steps that a user took in their environment individually. It only takes care of the final result.

I had a lambda function that was being used in an AWS Appsync pipeline. As the needs of the project changed, I determined that this function's name no longer matches the task that it carries out. I wanted to rename the function. This requires one to delete the original function and create a new one with the new name - as there is no command that amplify provides to rename a function. Fair enough. However, this isn't the main problem. You see - since the function was used in an appsync pipeline (a custom resolver at that too - and not just a @function directive) - I had to:

first remove the reference to the old function name from the pipeline
remove the old function
create the new function with the new name
add the new named function back to the pipeline

or so I thought. Turns out, I had to not just carry the above out, I had to carry them out individually - wherein I had to:

carry out one task
commit the changes
move the changes into the main environment
and repeat the above steps again for the next task.

See the problem here - I could not carry out all four tasks and then commit the changes - and have them merged with the main environment. I had to repeat the commit and the move-to-the-main-environment tasks for each individual task I had!

Why so? It turns out amplify would do things out of order (or perhaps it would do them in parallel instead of in series). It would remove the old function first, then update the pipeline with the new function name - and would then throw an error because the new function had not yet been created!! Had it tracked the order in which the individual steps were performed and then proceeded to adhere to this order when updating the main environment, we would not have to go through the effort of repeating the tasks again and again for each individual step.

With databases, when one modifies it, they would create migrations to keep track of the order in which the modifications are introduced. When a new instance of the database is created or has to be updated, it follows the migration order precisely to avoid the issue we currently face above.

But this "migration" feature is missing in Amplify. As a consequence, a task that could take a couple of minutes when using the AWS console or CLI now could take couple of hours, as you wait on your team to merge the changes of each individual task before you can proceed with the next task - only because amplify's design is flawed and does not really cater to "teams" as it would like you to believe.

Debug "Resource not accessible by integration" error when working with github's graphql endpoint

Mithun Kamath — Sun, 14 Nov 2021 11:12:41 +0000

When working with Github's graphql explorer, you will find that all your queries work seamlessly - but when you try to run the same queries in your application, you could run into permission issues.

While the explorer requests a very wide range of permissions, your application would probably be restrictive in the type of permissions it requests from your users to execute the graphql queries against their account.

In such a scenario, debugging which permission is needed could be difficult - here's an example query and the error thrown for that query to github's graphql endpoint:

The query:

{
  user(login: "callmekatootie") {
    repositories(first: 10) {
      nodes {
        collaborators(first: 10) {
          totalCount
        }
      }
    }
  }
}

and the error:

{
  "type": "FORBIDDEN",
  "path": [
      "user",
      "repositories",
      "nodes",
      0,
      "collaborators",
      "totalCount"
  ],
  "extensions": {
      "saml_failure": false
  },
  "locations": [
      {
          "line": 4,
          "column": 9
      }
  ],
  "message": "Resource not accessible by integration"
}

One gets the error Resource not accessible by integration. When one google's this error, there aren't quality results that can help you out.

This error basically talks about a permission issue. When working with Github apps, you may not have set the necessary permission to access the resource you need. Hence the Resource not accessible by integration error.

So - how would one go about finding out which permission is needed in this case?

The graphql documentation does not talk about this - weirdly, one first needs to instead go to their REST api documentation. In our case, since we are interested in the count of the collaborators on a repository, we would look for the REST api that fetches the collaborators for a repository. That would be the list repository collaborators endpoint:

GET /repos/{owner}/{repo}/collaborators

Now that we know the endpoint path, we then head over to the permissions required for Github Apps page and proceed to locate this endpoint (Do a Ctrl / Cmd + F in your browser and search for the endpoint you need).

We find the above endpoint under the Collaborators permission list:

which, if you scroll up, falls under the Metadata permissions. Boom! There you go! Head over to your Github App and under Repository permissions, provide read-only access for the Metadata permissions:

That should resolve your "Resource not accessible by integration" issue.

Amplify Quirks

Mithun Kamath — Thu, 11 Nov 2021 07:17:32 +0000

As I work on a project using aws amplify, I am going to document some quirks that I come across (or some plain old stupidity on my part). The documentation for amplify has room to improve but it is definitely frustrating to work with it. I document my findings here for my future self, if I have to work using amplify again.

I dislike that I have to go to Amplify's discord and search through the history for any similar issues... What happened to having StackOverflow as the go-to for such activities?

Anyway, I intend to update this blog forever so you could keep checking back.

Appsync / Graphql

Combination of `private` and `groups` authorization types

Example:

type Query {
  myFooQuery: Bar @auth(
    rules: [
      { allow: private },
      { allow: groups, groups: ["Admin"] }
    ]
  )
}

Expectation When I define both private and groups authorization, say on a query, users with a valid jwt token as well as users that belong to the cognito user group defined in the authorization rule are allowed to execute that query. I expected this to be an OR condition:

EITHER users that have a valid jwt token
OR users that belong to a specific cognito user group

just like the other authorization type combinations (such as a user can either use an api key or the owner can update their own record).

Actual The group authorization rule takes precedence over the private authorization rule. Thus, even if a user has a valid jwt token, they cannot execute the query. Not unless they belong to the group specified.

Comment Granted that having both private and groups authorization types is redundant - if a user belongs to a group, it would imply they have a jwt token and thus having just private authorization type would suffice. But I expected the combination to always be OR'ed but turns out that isn't the case when it comes to the private and groups authorization types.

Hours Spent Debugging 4 hours. Had to check the generated request mapping template, learn VTL, learn how to log in VTL ($util.error()!!!) and then came to this conclusion.

AWS AppSync with Amplify: How to ensure a field in a type is unique

Mithun Kamath — Sun, 07 Nov 2021 10:20:48 +0000

I will cut to the chase - you are probably here from a Google search tearing your hairs out trying to figure how to ensure that a field in your graphql schema is unique, and you are using AWS AppSync in conjunction with AWS Amplify. Amplify's documentation leaves too much to be desired. Here's how to do it:

Things to know before we proceed

Amplify provides a CustomResources.json file under the stacks folder that you will make use of. In this file, we specify the CloudFormation templates that Amplify needs to create - that will help us out with our goal.

We'll be creating:

A global secondary index in the dynamodb table associated with your schema
A lambda function that checks for the uniqueness of the field in the schema
An IAM role that allows Appsync to invoke this lambda function
A lambda data source associated with the previous lambda function
An Appsync Function that invokes the lambda function, with its request and response mapping templates
Our Mutation Appsync Function, with its request and response mapping templates.
Lastly, our pipeline that brings them all together.

This is the graphql schema we'll be using in our example:

type User @model {
  id: ID!
  name: String
  email: String
  username: String
}

and we'll be attempting to ensure that the username field is unique.

Word of caution: This post will be a lengthy one and so will be the steps you need to carry out to do something as simple as ensuring uniqueness. But the steps are very easy to follow.

Step 1: Create a Global Secondary Index (GSI)

This GSI will be created on the dynamodb table associated with our User type. This step is straightforward. We'll use the @key directive that amplify provides, which will create the GSI for us

// Graphql schema

type User
  @model
  @key(
    fields: ["username"],
    name: "UsernameIndex",
    queryField: "getUserByUsername"
) {
  id: ID!
  name: String
  email: String
  username: String
}

This is creating an index named UsernameIndex in the dynamodb table associated with the User model. There's only 1 field in this index - the username. That was it. Run amplify push to create this resource in AWS.

Step 2: Lambda function that checks for uniqueness of the field's value

This one too is straightforward. Using amplify add function, create your lambda function which will query the table associated with the User schema in dynamodb, along with the index you created in Step 1. In my example, I have named this function as uniqueusernamecheck You can write this in any of the languages that lambda functions support. Here's a snippet of how one could write it in Nodejs:

// Lambda function - uniqueusernamecheck

exports.handler = async (event) => {
  const { input } = event.arguments

  // Adding this IF condition only to ensure that we run the check if the username was being set
  if (input.username) {
    const params = {
      TableName: process.env.USERTABLE_NAME,
      IndexName: 'UsernameIndex',
      KeyConditionExpression: 'username = :username',
      ExpressionAttributeValues: {
        ':username': input.username
      }
    }

    // We queried the database associated with the `User` model directly, but feel free to use the `getUserByUsername` query too
    const record = await docClient.query(params).promise()

    if (record.Count > 0 && record.Items[0].id !== event.identity.sub) {
      throw new Error(`${input.username} has been taken. Try another username`)
    }
  }

  // All is well
  return {}
};

Feel free to change the values in the params object to suit your needs.

What we are doing in this function is checking if a user with that username exists, and if yes, we throw an error. Else, we are returning (an empty object).

As before, run amplify push to create this function in AWS.

Step 3 - An IAM role that allows AppSync to invoke the earlier created Lambda function

CustomResources.json comes into play now. We'll be using this to create the IAM role. The Resources key on the object in this file is where we'll be declaring all our resources starting with the IAM role. This is what our IAM role definition will look like:

// CustomResources.json

"UniqueUsernameCheckLambdaDataSourceRole": {
  "Type": "AWS::IAM::Role",
  "Properties": {
    "RoleName": {
      "Fn::Sub": [
        "UniqueUsernameCheckLambdaDataSourceRole-${env}",
        { "env": { "Ref": "env" } }
      ]
    },
    "AssumeRolePolicyDocument": {
      "Version": "2012-10-17",
      "Statement": [
        {
          "Effect": "Allow",
          "Principal": {
            "Service": "appsync.amazonaws.com"
          },
          "Action": "sts:AssumeRole"
        }
      ]
    },
    "Policies": [
      {
        "PolicyName": "InvokeLambdaFunction",
        "PolicyDocument": {
          "Version": "2012-10-17",
          "Statement": [
            {
              "Effect": "Allow",
              "Action": [
                "lambda:invokeFunction"
              ],
              "Resource": [
                {
                  "Fn::Sub": [
                    "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:uniqueusernamecheck-${env}",
                    { "env": { "Ref": "env" } }
                  ]
                }
              ]
            }
          ]
        }
      }
    ]
  }
},

Feel free to change the names - remember that in our example, our function was named uniqueusernamecheck and that's what we have used.

At this point, we don't really have to, but it would be best to run amplify push to create this IAM role. We can also wait until we complete Step 7 and run it once to create all the resources we need.

Step 4 - Create a Lambda data source

We have created our index, the lambda function that does the uniqueness check for us and the IAM role for Appsync to invoke this function. Moving along, we shall now create a data source for the lambda function in Appsync (in the same CustomResources.json file - you can add this below the one you created in Step 3:

// CustomResources.json

"UniqueUsernameCheckLambdaDataSource": {
  "Type": "AWS::AppSync::DataSource",
  "Properties": {
    "ApiId": {
      "Ref": "AppSyncApiId"
    },
    "Name": "UniqueUsernameCheckLambdaDataSource",
    "Type": "AWS_LAMBDA",
    "ServiceRoleArn": {
      "Fn::GetAtt": [
        "UniqueUsernameCheckLambdaDataSourceRole",
        "Arn"
      ]
    },
    "LambdaConfig": {
      "LambdaFunctionArn": {
        "Fn::Sub": [
          "arn:aws:lambda:${AWS::Region}:${AWS::AccountId}:function:uniqueusernamecheck-${env}",
          { "env": { "Ref": "env" } }
        ]
      }
    }
  },
  "DependsOn": "UniqueUsernameCheckLambdaDataSourceRole"
},

Note the DependsOn attribute at the end - which makes sure that the IAM role to call the lambda function is created BEFORE the data source gets created. We are also ensuring that we call the function we created in Step 2 - with the environment suffix (since amplify adds that automatically if you are working in multiple environments)

Hang in there - we are halfway through. Most of this is just copy and paste, with the values substituted to suit your scenario.

Step 5 - Create an Appsync Function

Having created the Lambda data source in the previous step, we now create an Appsync function - the function basically invokes the lambda function we created earlier as well as the request mapping template (called BEFORE our lambda function executes) and response mapping template (called AFTER our lambda function executes)

Step 5A - Create the request and response mapping template(s)

The CustomResources.json file we have been editing is in the stacks folder. Look for a folder named pipelineFunctions, which should be a sibling of the stacks folder. In this folder we'll define the request and response mapping templates:

This is the request mapping template - create a file named InvokeUniqueUsernameCheckLambdaDataSource.req.vtl under the pipelineFunctions folder:

## pipelineFunctions/InvokeUniqueUsernameCheckLambdaDataSource.req.vtl

## [Start] Invoke AWS Lambda data source: UniqueUsernameCheckLambdaDataSource. **
{
  "version": "2018-05-29",
  "operation": "Invoke",
  "payload": {
      "typeName": "$ctx.stash.get("typeName")",
      "fieldName": "$ctx.stash.get("fieldName")",
      "arguments": $util.toJson($ctx.arguments),
      "identity": $util.toJson($ctx.identity),
      "source": $util.toJson($ctx.source),
      "request": $util.toJson($ctx.request),
      "prev": $util.toJson($ctx.prev)
  }
}
## [End] Invoke AWS Lambda data source: UniqueUsernameCheckLambdaDataSource. **

This passes on all the info (and more) that the lambda function needs to perform the uniqueness check.

And this below is the response mapping template - create a file named InvokeUniqueUsernameCheckLambdaDataSource.res.vtl under the pipelineFunctions folder:

## pipelineFunctions/InvokeUniqueUsernameCheckLambdaDataSource.res.vtl

## [Start] Handle error or return result. **
#if( $ctx.error )
  $util.error($ctx.error.message, $ctx.error.type)
#end
$util.toJson($ctx.result)
## [End] Handle error or return result. **

...and this returns the error or results of the lambda function execution. In our scenario, an error occurs if the username has been taken. There is no "result" returned from the lambda function - if there's no error, we simply move on to the next resolver in our pipeline (further below).

Step 5B - Actually define the Appsync function

Having defined the request and response mapping templates, we now come back to the CustomResources.json file and define the appsync function that uses it:

// CustomResources.json

"InvokeUniqueUsernameCheckLambdaDataSource": {
  "Type": "AWS::AppSync::FunctionConfiguration",
  "Properties": {
    "ApiId": {
      "Ref": "AppSyncApiId"
    },
    "Name": "InvokeUniqueUsernameCheckLambdaDataSource",
    "DataSourceName": "UniqueUsernameCheckLambdaDataSource",
    "FunctionVersion": "2018-05-29",
    "RequestMappingTemplateS3Location": {
      "Fn::Sub": [
        "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/pipelineFunctions/${ResolverFileName}",
        {
          "S3DeploymentBucket": {
            "Ref": "S3DeploymentBucket"
          },
          "S3DeploymentRootKey": {
            "Ref": "S3DeploymentRootKey"
          },
          "ResolverFileName": {
            "Fn::Join": [".", ["InvokeUniqueUsernameCheckLambdaDataSource", "req", "vtl"]]
          }
        }
      ]
    },
    "ResponseMappingTemplateS3Location": {
      "Fn::Sub": [
        "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/pipelineFunctions/${ResolverFileName}",
        {
          "S3DeploymentBucket": {
            "Ref": "S3DeploymentBucket"
          },
          "S3DeploymentRootKey": {
            "Ref": "S3DeploymentRootKey"
          },
          "ResolverFileName": {
            "Fn::Join": [".", ["InvokeUniqueUsernameCheckLambdaDataSource", "res", "vtl"]]
          }
        }
      ]
    }
  },
  "DependsOn": "UniqueUsernameCheckLambdaDataSource"
},

This one depends on the data source (Step 4) having been created earlier. You can also see that it refers to the request and response mapping templates we created.

Step 6 - Create a Mutation Appsync Function

In Step 5, we create an appsync function that invokes the lambda function, along with the request and response mapping templates.

These request and response mapping templates are specific to the invoked lambda function only.

We now create another appsync function, which actually updates our user (ONLY IF THE UNIQUENESS CHECK PASSES).

Step 6A - First define the request and response mapping templates

You could define these yourselves, or you could have amplify generated these for you. The quickest way to do this is to execute amplify mock api - and while the mock api server is running, look into the resolvers folder to find a bunch of auto generated files. Since we are dealing with mutations and my mutation is named updateUser, I look for Mutation.updateUser.req.vtl and Mutation.updateUser.res.vtl files. On finding them, I simply open them and make a very minor edit - like adding a space or a newline at the end of the file. Terminate the mock api server - all the generated files disappear except the ones you edited. Voila! You have an auto generated mutation resolver - the best part about this is that it takes into consideration any @auth directives you have defined too.

With the request and response mapping templates now ready, create a file named MutationUpdateUserFunction.req.vtl and MutationUpdateUserFunction.res.vtl in the pipelineFunctions folder. Copy the contents of the auto generated files into these:

Copy the contents of Mutation.updateUser.req.vtl into MutationUpdateUserFunction.req.vtl
Copy the contents of Mutation.updateUser.res.vtl into MutationUpdateUserFunction.res.vtl
DO NOT erase the files under resolvers yet - just clear the contents. We'll be updating these further below.

Step 6B - Now define the appsync function

Come back to the CustomResources.json file and proceed to define the appsync function:

// CustomResources.json

"MutationUpdateUserFunction": {
  "Type": "AWS::AppSync::FunctionConfiguration",
  "Properties": {
    "ApiId": {
      "Ref": "AppSyncApiId"
    },
    "Name": "MutationUpdateUserFunction",
    "DataSourceName": "UserTable",
    "FunctionVersion": "2018-05-29",
    "RequestMappingTemplateS3Location": {
      "Fn::Sub": [
        "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/pipelineFunctions/${ResolverFileName}",
        {
          "S3DeploymentBucket": {
            "Ref": "S3DeploymentBucket"
          },
          "S3DeploymentRootKey": {
            "Ref": "S3DeploymentRootKey"
          },
          "ResolverFileName": {
            "Fn::Join": [".", ["MutationUpdateUserFunction", "req", "vtl"]]
          }
        }
      ]
    },
    "ResponseMappingTemplateS3Location": {
      "Fn::Sub": [
        "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/pipelineFunctions/${ResolverFileName}",
        {
          "S3DeploymentBucket": {
            "Ref": "S3DeploymentBucket"
          },
          "S3DeploymentRootKey": {
            "Ref": "S3DeploymentRootKey"
          },
          "ResolverFileName": {
            "Fn::Join": [".", ["MutationUpdateUserFunction", "res", "vtl"]]
          }
        }
      ]
    }
  }
},

Note that it refers to the request and response mapping

Step 7 - FINALE - Create the appsync pipeline

Loki - You must be truly desperate to come to me for help - if you have made it till this step - thank you for sticking with it.

We now bring them all together - a pipeline that will first invoke our uniqueness check and then actually carry out the mutation of the check passes.

We have to first create - you guessed it - a request and response mapping template. These are invoked when the pipeline starts and when the pipeline ends.

In the previous step, we have the Mutation.updateUser.req.vtl and Mutation.updateUser.res.vtl files under the resolvers folder. We are going to update the contents of these files (remember - you had copied the contents of these files to their equivalent in the pipelineFunctions folder).

## resolvers/Mutation.updateUser.req.vtl

## [Start] Stash resolver specific context.. **
$util.qr($ctx.stash.put("typeName", "Mutation"))
$util.qr($ctx.stash.put("fieldName", "updateUser"))
{}
## [End] Stash resolver specific context.. **

## resolvers/Mutation.updateUser.res.vtl

$util.toJson($ctx.prev.result)

With that out of the way, we now define our pipeline in the CustomResources.json file:

// CustomResources.json

"MutationUpdateUserResolver": {
  "Type": "AWS::AppSync::Resolver",
  "Properties": {
    "ApiId": {
      "Ref": "AppSyncApiId"
    },
    "TypeName": "Mutation",
    "FieldName": "updateUser",
    "Kind": "PIPELINE",
    "PipelineConfig": {
      "Functions": [
        {
          "Fn::GetAtt": ["InvokeUniqueUsernameCheckLambdaDataSource", "FunctionId"]
        },
        {
          "Fn::GetAtt": ["MutationUpdateUserFunction", "FunctionId"]
        }
      ]
    },
    "RequestMappingTemplateS3Location": {
      "Fn::Sub": [
        "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/${ResolverFileName}",
        {
          "S3DeploymentBucket": {
            "Ref": "S3DeploymentBucket"
          },
          "S3DeploymentRootKey": {
            "Ref": "S3DeploymentRootKey"
          },
          "ResolverFileName": {
            "Fn::Join": [".", ["Mutation", "updateUser", "req", "vtl"]]
          }
        }
      ]
    },
    "ResponseMappingTemplateS3Location": {
      "Fn::Sub": [
        "s3://${S3DeploymentBucket}/${S3DeploymentRootKey}/resolvers/${ResolverFileName}",
        {
          "S3DeploymentBucket": {
            "Ref": "S3DeploymentBucket"
          },
          "S3DeploymentRootKey": {
            "Ref": "S3DeploymentRootKey"
          },
          "ResolverFileName": {
            "Fn::Join": [".", ["Mutation", "updateUser", "res", "vtl"]]
          }
        }
      ]
    }
  },
  "DependsOn": [
    "MutationUpdateUserFunction",
    "InvokeUniqueUsernameCheckLambdaDataSource",
    "InvokeCalculateProfileCompletionLambdaDataSource"
  ]
}

That's about it!

At this point, our resources are only defined - not created. Execute amplify push and your resources should get created in AWS. Your updateUser mutation would now check for uniqueness of the username field.

It's bonkers that for something so simple, amplify makes one sweat it out. I had to google a lot and would like to mention 2 posts that helped me out:

This stackoverflow answer - outdated, but points in the right direction:

Jan 4 '19 Comments: 1 Answers: 2

AWS-Amplify provides a couple of directives to build an GraphQL-API. But I haven't found out how to ensure uniqueness for fields.

I want to do something like in GraphCool:

type Tag @model @searchable {
  id: ID!
  label: String! @isUnique
}

This is an AWS-Amplify specific question. It's not about how…

Open Full Question

and this article right here on Dev.to - they talk about how to have business logic BEFORE your mutation executes. I don't think I could have figured out the above without their pointers. Cool stuff.

Add Pipeline Resolvers in AWS Amplify

Satvik Daga ・ Jul 18 '20

#aws #graphql #react #serverless

My journey developing the Github Action to Jazz up the code blocks in issues.

Mithun Kamath — Thu, 20 Aug 2020 06:07:33 +0000

Follow up of my submission to the #ActionsHackathon that you can find below:

Jazz up the code blocks in Github Issues

Mithun Kamath ・ Aug 19 '20

#actionshackathon #github #showdev #javascript

My Github Action takes fenced code blocks in the issues and generates beautiful images for them using Carbon and replaces the code block to display the image instead. Why? Because (A) it looks slick, (B) It allows formatting the code and styling the syntax using guidelines you chose and (C) it makes it easy to read the code on mobile devices since one no longer has to scroll but only zoom.

Here's a document of my project journey - from how I started to the problems I encountered along the way and how I managed to resolved them to end up with a published version:

I started off by first going through Github Actions' awesome tutorial on how to create an Action. I know Nodejs and thus felt right at home taking the Javascript based Actions approach.
After that, I started off my exploring whether Carbon had any APIs. Could not find anything official but a search through their issue trackers led me to Carbonara, which allowed calling their service to generate the images

petersolopov / carbonara

API for carbon

Carbonara

API for carbon.

Getting started

Send POST https://carbonara.solopov.dev/api/cook to take an image of code snippet.

How it works

Puppeteer visit https://carbon.now.sh.
Mapping all params in properly URL params.
Taking a screenshot of the editor.

POST `/api/cook`

Body is JSON or multipart with next params:

parameter	default	type	description
`code` (required)	string	code snippet
`backgroundColor`	`"rgba(171, 184, 195, 1)"`	string	hex or rgba color
`dropShadow`	`true`	boolean	turn on/off shadow
`dropShadowBlurRadius`	`"68px"`	string	shadow blur radius
`dropShadowOffsetY`	`"20px"`	string	shadow offset y
`exportSize`	`"2x"`	string	resolution of exported image, e.g. `1x`, `3x`
`fontCustom`	`""`	string	custom woff font's contents, encoded in base64
`fontSize`	`"14px"`	string	font size
`fontFamily`	`"Hack"`	string	font family, e.g. `JetBrains Mono`, `Fira Code`. See all names in carbon exported config.
`firstLineNumber`	`1`	number	first line number
`language`	`"auto"`	string	programing language for properly highlighting. See name in carbon exported config

…

View on GitHub

That was my first step - to generate the images from the code blocks. Having extracted the code block using a markdown parser library, I used the Carbonara API to generate the image. I first tested it locally and then used debug logs to verify that it indeed was generated in Github.
Having confirmed that, I moved to the next step - storing the image. I could not find any Github API to upload images / files to (the ones where we can upload within the issue itself) so had to resort to a third party API - Imgur in my case. I was familiar with Imgur and their API documentation was easy to follow so settled with that as my image host.
Finally I brought the two together: Generating the image and then hosting it on Imgur.

Comment for #1

callmekatootie commented on Aug 17, 2020

Moment of truth.... Added the final bit - of updating the comment with the image and removing the code block... Let's see if this works now...

View on GitHub

Now that I had the bare minimum completed, I decided to focus on improving it. Using Prettier came to mind. Wouldn't it be better if the code was formatted too, to make it easier to read and remove the responsibility of readability from the author? I quickly explored their documentation to determine how to use it programmatically and integrated it into my Action.
But Alas! Looks like I broke something. It did not generate the image. The logs also did not throw up anything promising. At first I assumed I had not integrated Prettier correctly. But that did not throw up anything promising. Frustrated at not being able to determine what caused the failure I traced my steps back and figured out that, during the download of the generated image and subsequent upload, I might not have coded it properly. If the image had a larger size, it would end up with a partial download and that might have contributed to my failure earlier.

Apr 17 '20

146

Actually, I believe the previously accepted answer has some flaws, as it will not handle the writestream properly, so if you call "then()" after Axios has given you the response, you will end up having a partially downloaded file.

This is a more appropriate solution when downloading slightly larger files:

…

Open Full Answer

I corrected my code to correctly download the image in full and tried again. Sweet! It worked. That indeed was the cause of the earlier failure.
Onwards! I then decided to test my inputs - that allowed the user to specify the configurations for Prettier and Carbon. They worked well.
I then came across a scenario where Carbon had not styled the code properly. It failed midway and assumed everything to be a string after a certain line, and styled it as such. That made me realize that there could be a possibility that the Action may fail, and on such a occasion, losing the original code block might make matters worse. Thus, I decided to add that original source code into the issue after the generated image, but as a collapsed element.
I experimented with multiple markdown formats before settling on one that looked good.

Comment for #1

callmekatootie commented on Aug 18, 2020

trying again...

View raw code

function HelloWorld({greeting = "hello", greeted = '"World"', silent = false, onMouseOver,}) {

  if(!greeting){return null};

     // TODO: Don't use random in render
  let num = Math.floor (Math.random() * 1E+7).toString().replace(/\.\d+/ig, "")

  return <div className='HelloWorld' title={`You are visitor number ${ num }`} onMouseOver={onMouseOver}>

    <strong>{ greeting.slice( 0, 1 ).toUpperCase() + greeting.slice(1).toLowerCase() }</strong>
    {greeting.endsWith(",") ? " " : <span style={{color: '\grey'}}>", "</span> }
    <em>
    { greeted }
    </em>
    { (silent)
      ? "."
      : "!"}

    </div>;

}

outside text

View on GitHub

All is well! I finally ended up doing some housekeeping - cleaning up my code, renaming my Action, adding a README etc and finally published my Action to the marketplace

You can actually see, step by step, how I arrived at the final version of my Action by checking this Github issue out. Every time I added significant code, I would test my Action out.

My test issue #1

callmekatootie posted on Aug 16, 2020

Test issue to test my github action that will convert code snippets to beautiful images using Carbon.

const test = require('my-test-module')

console.log('Testing my stuff')

View on GitHub

References

Cover Photo by Nick Morrison on Unsplash

Jazz up the code blocks in Github Issues

Mithun Kamath — Wed, 19 Aug 2020 15:50:49 +0000

My Workflow

My Github Action is named Carbonate

Diving right in - Let's see if you can figure out what my Action does. Here's what your issue(s) look like BEFORE my Action takes over:

and here is what they will look like after my action has run:

Spot the difference?

The workflow of my Github action is as follows:

It extracts the code block for the issue description / comment and generates beautiful images for them
It then inserts the image at the code block
It also retains the original code block as a collapsed detail in the same issue / comment body

Additionally, it

Allows formatting the code using Prettier and controlling the styling of the images generated
Supports the following events:
- issue_comment:
  - types: created
- issues:
  - types: opened

Submission Category:

I am submitting this Action under the Phone Friendly category. Although I am of the opinion that Maintainer Must-Haves would also have been a good category to submit under (Why wouldn't any code maintainer not want to have some gorgeous looking images of the code??), I am submitting this for the Phone Friendly category.

On mobile devices, reading the code in its text form can be a difficult task. It puts the responsibility on the author to format their code correctly, to specify the language of the fenced code block in the markdown and to adhere to any guidelines the project may have. This could lead to discouragement of reporting the issues and potential embarrassments.

Automatically formatting the code for the author eliminates this hurdle. Having (oh-my-god-they-are-so-beautiful) images generated for the code allows folks to easily read and view the code in its entirety. No more scrolling - only zooming, which is an easier task.

The benefits of this Action for mobile device users are clear and hence I submit it under the Phone Friendly category.

Yaml File or Link to Code

callmekatootie / carbonate

Github Action to format fenced code blocks in github issues as images. Originally created as part of DEV Github Actions hackathon: https://dev.to/callmekatootie/jazz-up-the-code-blocks-in-github-issues-52e6

Carbonate

Jazz up the code blocks in your issues. Generate beautiful images for them to make it easier to follow. Meant to be used as a Github Action.

BEFORE

AFTER

First appeared at: https://stackoverflow.com/a/61269447/2104976

Features

The workflow of this action is as follows:

It extracts the code block for the issue description / comment and generates images for them
It then inserts the image at the code block
It also retains the original code block as a collapsed detail in the same issue / comment body

Additionally, it

Allows formatting the code using Prettier and controlling the styling of the images generated
Supports the following events
- issue_comment
  - types: created
- issues
  - types: opened

Not supported (yet)

Generating images from multiple code blocks in the same issue description / comment
Generating images after the issue description / comment has been edited

Advantages of code images over code blocks

Easy to view and…

View on GitHub

Additional Resources / Info

I only now created this Action and thus there are no repositories that are currently using it (at the time of writing this). However, I am using the Action in my Action's own repository - so feel free to get a demo of it in action by creating issues here. (Don't forget to include fenced code blocks in your comments)

I am making use of Carbon to generate the images and the images themselves are hosted on Imgur. The code is optionally formatted using Prettier.

Cover Photo by Joshua Aragon on Unsplash
The content of the issue in the screenshots were actually borrowed from here. The answer helped me in fixing an issue with my code and I found it suitable to demo.

An interesting quirk about Github labels

Mithun Kamath — Thu, 08 Aug 2019 09:01:08 +0000

Earlier this Sunday I was trying to create labels for my Github Issue Tracker and I observed something. See if you can spot it below:

Notice how the label text changes color? And not just the label's background? Even more quirky is that the label color is either white or black and it's always a contrast to the label's background. If the background color is dark, the label text is colored white and black otherwise.

Cool! And weird too! How does Github know if the color that I set needs a black or white label text?

Curious, I first tried to determine if this happens in the front end itself or if it happens at the backend. A quick check of the Network tab tells me that it's actually happening on the backend - when we try to assign a color, the color is passed to the backend that responds with the span tag that needs to be showed, with an inline style that sets the label text to either black or white. I could not reverse engineer this further and hit a dead end.

I then googled how to determine if a color is dark or light in Node.js (to correspondingly add a black or white color to the label's text) and came across the color module. This has the two following functions that determine if the color passed to is dark or light:

color.isLight();
color.isDark();

Cooool! Going through the source code of this module to figure out what they do in these methods led me to this blog that goes into detail about how this works. I'd hate to give a TL;DR version and if it's any solace, that link isn't that long to read.

I'd imagine Github is using one of the two methods in that blog (or perhaps the color module itself) and returning the final element to display. Small but coool feature to provide.

DEV Community: Mithun Kamath

How did they do it | Control the lights in a room using your hand

The Smart Bulb

The setup

The code

index.html

script.js

initVideo()

initPoseDetector()

analyzeHandGesture()

recognizeGesture()

start()

Finale

Be(a)ware - AWS Amplify does not have migrations

Debug "Resource not accessible by integration" error when working with github's graphql endpoint

Amplify Quirks

Appsync / Graphql

Combination of private and groups authorization types

AWS AppSync with Amplify: How to ensure a field in a type is unique

Things to know before we proceed

Step 1: Create a Global Secondary Index (GSI)

Step 2: Lambda function that checks for uniqueness of the field's value

Step 3 - An IAM role that allows AppSync to invoke the earlier created Lambda function

Step 4 - Create a Lambda data source

Step 5 - Create an Appsync Function

Step 5A - Create the request and response mapping template(s)

Step 5B - Actually define the Appsync function

Step 6 - Create a Mutation Appsync Function

Step 6A - First define the request and response mapping templates

Step 6B - Now define the appsync function

Add Pipeline Resolvers in AWS Amplify

Satvik Daga ・ Jul 18 '20

My journey developing the Github Action to Jazz up the code blocks in issues.

Jazz up the code blocks in Github Issues

Mithun Kamath ・ Aug 19 '20

petersolopov / carbonara

API for carbon

Carbonara

Getting started

How it works

POST /api/cook

Comment for #1

Comment for #1

My test issue #1

References

Jazz up the code blocks in Github Issues

My Workflow

Submission Category:

Yaml File or Link to Code

callmekatootie / carbonate

Github Action to format fenced code blocks in github issues as images. Originally created as part of DEV Github Actions hackathon: https://dev.to/callmekatootie/jazz-up-the-code-blocks-in-github-issues-52e6

Carbonate

Features

Not supported (yet)

Advantages of code images over code blocks

Additional Resources / Info

An interesting quirk about Github labels

Combination of `private` and `groups` authorization types

POST `/api/cook`