DEV Community: Royal Bhati

DynamoDB Streams

Royal Bhati — Fri, 18 Jun 2021 06:28:28 +0000

While working on a backend service, I came across this problem where I had to retrieve a set of data of a specific type, but that type wasn’t having any index and the data was also updating frequently.

Since the data was constantly changing, the first thing that came to mind was to attach a trigger. However, when looking at the documentation on how to add triggers to DynamoDB in AWS, I opted to go with another approach :p

By the time I had time to consider, there were three viable options that were obvious without lengthy consideration

→ Complete scan of the table and filter the results
From the very first moment it entered my head, I knew this is a very inefficient solution

→ Add an Index for that type

There were a couple of reasons I didn’t go along with this idea

That wasn't a very frequently queried data
Already had so many indexes so didn't want to add one more just yet before trying other solutions

→ Run a cron-job
A cron-job that will perform a table scan, store the result in a cache, or store it in another table.

A Great man once said, If you can't think of anything else, run a cron-job.

Back in my mind, I wasn’t quite convinced with any of the above solutions and so I gave Dynamo db triggers try. I was aware of Dynamo db streams but never tried them.

DynamoDB Streams are basically triggers like that we have in Relational DBs but the only difference is that it generates a stream of events when there is a data change and you can easily pipe that stream into Kinesis or a Lambda (In my case it was lambda).

AWS made the things look difficult but the process was not complex at all.

All I had to do was the following -

Setup a stream on the dynamodb table which was just a one click action.
Create a new lambda function and attach it to the dynamodb stream which was also fairly easy
Rest was basic stuff of listening the data then processing it and finally saving it to the new table

DynamoDB streams are much more than what I have covered in my use case. You can read more about it here : -https://aws.amazon.com/blogs/database/dynamodb-streams-use-cases-and-design-patterns/

Go project layout

Royal Bhati — Mon, 08 Feb 2021 12:36:09 +0000

Go is very flexible when it comes to writing apps and structuring them.
Though, you should always write idiomatic Go when possible.

While not thinking about structuring your apps is okay, but as your app starts to grow, you have to have a structured architecture so that it's easy for you and other devs to know how things connect in your app. It also avoids duplication of code and, maybe, will help you scale heights.

I was looking for that one structure I could follow, but almost everyone has their stuff.
So, I started reading blogs and even started reading open-source projects and finally picked the best parts.
Before showing you the structure, I'll try to build your mental model for this architecture by this analogy.

Let's say you want to start a business and the product your building is a Dancing Doggos App, which will contain dancing videos and photos of dogs.

Exciting right?

Now, what should we do next?

The first that comes to my mind is management that can manage our product specific to our needs.
So we need to hire product managers, CEO, CTO and other fancy managers with those suits and ties :p

Next, we need is the people who'll be working under our Managers.
So, marketing team, sales team, and a team of amazing people aka developers :p

Now that we have everything ready to go, we need to think about how everything will work.
The following few lines are going to be the crucial part of all the useless stuff I wrote.

Product Owners -

They want their idea out in the market.
They only talk to the managers.
They don't care how you do it ( of-course not if its unethical :p)

Managers -

They are the people who are responsible for getting the work done.
They manage the people working under them to work according to the needs
Their work is mostly product specific.
Their decisions and choices are governed by the product they are building. (A Manager who is working for Pepsi will have to make different decisions if he starts working for Gucci)

Teams -

They usually don't care what you are building
They usually don't have to change their decisions based on the product (A Go Developer working on Go in Pepsi will also be working on Go if he makes a switch to Gucci).
Teams are usually independent and don't require any other team to make their work done. ( Marketing team doesn't depend on Devs team to work)

You might be thinking I have read so much and still haven't found anything I can take away from this but please bear 🧸 with me for the next few minutes.

Building upon our business workflow, we can structure our app.

/app

Product owner

This folder contains our business logic, and that's it.

This directory doesn't care about what database driver you are using or which caching solution your using, or any third-party apps.

Its only job is to provide the business-related details, including structuring the models, authorizing the clients, storing them in the database, or sending it to the clients.

/platform

teams

This folder contains all the platform-level logic that will build up the actual product, like setting up the database, Redis instance, or maybe a router.

All of the packages inside this directory are independent of each other.

Packages inside this folder also don't care what's inside /app or /cmd

/cmd

Management

This folder contains all the application-specific code tailored only for your business use case—things like configuration, logging, and DB migrations.

This directory talks to the /app for instructions, and then it takes the services offered by /cmd and finally delivers the product
This directory is the Mitochondria of our application (Bad science pun, sorry :p)

tl;dr

Your app's business-related logic shouldn't care about the platform and app-level code like routes, databases, and third-party packages you are using.
Your Application-level code (/cmd) like configurations and routes will vary according to different services or business ideas.
Your Platform level code doesn't care what you are building, and they should only have a single purpose. It shouldn't produce any side-effects inside your business logic.

Here is the WIP project uses the above structure

The project below is just a work in progress and only for my learning purpose, but the structure is handy if you want to use it in your next project.

https://github.com/royalbhati/url-shortener

Remove unwanted properties from Javascript Objects using ES6 destructuring

Royal Bhati — Tue, 26 May 2020 09:59:46 +0000

lets say we received an object from an api response which contains some sensitive data


const userData = {
  name:"Rick Sanchez",
  email:"ricksanchez@getSchwifty.com"
  address :" Dimension C-137",
  sensitiveInfo:"wubba lubaa dub dub",
  secretportalkey:"ABCDEF"
}

Now we want to pass it to some third party api but we dont want to pass the sensitiveInfo and secretPortal key to the Api

So, How do we remove that from the Object ?

Simple, just use the power of destructuring

const {sensitiveInfo,secretportalkey,...notSoSensitiveData} = userData

/* Now our notSoSensitiveData will contain all the properties except the ones which
 are destructured
*/

console.log(notSoSensitiveData)

/* {
  name:"Rick Sanchez",
  email:"ricksanchez@getSchwifty.com"
  address :" Dimension C-137",
}
*/

Understanding weird parts of Javascript

Royal Bhati — Mon, 25 Nov 2019 21:20:01 +0000

Since JavaScript is a weakly-typed language, values can also convert automatically between different types, and it is called implicit type coercion. People who don't understand the quirks of javascript tend to make fun of the language by sharing memes like this.

Now in this post, I am going to walk through every example given in this meme and try to explain briefly and try to link references if possible.

While every single example mentioned in this meme would have been a standalone long post, I kept it short to keep it concise and beginner-friendly. Might throw some more light on examples separately in future.

Let's get started

console.log(typeof NaN) // "number";

According to the ECMAScript standard, Numbers should be IEEE-754 floating-point data. This includes Infinity, -Infinity, and also NaN.

NaN stands for Not a number.

Let's see when NaN is returned :

Division of zero by zero. ( 0 / 0)
Dividing an infinity by infinity. (Inf / Inf)
Multiplication of an infinity by a zero. ( Inf * 0)
Any operation in which NaN is an operand. (NaN + 2)
Converting any undefined or non-numeric string to a number. Number("abc")

Saw anything common?
NaN is returned only when there is a numerical operation.

By definition, NaN is the return value from operations that have an undefined "numerical" result.
So, It is evident that the type of NaN would be a number.

Reference :

2 .

console.log(999999999999) // 10000000000000;

This happens because JavaScript only supports 53-bit integers.
All numbers in JavaScript are floating-point which means that integers are always represented as

sign × mantissa × 2^exponent

The fraction occupies bits 0 to 51, the exponent occupies bits 52 to 62, the sign occupies bit 63.
So js suffers from a loss of precision where the least significant digits disappear if the number is huge.

There has been a recent addition to javascript "BigInt" which solves the problem of representing whole numbers larger than
2^53 - 1

References:

console.log(0.5 + 0.1 == 0.6); // true
console.log(0.1 + 0.2 == 0.3); //false

I have already mentioned it above but gonna repeat it, JavaScript uses a 64-bit floating-point representation according to IEEE 754.
64-bit Binary floating-point format cannot accurately represent a number like 0.1, 0.2 or 0.3 at all. While most of the languages round off the number to give results as expected but JS don't.

If you convert 0.1 to binary representation you would end up with a 0.00011001100110011.... (endless repetition of 0011 ).
This post explains it in detail
In double-precision floating-point (JS), 53 bits are used, so the otherwise infinite representation is rounded to 53 significant bits. So results are always inaccurate in decimal.
This stack overflow answer also explains it very well - https://stackoverflow.com/a/28679423

References :

https://tc39.es/ecma262/#sec-ecmascript-language-types-number-type

4 .

console.log(Math.max())  //-Infinity
conosle.log(Math.min()) //+Infinity

First Let's make it clear that...
THEY DON'T RETURN LARGEST OR MINIMUM NUMBERS, for those needs we have Number.MAX_VALUE and NUMBER.MIN_VALUE.

Math. max() and Math. min() are static methods that return maximum and min values among the respective arguments.
So according to the spec, if you call them without any arguments, they would return -Inf and +Inf.

While the spec doesn't say anything regarding why it does that, so I looked into the chromium source code to find that out.
To be honest, I found what I was thinking i.e. whenever you call the Math. max() method with a single argument(Math. max(100)), it compares it with -Infinity and returns the Number itself because if it is a valid number, it would always be more significant than -Infinity.
Same goes for Math.min().
So when there is no argument to compare it returns -Infinity as its the max value between nothing and -Inf.

console.log([]+[]) // ""

According to the spec, when javascript encounters The Addition Operator ( + ) it performs following steps.

These steps will build foundation for upcoming few examples.

a. Convert both operands to Primitive values
b. If any of the operand is type String then return result as string concatenation
c. Else convert both the Operands to Number using ToNumber()
d. If type of one operand is different from other then throw a TypeError
e Else return the mathematical sum

So lets walkthrough our example :

a. First operands are first converted to their primitive values if they are not, which in this case are not primitive .

b. Now the ToPrimitive converts the object type to a primitive type. Along with the input, ToPrimitive also accepts an optional "Preferred type" parameter which is provided to give ToPrimitive a hint of the type.

c. After converting to the primitives if any of the primitive is of type string,then string concatenation takes place which in this case is true (explained below) and we see a "" as a result.

Let's se how ToPrimitive works :

When there is no hint is given, ToPrimitive defaults the hint to Number.
After deciding the hint, it checks it against list of two methods in a defined order.
[valueOf, toString] in the case of hint Number and reverse in the case of String.
In this case it uses the default hint so the following steps are taken
a) [].valueof returns Array itself ans since its not primitive, so it goes to the second method
b) [].toString returns "" and since it returns a primitive value it is returned as a primitive value.

References :
https://tc39.es/ecma262/#sec-addition-operator-plus

6 .

console.log([]+{}) // "[object Object]"

Adding to the explaination above, {}.toString is [object Object] so by string concatenation we get this result.

{} + []

Now this example would return the same result as [] +{}.

But does that mean the meme has a typo?

NO, but If you try that example in the console of chrome or firefox it would return 0.
This is becaus Object literal in starting is treated as an empty code block and is ignored by interpreter and we are left with this
expression " + [] ".
Now the unary "+" operator converts its operand to Number and Number([]) equals to zero.

Reference:
-https://tc39.es/ecma262/#sec-unary-plus-operator


console.log(true + true + true) // 3
console.log( true - true) //0

Now according to algorithm in the 5th point, we already have a primitive value i.e boolean and since none of the operand is a string we convert the first two operands to Numeric and according to the spec, ToNumber(Boolean) is 1 for true and 0 for false.
So true + true gives 1+1 = 2

No we have "2 + true" which is processed same as we did for the first two operands and we get 3.

So both of the results makes sense now.

Reference :
- https://tc39.es/ecma262/#sec-tonumber

9 .

console.log(true ==1) //true
console.log(true ===1) //false

First expression is compared using Abstract Equality Comparison which allows coercion so according to the spec

If Type(x) is Boolean, return the result of the comparison ToNumber(x) == y.

So using that we get ToNumber(true) ==1

Second expression is compared using Strict Equality Comparison which does not allow coercion so according to the spec

If Type(x) is different from Type(y), return false.

Clearly, types are different so result is false

10.

console.log((!+[]+[]+![]).length) // 9

This is was a mind fuck when I was first saw this :p

Lets divide this in four pieces.

lets begin with !+[]

Now here, we don't have two operands but two unary operators (! & +) and since ! and + has same precedence we start from left to right.
We first encounter "!" whose associativity is right to left so we evaluate "+[]" which results in 0 (we have already discussed why the result is zero in 7th post)

Now negation operator according to the spec converts the operand into boolean if its not already so if we convert 0 to boolean we get a false.
So, "!false" returns true which is of type boolean.

Now we have (true + [] + ![]).length

Taking "true + []" for evaluation according to the rules of addition operator we get a result "true" of type string because the primitive value of empty array is an empty string and if any of the operand is of type string we perform string concatenation.

Now we are left with ("true"+![]).length

So according to the rules of "!" operator we convert [] to boolean which results in true (According to the spec ToBoolean(object) returns true).
So now,placing true in place of empty array we get "!true" which results in false and since one of our operand is of type string we concatenate the operands which results in "truefalse"

Now its obvious why console.log("truefalse".length) returns 9

Reference :
- https://tc39.es/ecma262/#sec-toboolean
- https://tc39.es/ecma262/#sec-logical-not-operator

11.

console.log(9+"1") //91

We have already discussed this but I am gonna mention it again.
According to js spec if any one of the operand's primitive value is of type string then we concatenate the operands's primitive value which results in "91"

console.log(9-"1") // 90

According to the spec, In subtraction operands are coerced to Number and if the ToNumber results are valid then final result is a mathematical subtraction.

console.log([]==0) //true

As mentioned earlier, Double equals uses Abstract Equality Comparison which allows coercion so our empty array is coneverted to its primitive value which is "" and according to the spec

If Type(x) is String and Type(y) is Number, return the result of the comparison ToNumber(x) == y.

and ToNumber("") is 0 so that is why we get a true.

So next time if anyone shares this meme with you then you have answers to keep them shut.

Temp

Royal Bhati — Mon, 25 Nov 2019 13:45:47 +0000

Write your first webpack plugin

Royal Bhati — Tue, 19 Nov 2019 14:21:39 +0000

If you have clicked on the title, then I assume you have a basic understanding of webpack and webpack plugins.

Plugins are great and helps you leverage the power of webpack.
There are tons of plugins available for webpack, some of the famous ones are HtmlWebpackPlugin , HotModuleReplacementPlugin and webpack-bundle-analyzer.

While these are great but there are always cases when you need a project specific plugin and it gets worse when you are unable to find any plugin for your needs.

How about creating your own Plugin ?

In this post we will create a plugin to remove 'console.logs' from the production build.

Let's begin by creating a simple react project.

If your react project is not setup then follow these steps to create a basic react starter.

$ mkdir myplugin && cd myplugin
$ npx react-starter-lite

react-starter-lite is a simple boilerplate package to easily get your hands dirty on react projects.

Run the app to see if its working properly


$ yarn dev:hot

After setting up the directory create a file named logRemover.js in your project directory and paste the following code.

//logRemover.js

module.exports = class RemoveLogs {
     constructor(options){
     this.options = options
}

  apply(compiler) {
    console.log("Hello from the new plugin")
};

Lets see whats happening in the code.

In the first line we are exporting the class we are creating.Plugins can be created using functions also but I have used classes here.
Constructor has an paramter options which accepts arguments related to the logic of the plugin. In this post we won't be needing the options but I showed you just show that you know.
apply() is a mandatory method which recieves a compiler instance as an argument which contains basically everything you need to do make your plugin.Since we are not using compiler yet I will tell you a bit more compiler when we are done with basic setup.
- Console.log() just to see if our setup is working

After setting up the logRemover.js we need to invoke that plugin in our webpack configuration.

In the project directory we have three webpack configurations, one is the base config and the other two are development and production configs.

Since we just need to remove console.logs from the production build only, we need to add that plugin in the webpack production config.
But for testing the setup, for now, we'll add that to development config and will change that later when we'll have logic for our plugin.

So open webpack.config.dev.js and invoke the plugin like this.


//webpack.config.dev.js

const merge = require("webpack-merge");
const baseConfig = require("./webpack.config-base");
const RemoveLogs = require('./logRemover.js');

module.exports = merge(baseConfig, {
  mode: "development",
  devServer: {
    port: 9000
  },
  devtool: "source-map",
  plugins: [new RemoveLogs()]
});

Now lets test if our setup was successfull.
Stop if the webpack server is running and run

$ yarn dev:hot

Now if you look in to you terminal you'll see that console.log statement.Since there is so much on terminal, you need to look carefully to find your log message.

Congrats if you have made this far :p

Now since our basic setup is working let's write the logic for our plugin inside logRemover.js


//logRemover.js

const fs = require("fs");
module.exports = class RemoveLogs {
  apply(compiler) {
    compiler.hooks.done.tap("RemoveLogs", stats => {
      const { path, filename } = stats.compilation.options.output;
      try {
        let filePath = path + "/" + filename;
        fs.readFile(filePath, "utf8", (err, data) => {
          const rgx = /console.log\(['|"](.*?)['|"]\)/;
          const newdata = data.replace(rgx, "");
          if (err) console.log(err);
          fs.writeFile(filePath, newdata, function(err) {
            if (err) {
              return console.log(err)
            }
            console.log("Logs Removed");
          });
        });
      } catch (error) {
        console.log(error)
      }
    });
  }
};

Let's see what this code is doing

1) apply(compiler) method takes in a compiler instance which contains events.
Compiler module is the main engine of the webpack which emits events just like dom. It extends Tapable lib to register and call plugins. In this logic we are tapping on hook "done" which is emitted after the compilation is completed and bundle is generated.

2) using these compiler events we call our plugin to do its job

3) Its job is nothing fancy but just to take the bundled js and pass it through a regex and remove all the console.logs and output the bundled js in the dist directory.

Now since we are done with the logic we will remove the plugin from the dev config in webpack.config.dev.js and put it under production config in webpack.config.prod.js


//webpack.config.prod.js

const merge = require("webpack-merge");
const baseConfig = require("./webpack.config-base");
const RemoveLogs = require("./logRemover.js");
module.exports = merge(baseConfig, {
  mode: "production",
  plugins: [new RemoveLogs()]
});

To see if its working we will write a console.log statement in App.js file

//src.App.js

import React from "react";
import "./styles.css";
const App = () => {
  console.log("test console.log statement ");
  return (
    <div className="card">
      <div className="content">
        {"{..."} &#9829;{"}"}
        <h5>! hate</h5>
      </div>
    </div>
  );
};

export default App;

Now stop the webpack server and generate the build by


$ npm run build

Now open the index.html from the dist directory and check if there any console.log statement or you can check in the bundeled js in the editor by doing ctr/cmd + F in bundeled file.

That's all for today :)

Please feel free to comment and ping me if you have any doubts.

Here is the github project if you want to see the full source code :

webpack-LogRemover-plugin

Additional Reading :

Compiler

Plugins