DEV Community: Hari Krishnan

FSD - 1.1.4 - Setting up Docker

Hari Krishnan — Sun, 05 Jan 2025 07:40:16 +0000

What is Docker ?

Docker is an open-source platform that allows developers to automate the deployment of applications within lightweight, portable containers. These containers bundle the application code with all its dependencies, libraries, and configurations, ensuring it runs consistently across various environments.

Why do we need Docker ?

Portability: Docker containers run the same way on any system—whether it's your laptop, a testing server, or a production cloud instance.
Consistency: Eliminates the "works on my machine" problem by ensuring uniform environments.
Efficiency: Containers are lightweight and use fewer resources than traditional virtual machines.
Rapid Deployment: Docker simplifies and speeds up the deployment process.
Isolation: Each container is isolated, reducing conflicts between dependencies and improving security.

Using Docker for Development

1. Set Up a Local Development Environment

The Problem Without Docker:

You need to install the right version of Node.js, databases (like MongoDB), or other tools manually.
Different versions or setups can cause "it works on my computer but not on yours" issues.

How Docker Helps:

You use Docker images (pre-configured setups) for Node.js and any other tools you need.
Your team gets the same environment on their machines with just one file

2. Collaborate Easily with Your Team

The Problem Without Docker:

One teammate might have Node.js 18, another has Node.js 16, and things break.
Setting up tools for each new project takes time.

How Docker Helps:

Share the docker-compose.yml file with your team.
Everyone runs docker-compose up and gets the exact same environment, regardless of their computer.

Why this is helpful:

No "works on my machine" issues.
New team members can start coding immediately without setup headaches.

3. Test Different Setups Easily

Example

Your app works with Node.js 18, but you want to test it with Node.js 16.
Instead of reinstalling Node.js, just switch the version in your Docker setup and restart

Using Docker for Production

App Deployments

Think of Docker as a way to pack your app into a "box" (called a container) that includes everything it needs to run. This makes sure your app behaves the same, no matter where it's running.

Deploy Your Node.js App with Docker

Imagine you're building a Node.js app (e.g., a website or API) and want to run it on a server for users to access. Instead of setting up Node.js manually on the server, you package your app with Docker.

Steps:

Create a Docker Image:

This is like making a blueprint for your app that includes:

Your app code.
Node.js (the version your app needs).
Any libraries your app uses.

Push the Image to a Registry:

Think of this as uploading your "box" to a storage area (e.g., Docker Hub or AWS).

Run the App on a Server:

Use the image to start a container on a server.

Example:

You have a Node.js app that runs on port 3000. When you run the container, your app is ready to serve requests on that port.

Why this is helpful:

No worries about whether the server has the right Node.js version or dependencies.
If something works on your computer, it'll work on the server too.

SETTING UP DOCKER

Setting Up Docker on Ubuntu

Step 1: Update Your System

Run the following commands to update your system packages:

sudo apt update
sudo apt upgrade -y

Step 2: Install Docker

1. Uninstall Old Versions (if any):

sudo apt remove docker docker-engine docker.io containerd runc

2. Install Docker Using the Official Repository:

sudo apt install -y apt-transport-https ca-certificates curl software-properties-common
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce docker-ce-cli containerd.io

3. Verify Docker Installation:

docker --version

Step 3: Allow Docker Without sudo

Add your user to the docker group:

sudo usermod -aG docker $USER

Log out and log back in for the changes to take effect.

Step 4: Install Docker Compose

Docker Compose is a tool to manage multi-container applications.

Install it with:

sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose

Make it executable

sudo chmod +x /usr/local/bin/docker-compose

Verify the installation:

docker-compose --version

2. Setting Up Docker on macOS Using Colima

Colima is a lightweight, Docker-compatible container runtime for macOS that doesn’t require Docker Desktop. It uses qemu under the hood and integrates seamlessly with the Docker CLI.

We use this as Docker Desktop is not for free anymore.

Step 1: Install Colima

1. Install Colima via Homebrew:

brew install colima

2. Verify the installation:

colima version

Step 2: Install Docker CLI

Install Docker CLI tools via Homebrew:

brew install docker docker-compose

Verify the installation:

docker --version
docker-compose --version

Step 3: Start Colima

Start Colima with Docker support:

colima start

Verify Colima is running:

colima status

Step 4: Test Docker with Colima

Run the hello-world Docker image to confirm everything works:

docker run hello-world

Check running containers:

docker ps

At this point, we're setting up the Docker environment, which might feel overwhelming. Don't worry—later, we'll dive into what Docker is and how it will help us during development and production as we build the app.

FSD 1.1.3 - Setting up Version Control

Hari Krishnan — Tue, 03 Dec 2024 11:23:24 +0000

What is version control ?

Version Control is very important for any developer to keep track of code changes and simplify code management when we work with a team of developers.

It supports

Tracking Changes: Records changes to files over time, allowing you to see what changed, when, and by whom.
Collaboration: Allows multiple people to work on the same project without conflicts.
History: Keeps a record of changes, enabling you to revert to previous versions.
Branching: Lets you create separate branches for new features and merge them later.

Why do we need Version Control?

Collaboration: Multiple people can work on the same project without overwriting each other’s work.
Backup: Safeguards your code by allowing you to revert to a previous working state.
Tracking: Helps monitor project progress and identify when issues were introduced.
Experimentation: You can test new features safely using branches without affecting the main codebase.

In a development team, think of version control like a shared codebase where multiple developers work on different features (branches) without overwriting each other’s work. If a bug is introduced, you can track changes, revert to a previous version, and merge the new features once they’re tested and ready. It ensures smooth collaboration and helps manage project history.

To implement version control in this course, we will be using Git. Git is a distributed version control system used to track changes in code and manage source code history.

Git is the foundation for platforms like GitHub, GitLab, and Bitbucket, which provide additional collaboration tools like pull requests and issue tracking.

We will be using Github as the platform for application development.

CREATING A GITHUB ACCOUNT

SETTINGS UP GIT on macOS

Install Git:

Open the Terminal.
Check if Git is already installed by running:

git --version

If not installed, Git is often pre-installed on macOS. If it’s not, install it via Homebrew:

brew install git

Configure Git:

Set your global username and email for Git (used for commits):

git config --global user.name "Your Name"
git config --global user.email "youremail@example.com"

Verify Configuration:

Verify your settings:

git config --list

Verify the installation

git --version

SETTINGS UP GIT on Ubuntu

Install Git:

Open the Terminal.
First, update the package list:

sudo apt update

Install Git:

sudo apt install git

Configure Git:

Set your global username and email for Git (used for commits):

git config --global user.name "Your Name"
git config --global user.email "youremail@example.com"

Verify Configuration:

Verify your settings:

git config --list

Verify the installation

git --version

FSD 1.1.2 - Setting up a code-editor

Hari Krishnan — Tue, 03 Dec 2024 11:03:30 +0000

What is a code-editor ?

A code editor is a software tool specifically designed for writing and editing code. It is mainly used for

Syntax Highlighting: It colors different parts of the code (like keywords, variables, and functions) to make it easier to read and understand.
Autocompletion: It suggests code snippets, variable names, and functions as you type, helping you write code faster and with fewer errors.
Error Detection: Code editors often highlight errors or issues in your code, such as missing semicolons or typos, to help you debug and fix problems.
Integrated Tools: Many editors come with built-in features like version control, debugging tools, and terminal access, streamlining the development process.

Examples of popular code editors include Visual Studio Code, Sublime Text, and Atom.

Editor Installation

Visit the offical VS code website - https://code.visualstudio.com/download
There are many other alternative editors, but it is recommended to use VS code as it is open-source, built by Microsoft and has very good community support.

FSD 1.1.1 - Setting up Node

Hari Krishnan — Tue, 03 Dec 2024 10:51:01 +0000

Installing Node Js

Our application development is with React and Node, so both relies on node. In this step we will setup node in our machine

Steps to set up Node.js Using NVM (Node Version Manager)

We are using NVM to setup node because setting up only node from its official website allows you to setup specific versions of node, whereas when we use NVM it allows to switch between node versions locally and manage different versions of node in the system.

STEP 1 : Install NVM:

Open your terminal.
For macOS or Linux, run the following command to download and install NVM:

curl -o- https://raw.githubusercontent.com/nvm-sh/nvm/v0.39.5/install.sh | bash

Restart your terminal to apply the changes. Alternatively, you can load NVM manually by running:

For bash users:

source ~/.bashrc

For zsh users (on macOS, it's usually zsh):

source ~/.zshrc

This will load the NVM environment variables.

Verify NVM installation by running:

nvm --version

STEP 2 : Install Node.js Version 18

You can install the latest version as well.

Install Node.js version 18 using NVM:

nvm install 18

This will download and install the latest stable Node.js version 18.x.x.

Set Node.js version 18 as the default version (optional, but recommended):

nvm alias default 18

This ensures that Node.js 18 will be the default version used in new terminal sessions.

Verify the Node.js version by running:

node --version

This should output something like v18.x.x, confirming that Node.js version 18 is installed.

Verify the NPM version (optional):

Since NPM comes bundled with Node.js, check the NPM version:

npm --version

This should output the version of NPM installed along with Node.js.

STEP 3 : Managing Node.js Versions (Optional)

NVM allows you to easily manage and switch between multiple versions of Node.js:

To switch to a different version (e.g., version 14):

nvm use 21

To list all installed Node.js versions:

nvm ls

To install a specific version (e.g., version 16):

nvm install 16

Summary

Install NVM using the curl or wget script.
Install Node.js version 18 with nvm install 18.
Set Node.js version 18 as the default with nvm alias default 18.
Verify the installation with node --version and npm --version.
Optionally, manage multiple Node.js versions using NVM.

This process is identical for both macOS and Ubuntu.

Full Stack - 1.1 - Setting up the environment

Hari Krishnan — Mon, 02 Dec 2024 05:33:18 +0000

What is an Environment ?

An environment contains the resources required for developing an application, both at the software and hardware levels.

Hardware

A personal computer with at least 8GB of RAM, an Intel i5 equivalent processor, and 256GB of storage is essential.

Software

A Linux-based operating system with basic command line tools enabled is necessary. It is advisable for software developers to use Linux, as it is lightweight, open-source, and developer-friendly.

To dig deeper, here is what you need to set up on your personal computer:

System Level Setup:

Install Node.js and a package manager (e.g., npm).
Set up a code editor (e.g., Visual Studio Code) and version control (e.g., Git).
Set up Docker.
Set up Git.
Setup Postgres.

FSD 1.0 - Introduction to Full Stack Development

Hari Krishnan — Tue, 26 Nov 2024 08:07:28 +0000

In this course, we will be building an e-commerce application using the below tech stack

React
Node with Express JS
Mongo DB
Typescript
AWS Cloud [Amazon Web Services]
Docker

React

React is a JavaScript library for building web applications, particularly for single-page applications. It is maintained by Meta (formerly Facebook) and a community of developers. React is known for its efficiency, flexibility, and component-based architecture, making it a popular choice for front-end web development.

Node.js

Node.js is an open-source, cross-platform runtime environment that allows you to execute JavaScript code on the server side, outside of a browser. We will use node with Express JS for our backend in this course.

Express JS

Express.js is a web application framework built on top of Node.js, which simplifies and organizes backend development by providing tools and features like routing, middleware, and HTTP request handling. With Express, you can easily structure your backend code for tasks like:

Building RESTful APIs.
Handling routes and defining endpoints.
Managing middleware for request processing (e.g., authentication, logging, etc.).
Integrating with databases (e.g., MongoDB, MySQL).
Serving static files or data to the front-end.

By combining Node.js (for its runtime environment and asynchronous handling capabilities) with Express.js (for its organized and developer-friendly API), you can efficiently write and maintain the backend logic for your application.

Mongo DB

MongoDB is a NoSQL database that is designed for storing, managing, and retrieving large amounts of data in a way that is highly scalable and flexible. Unlike traditional relational databases (e.g., MySQL, PostgreSQL), which use tables and rows, MongoDB stores data in documents using a JSON-like format called BSON (Binary JSON).

Typescript

TypeScript is a superset of JavaScript developed by Microsoft that adds static typing and other features to JavaScript.
TypeScript code is transpiled into plain JavaScript, which can run in any environment that supports JavaScript (e.g., browsers, Node.js).

 let name: string = "John"; // Type-safe
 name = 42; // Error in TypeScript

Prevents unintended bugs caused by incorrect data types or usage.

  function add(a: number, b: number): number {
    return a + b;
  }
  add(5, "10"); //

There are lot of advantages in using typescript over javascript, let's look into it in detail later. But for now let's go ahead with Typescript.

Docker and AWS

What is Docker ?

Docker is an open-source platform that enables developers to build, package, and deploy applications in lightweight, portable containers. These containers include everything needed to run an application (code, runtime, libraries) and ensure consistency across environments.

AWS for Deployment

Amazon Web Services (AWS) is a comprehensive and widely adopted cloud computing platform offered by Amazon. It provides over 200 fully featured services from data centers globally, enabling individuals, companies, and governments to access on-demand computing resources.

We will be using AWS to deploy and make our application live for users across the globe.

Oh my god !!! Everything is new to me.

No worries. This course will teach you in detail every individual tech stack you see here. You don't have to go learn every single thing and then come build the application. We will be learning each technology here practically as we move forward.

You cannot learn every stack end-to-end spending hours, even if it is done so, you should implement it real-time to get expertise. This course will teach you what all base knowledge you should be having in each stack and domain so that you will be able to adapt to new technologies easily.

In addition to the above, you will also be gaining hands on expertise in

Version Control [GIT]
HTML/CSS
Industry Best Practices for software development

Bridging the Gap: Integrating Microsoft Copilot with Zendesk Using Sunshine Conversations

Hari Krishnan — Fri, 07 Jun 2024 08:15:28 +0000

Introduction

In today's digital landscape, seamless integration between various tools is essential for efficient operations and enhanced user experience. Recently, we embarked on a journey to integrate Microsoft Copilot with Zendesk, despite the lack of direct connectivity. This blog post details our collaborative effort with Zendesk and Microsoft, leveraging Sunshine Conversations to achieve this integration.

The Challenge

Microsoft Copilot is a powerful tool for automating tasks and enhancing productivity. However, it does not have native connectivity to Zendesk, a popular customer service platform. This posed a significant challenge, as no existing solutions or community efforts had addressed this gap.

The Solution

To overcome this challenge, we worked closely with both Zendesk and Microsoft. Our solution involved leveraging Sunshine Conversations, a powerful API for integrating various messaging channels. The process was as follows:

Sequence of API Calls: When an agent interaction is initiated, we trigger a series of API calls – createUser, createConversation, and postMessage. These calls create a ticket in Zendesk and pass all basic information, including the conversation transcript.

**Webhook Creation: **We developed a webhook using JavaScript/TypeScript and deployed it on a serverless solution. This webhook is responsible for sending and receiving messages between Copilot and Zendesk via Sunshine Conversations.

Event Handling: The webhook listens for the ticket:closed event from Zendesk. Upon receiving this event, the webhook ends the conversation on our side, ensuring a seamless closure.

Step-by-Step Implementation

Step 1: Sequence of API Calls

When a user requests to talk to an agent, the following API calls are made:

createUser: This API call creates a new user in the Sunshine Conversations platform.
createConversation: This call initiates a new conversation for the user.
postMessage: Finally, this API sends the initial message from Copilot to the newly created conversation.

This sequence ensures that all relevant information is captured and a ticket is created in Zendesk.

Step 2: Webhook for Message Handling

We developed a webhook that handles communication between Copilot and Zendesk. The webhook performs the following tasks:

Sends messages from Copilot to Zendesk via Sunshine Conversations.
Receives responses from Zendesk and relays them back to Copilot.
The webhook is deployed on a serverless platform, ensuring scalability and reliability.

Step 3: Event Handling

The webhook listens for the ticket:closed event from Zendesk. When this event is detected, the webhook performs necessary actions to end the conversation on our side, maintaining consistency and ensuring a smooth user experience.

Conclusion

By collaborating with Zendesk and Microsoft, and leveraging Sunshine Conversations, we successfully bridged the gap between Microsoft Copilot and Zendesk. This integration enables seamless communication and enhances the overall efficiency of our customer service operations.

Our journey demonstrates the power of collaborative efforts and innovative solutions in overcoming technical challenges. We hope this blog post provides valuable insights and inspiration for your own integration projects.

Open API & Client-SDK [Best Practices]

Hari Krishnan — Wed, 08 Nov 2023 06:29:49 +0000

Introduction

In today’s interconnected digital landscape, APIs (Application Programming Interfaces) play a crucial role in enabling seamless communication and integration between different software systems. With the increasing complexity of modern applications, having a well-defined and standardized API contract becomes essential. In this blog post, we will explore the significance of Open API contracts and how the generation of client SDKs using Open API generators can benefit both front-end developers and consumers. Additionally, we will highlight the drawbacks of not having this practice in place.

1. Understanding Open API Contracts

Open API contracts, often defined using the OpenAPI Specification (formerly Swagger), serve as a formal definition of an API. They outline the available endpoints, request/response payloads, authentication methods, error codes, and other important details. Open API contracts provide a clear and structured way to document and communicate the capabilities of an API, making it easier for developers to understand and consume it.

2. Advantages for Front-end Developers

a. Improved Developer Experience : Open API contracts provide a single source of truth for front-end developers. They can easily refer to the contract to understand the API’s capabilities without digging through extensive documentation or consulting back-end developers.

b. Efficient Collaboration : With Open API contracts, front-end and back-end teams can collaborate more effectively. The contract serves as a shared understanding, enabling developers from different teams to work in parallel, reducing dependencies, and accelerating development cycles.

c. Code Generation : Open API generators can automatically generate client SDKs based on the contract. This eliminates the need for manual coding of API integrations, saving time and effort. Generated SDKs provide ready-to-use client libraries, reducing the chances of errors and ensuring consistent API consumption across multiple platforms.

d. Rapid Iteration : Open API contracts allow for rapid iteration and evolution of APIs. By defining the contract first, changes and additions to the API can be made more easily without breaking existing integrations. Front-end developers can adapt their codebase to new API versions by regenerating the SDK, streamlining the update process.

3. Benefits for Consumers

a. Easy Integration : Open API contracts empower consumers of an API to quickly understand its capabilities, reducing the learning curve. With generated client SDKs, they can integrate the API into their applications with minimal effort, enabling faster time-to-market.

b. Reduced Errors : Client SDKs generated from Open API contracts ensure adherence to the defined API structure. This reduces the risk of manual errors while making API calls, improving the overall reliability and quality of integrations.

c. Consistent Experience : When multiple consumers use the same generated SDK, the API integration experience becomes consistent across different applications. This consistency enhances usability and reduces confusion for end-users interacting with those applications.

4. Drawbacks of Not Having Open API Contracts and SDK Generation

a. Increased Development Complexity : Without a standardized API contract, front-end developers have to rely on fragmented and possibly outdated documentation. This increases the complexity of understanding and integrating with the API, leading to longer development cycles and potential errors.

b. Inconsistent Implementations : In the absence of an Open API contract, different front-end developers might interpret the API’s behavior differently. This can result in inconsistent implementations, making it harder to maintain and update the API over time.

c. Manual Integration Efforts : Without client SDK generation, front-end developers have to write custom code to integrate with each API. This manual approach is time-consuming, error-prone, and increases the chances of inconsistencies across different parts of an application.

d. Compatibility Issues : In the absence of a contract-driven approach, backward-incompatible changes to an API can break existing integrations, leading to disruptions and potential customer dissatisfaction.

Sensible Defaults [Cloud Infrastructure]

Hari Krishnan — Wed, 08 Nov 2023 06:20:55 +0000

In any data-focused business, managing cloud operations is crucial for the effective handling of your digital infrastructure. Drawing from my previous experience in software development, I've learned that defaults play a fundamental role in establishing, deploying, and overseeing your cloud resources. They ensure that everything is secure, dependable, and in line with your business objectives.

Sensible Defaults in Cloud Operations:

Automated Deployments

Our approach revolves around continuous integration and deployment (CI/CD), using strong pipelines that automate software releases.

This ensures your applications are deployed consistently and with minimal risk across different environments.

Monitoring and Alerting

We implement monitoring capabilities using tools, allowing us to capture and analyze metrics in real time. Our predefined thresholds trigger alerts, enabling us to address potential issues before they impact your operations.

Incident Management

We should always have an incident response plan documented, grounded in industry best practices, which streamlines communication and coordination during critical moments. With predefined roles and escalation paths, we address incidents, minimizing disruption and ensuring a speedy recovery.

Logs as Insights

We should recommend centralizing logs in a dedicated platform, enabling us to gain insights into your system’s behavior, diagnose anomalies, and optimize performance. Our log analysis helps discover patterns and enhances your infrastructure’s reliability.

Security

Security should be our top priority. Our defaults include regular security assessments, vulnerability scans, and policy enforcement to safeguard your data and applications from evolving threats.

Scaling

Every team should continuously monitor resource utilization, ensuring your infrastructure scales efficiently based on demand. Our capacity planning is rooted in performance metrics, ensuring a speedy user experience during peak times.

We should often provision the infra to auto scale during peak times and spin down when there is very less demand.

Software Updates

We should manage software updates and patches rigorously, ensuring that your systems remain protected and resilient against emerging vulnerabilities. Infra is provisioned in such a way that stakeholders are notified about vulnerabilities and in times we also auto upgrade the patches for sure cases where we know the system won’t break due to transitive dependencies.

Compliance

We must stay aligned with industry regulations and standards, ensuring that your cloud operations remain compliant and well-governed. We always take into consideration the security and governance policies of the client and their domain while provisioning the infrastructure.

Documentation

We should document every step of our process, providing you with clear insights into your infrastructure. This helps your team with the knowledge needed to manage and optimize your resources effectively.

Flutter in-app purchases setup

Hari Krishnan — Fri, 24 Jun 2022 21:21:43 +0000

In app purchases are those which you make in any android or iOS store. It might be directly the app itself, or advanced features in the app or just game coins in case of a game app.

There are three main types of in-app purchases.

1. Consumables

Consumables are products that are depleted and they can be purchased multiple times. Simple example would be coins in a game.

2. Non-consumable

They are one time purchase from the app provider and it never needs to purchased again. Like a premium feature in some app you are using.

3. Subscription

This type gives the user access to the purchase for a given amount of time. Then after it expires you have to purchase it again or you can make recurring payments to the same.

Setup process has some differences for android and iOS.

Initial setup for iOS

For iOS you need to have an apple developer account
You need to have a release of your iOS app in AppStore connect. It doesn't need to be on the public app store, just on the app store connect for use with things like test flight. Refer https://docs.flutter.dev/deployment/ios
In app store connect, go to My Apps -> Your app -> Features (In-app purchases)
It will prompt for the in-app purchase type, for now you can try creating a consumable in-app purchase. Keep a note of the product id since this has to be the same in google play store also.

Check the availability check box to make the in app purchase available in the app store.

Under price schedule, you have to select the price category for the in app purchase. Feel free to select your own country and respective currency. Start date is the date when that specific price will take effect in the app store for that purchase.

Select the tax category in which your app lies, for example if your app is a game app, try selecting the Games category and for each purchase the actual tax for that category would be chargeable
After creating the in-app purchase, you have to open your project in XCode and enable the in-app purchases capability. Under signing and capabilities create the in app purchase capability
When you are again building, make sure you add the in app purchase capability to the provisioning profile you are using to sign your app

Initial setup for Android

For Android, you should be having a google play developer account. You also should have a signed release apk uploaded in google play. Refer https://docs.flutter.dev/deployment/android
Once you create a singed apk, you will need to atleast create an alpha release for your app in google play. This will allow you to test the google play billing api locally in your test device
Google billing api won't work if you have not created a release for your app. You also want to add the email address of the tester of that app.
Go to Monetize -> Products -> In-app products. You will be prompted to set up a google play merchant's account. For that you need to create a payment profile which can be one of google cloud's billing account or you can create one.
So here you won't be prompted for the purchase type as it did in iOS. You have to handle the type of purchase from code.
While creating the unit in the play store, give the name, description, status to active, price and the same product id which you used in iOS while creating the in-app purchase.

Plugin for implementing in app purchase

https://pub.dev/packages/in_app_purchase

Typescript - Advanced Types

Hari Krishnan — Wed, 15 Jun 2022 02:34:25 +0000

1. Intersection types

Intersection types allow us to combine multiple types into one type. In the below code consider we combine the left and the right type to get the combined Intersection Type.

type LeftType = {
  id: number;
  left: string;
};

type RightType = {
  id: number;
  right: string;
};

type IntersectionType = LeftType & RightType;

const joinedValue: IntersectionType = {
  id: 1,
  left: "left value",
  right: "right value",
};

This intersection can also be achieved using interfaces. With the above code, left and right can be separate interfaces and we can have an intersection interface which implements both the interfaces.

2. Type Guards (typeof, in , instanceof)

typeof

Consider we have a union type which holds two other types. And also consider you need to add or concatenate based on whether it is a number or string, how to identify in runtime whether the incoming value is a string or a number ?

type UnionType = string | number;

function add(a: UnionType, b: UnionType) {
  return a + b; 
  // error: Operator '+' cannot be applied to types 'UnionType' and 'UnionType'.
}

Typescript cannot add or concatenate because it cannot infer the exact type it received in the add method. Type guards can be used here to check the type in runtime and add or concatenate the values accordingly. So the method will be modified as

function add(a: UnionType, b: UnionType) {
  if (typeof a === "string" || typeof b === "string") {
    return a.toString() + b.toString();
  }
  return a + b;
}

This keyword is provided by javascript itself. The in keyword can be used in cases where we have to check if an object has a property we are looking for in it.

Consider the same example above where we had two types the left and the right. Now we want to have a common print method which will print the information based on the left or right type we pass as input to the method.

We can find from the above image that typescript cannot identify the left and right properties as they can vary based on the input given to the method. In this case we can make use of the in keyword to check if the properties are present, and then typescript will be satisfied to ensure that the properties are present and it will print them.

type LeftType = {
  id: number;
  left: string;
};

type RightType = {
  id: number;
  right: string;
};

function printTypeInformation(typeInfo: LeftType | RightType) {
  console.log("id : " + typeInfo.id);
  if ("left" in typeInfo) {
    console.log(typeInfo.left);
  }
  if ("right" in typeInfo) {
    console.log(typeInfo.right);
  }
}

instance of

Consider we have two classes one as the Car and the other as the Truck. Both the classes will have a common method to drive and only the truck will have the loadGoods method.

We also have a common type called Vehicle which contains the union of Car and Truck.

Consider the case we want to have a method called useVehicle and we want to make use of all the methods available in the vehicle instance.

Now here also there is a situation where we cannot make use of the loadGoods method as it cannot be inferred by typescript. In this case we can make use of the instanceof to check in runtime and access their methods accordingly.

class Car {
  drive() {
    console.log("Driving a car");
  }
}

class Truck {
  drive() {
    console.log("Driving a truck");
  }

  loadGoods(amount: number) {
    console.log("Loading cargo : " + amount);
  }
}

type Vehicle = Car | Truck;

const car = new Car();
const truck = new Truck();

function useVehicle(vehicle: Vehicle) {
  vehicle.drive();
  if (vehicle instanceof Truck) {
    vehicle.loadGoods(100);
  }
}

useVehicle(car);
useVehicle(truck);

An important point to note here instead of classes we cannot apply the same for interfaces, because only classes are compiled to something that javascript can understand but javascript does not know what an interface is.

3. Discriminated Unions

A special type of type guard. It is a pattern which we can use when working with union types that makes implementing type guards easier.

There are disadvantages in using the in keyword as well as the instance of keyword. When using the in keyword we might do a typo in specifying the property name and in case of the instanceof keyword, we cannot apply the same for interfaces.

So discriminated union comes handy with addressing all the disadvantages. It is a suggested pattern to follow when we want identify types in runtime.

We simply add a type which would denote the type in the form of a string. And we use switch statement over this type to identify the type in runtime.

interface Car {
  type: "Car";
  carSpeed: number;
}

interface Truck {
  type: "Truck";
  truckSpeed: number;
}

type Vehicle = Car | Truck;

function printVehicleSpeed(vehicle: Vehicle) {
  let speed;
  switch (vehicle.type) {
    case "Car":
      speed = vehicle.carSpeed;
      break;
    case "Truck":
      speed = vehicle.truckSpeed;
  }
  console.log("vehicle speed : " + speed);
}

How can discriminated unions help in avoiding typos ?

When any union type property is used in a switch statement, as in the above image, typescript intelligently identifies the type and shows suggestions.

4. Utility Types

Typescript provides utilities that help to manipulate types easily. In order to use them, you need to pass into the <> the type you want to transform.

Partial<T>

It helps in making all the properties of a particular type optional. It will add a ? mark next to every field.

// initially all props in the below type are required by default
type User = {
  id: number;
  firstName: string;
  lastName: string;
};

const printUser = (user: Partial<User>) => {
  console.log(user);
};

printUser({ id: 1 });

printUser({ firstName: "first name", lastName: "last name" });

Since the printUser method received only a partial user type, we need not specify all the props.

Required<T>

It is opposite to the Partial utility, it makes all props of a particular types mandatory.

// lastName property is optional in the below User type
type User = {
  id: number;
  firstName: string;
  lastName?: string;
};

const printUser = (user: Required<User>) => {
  console.log(user);
};

printUser({ id: 1, firstName: "first name", lastName: "last name" });

Since the printUser method expects the User type with Required utility, you have to always provide all the props in the User Type.

Readonly<T>

The readonly utility will help in transforming all properties of type T, in order to make them not reassignable with a new value.

type User = {
  id: number;
  firstName: string;
  lastName: string;
};

const printUser = (user: Readonly<User>) => {
  console.log(user);
  // user.id = 2;
  // cannot assign value to any prop in user, as it is readonly
};

printUser({ id: 1, firstName: "first name", lastName: "last name" });

Here the print user method receives only a readonly type and inside the method, no property of the user type can be reassigned with a value.

Confluent - MQTT <-> KAFKA

Hari Krishnan — Sun, 22 May 2022 09:29:23 +0000

Spark structured streaming has some limitations on listening to MQTT data sources in its latest version, though there are some extensions available, they support only Spark version 2.0.

Supported Spark Streaming sources

Flume
Kafka
file sources
TCP/IP port
Kinesis
Twitter

For people looking for what MQTT is ?
https://mqtt.org/

So to make mqtt data available for spark to stream, we have to connect MQTT with Kafka, there are many tools available to achieve this. One such is https://www.confluent.io/

Follow the below steps to bring your MQTT device data to a Kafka queue.

INSTALL CONFLUENT

    -> install confluent in local machine (https://www.confluent.io/installation/)
    -> set the path for confluent cli

CHECK FOR THE STATUS OF SERVICES

    confluent local services kafka status

START KAFKA USING CONFLUENT CLI

    confluent local services kafka start

AFTER STARTING KAFKA, START THE KAFKA CONNECT SERVICE

    confluent local services connect start

INSTALL MOSQUITTO

We would need mosquitto to run an MQTT broker which will expose topics on which we can publish and subscribe data.
https://mosquitto.org/

    brew instal mosquitto

START MOSQUITTO BROKER LOCALLY

    brew services start mosquitto

CHECK IF WE CAN PUBLISH TO THE MOSQUITTO TOPIC

    mosquitto_pub -h localhost -p 1883 -t temperature -m "sample-msg-1"

*SUBSCRIBE TO THE TOPIC AND TRY SENDING MESSAGES *

    mosquitto_sub -h localhost -p 1883 -t temperature

INSTALL THE MQTT CONNECTOR PLUGIN

      confluent-hub install confluentinc/kafka-connect-mqtt:latest

RESTART AND START THE CONNECT SERVICE

      confluent local services connect stop && confluent local services connect start

CHECK IF MQTT PLUGIN IS INSTALLED USING THE BELOW COMMAND

      curl -s "http://localhost:8083/connector-plugins"

CONNECT KAFKA WITH MQTT BROKER USING THE BELOW COMMAND

curl -s -X POST -H 'Content-Type: application/json' http://localhost:8083/connectors -d '{
    "name" : "mqtt-source",
"config" : {
    "connector.class" : "io.confluent.connect.mqtt.MqttSourceConnector",
    "tasks.max" : "1",
    "mqtt.server.uri" : "tcp://127.0.0.1:1883",
    "mqtt.topics" : "temperature",
    "kafka.topic" : "mqtt.temperature",
    "confluent.topic.bootstrap.servers": "localhost:9092",
    "confluent.topic.replication.factor": "1",
    "confluent.license":""
    }
}'

CHECK IF THE CONNECTOR HAS A STATUS : RUNNING

      curl -s "http://localhost:8083/connectors"
      curl -s "http://localhost:8083/connectors/mqtt-source/status"

*CREATE A KAFKA TOPIC *

    kafka-topics --create --bootstrap-server localhost:9092 --replication-factor 1 --partitions 1 --topic mqtt.temperature

FINALLY TEST BY CREATING TWO CONSUMERS

   mosquitto_sub -h localhost -p 1883 -t temperature
   kafka-console-consumer --bootstrap-server localhost:9092 --topic mqtt.temperature --property print.key=true --from-beginning

Now if we publish the data to the mqtt topic, both the consumers created above will receive the data pushed in the mqtt topic.