DEV Community: Vivek Siva

Azure PaaS vs Iaas

Vivek Siva — Thu, 03 Aug 2023 16:27:37 +0000

Azure Monitor

Vivek Siva — Thu, 03 Aug 2023 16:05:44 +0000

Kinesis 101

Vivek Siva — Tue, 05 Oct 2021 15:52:17 +0000

example -

AppSync vs API Gateway

Vivek Siva — Tue, 05 Oct 2021 15:27:05 +0000

Handling Multiple Lambda functions

Vivek Siva — Tue, 05 Oct 2021 15:12:35 +0000

Don't do this -

Because it will
Cause these Problems -

Simple Solution -

How your architecture should look -

High Level -

SQS Differences

Vivek Siva — Tue, 05 Oct 2021 15:06:47 +0000

IAM best docs

Vivek Siva — Mon, 04 Oct 2021 16:15:19 +0000

Sites:

https://start.jcolemorrison.com/aws-iam-policies-in-a-nutshell/

https://forrestbrazeal.com/2019/02/18/cloud-irregular-iam-is-the-real-cloud-lock-in/

Codes:
https://github.com/iann0036/iamlive

developing post....

AWS Associate Architect Exam - Pointers

Vivek Siva — Mon, 23 Aug 2021 14:15:31 +0000

SO I recently cleared my AWS associate architect exam (2nd time) and here are the list of services / types that normally comes in the exam!

Type of questions

Most of the questions are based on

"when to use what" - which services are suited
"what's the more secure way"
"what's the most cheaper way"
"which of these requires less operational overhead"

Topics

s3 + Cloud front and OAI
S3 different use cases (when to use Standard, Glacier/ vault etc)
SQS + RDS / different queues and when to use FIFO vs standard
VPC peering and endpoints
ec2 auto scaling related (scheduled scaling/ cool down periods)
Redshift availability/durability / multi region
DynamoDB / Dax / DynamoDB streams
iam roles/policies and types (lambda to s3/ instance profile roles etc /cross account roles)
storage gateway (volumes/cached)
data sync vs s3 transfer acceleration
Kinesis data streams/ analytics/firehose
s3 select vs glacier select
RDS related (when to use aurora - latency/ read replicas vs Multi AZ )
Monitoring related (cloudtrail vs cloudwatch vs trusted advisor)
VPC scenarios (how to create networking). It needs the practical knowledge of setting up access to the Internet for the EC2 instances, for example from Igor (comments)

Aws compute inthirtyseconds :)

Vivek Siva — Wed, 20 Jan 2021 14:56:09 +0000

EC2 instances- Elastic Compute Cloud, this is essentially a virtual machine

In a typical data center, you use a hypervisor like vmware to create virutal machines on top of a physical server and you have a level of liberty to choose your own configuration like vCPU (virtual CPU) and RAM, in cloud , like AWS, its all pre configured! and there comes the AWS Instance families

Instance families :

General Purpose - ATM (a1, t2-t3 families with cpu burst, m5(intel)-m5a (amd) - m6g (graviton)

Compute: c5 -intel, c5a -amd, c6g graviton

Memory : R5, x1, z1

Accelerated Computing -: Gx, Px, INF1, F

Storage: Ix, Dx, Hx

fixed vs burstable:
Cx,Rx, Mx are fixed, static unit based ec2
Tx - CPU burstable - governed by CPU Credits. ex t3 nano has 2vpu and 6 cpu units burstable

Once you spin up server using any of those, it will be in any one of the below states

Instance state

Pending

Running

Stopping--> Stopped

Shutting-down --> Terminated

Started

aws serverless services

Vivek Siva — Thu, 14 Jan 2021 16:09:41 +0000

AWS Lambda automatically runs your code on highly available, fault-tolerant infrastructure spread across multiple Availability Zones in a single region without requiring you to provision or manage servers

AWS Fargate — a serverless compute engine for containers

Amazon DynamoDB — a fast and flexible NoSQL database

Amazon Aurora Serverless — a MySQL compatible relational database

Amazon API Gateway — a service to create, publish, monitor and secure APIs

Amazon S3 — a secure, durable and highly scalable object storage

Amazon Elastic File System — a simple, scalable, elastic file storage

Amazon SNS — a fully managed pub/sub messaging service

Amazon SQS — a fully managed message queuing service

Kafka 101

Vivek Siva — Fri, 08 Jan 2021 18:46:53 +0000

So what is Kafka?

It is a distributed event streaming platform

ok so what does it do :) ?

An event is a thing that happens. Any action happening in a computer or a service will result in a event, its usually getting stored in logs in a traditional system. Now this can be a file based log file or getting saved into databases.

But these are old school techniques and doesn't scale nicely, ever seen a peta byte scale database? so internet scale requires, new tech and Enter Kafka, a distributed event streaming platform, which is designed keeping all the scaling problems in mind!

To under stand more, lets take an example.

Example

lets imagine that you are taking an uber ride, what usually happens ?

1) You open the app and look for near by rides and this kicks off a series of internet calls. Between your app and the uber servers a lot of api calls were initiated, ex: the user app api, which will first log an event , stating that you have opened the app with its date and time, location etc. (yes! everything that you do in internet , every moment / event is tracked, welcome to internet ) and then your login which is another event..

2) Then you start seeing cars nearby , which involves maps api, a driver api, and an algorithm that calculates the distance between you and your driver(s) and then a lot of different sets of permutations combinations like which uber vehicle type you are requesting, is it peak pricing, are you looking for a shared pool, how many users are nearby etc. , each and every one of these actions will generate events that gets stored somewhere (hint : Kafka)

3) Then , you get matched to a driver and finally get a ride and driver will reach you in x mins, and once you get in your car, another api will determine the map traffic and your driver's speed for an arrival estimate again second by second events are getting generated.

4) finally your ride gets completed and then various api's like payment , review api's gets invoked and the ride comes to an end , by this time your ride alone would have created more than 100 events (barring the second by second map api calls)

but, why cant we use rdbms databases?

Now lets "assume" that in the united states 10000 people are booking a uber ride in any minute and each ride will create 100 events, and estimating every ride will take 10 mins.

So for the next ten minutes you will have approximately 10 million events generated and if we are thinking of storing it on a database (a rdbms) then good luck :) .

even though , theoretically this can still be stored in a database , the constant growth of these events will result in a place where you cant scale the rdbms anymore!

Currently , uber handles trillion such events .Uber processes trillions of messages and multiple petabytes of data per day. Imagine putting these on a database :D

So What's the solution

So how do we solve this? enter Kafka, with its topics and producers and consumers. and what are those?

1) a Kafka producer is something that shares the events, here the rider, driver etc.

2) the Kafka consumer is something that consumes these rider driver events and performs computations, like your cost of ride, your estimated time etc.!

check out this uber architecture and try to map the contents so far..

Uber has one of the largest deployments of Apache Kafka in the world.

Technical details

Topics

Events are organized and durably stored in topics.
Very simplified, a topic is similar to a folder in a filesystem, and the events are the files in that folder. An example topic name could be "payments". Topics in Kafka are always multi-producer and multi-consumer, it can have n number of producers writing and reading at same time..

Topics are partitioned, meaning a topic is spread over a number of "buckets" located on different Kafka brokers.

And then there are many API's

The Admin API to manage and inspect topics, brokers, and other Kafka objects.
The Producer API to publish (write) a stream of events to one or more Kafka topics.
The Consumer API to subscribe to (read) one or more topics and to process the stream of events produced to them.
The Kafka Streams API to implement stream processing applications and microservices. It provides higher-level functions to process event streams, including transformations, stateful operations like aggregations and joins, windowing, processing based on event-time, and more. Input is read from one or more topics in order to generate output to one or more topics, effectively transforming the input streams to output streams.
The Kafka Connect API to build and run reusable data import/export connectors that consume (read) or produce (write) streams of events from and to external systems and applications so they can integrate with Kafka. For example, a connector to a relational database like PostgreSQL might capture every change to a set of tables. However, in practice, you typically don't need to implement your own connectors because the Kafka community already provides hundreds of ready-to-use connectors.

More @ https://kafka.apache.org/intro

what is "whatis"

Vivek Siva — Thu, 05 Nov 2020 15:52:16 +0000

Ever wondered whether there is a quick tool in Linux / UNIX to tell what other commands are for?

enter whatis

just go to shell and use whatis with whatever command you think of
ex:
whatis ls
ls (1) — list directory contents

whatis w
w (1) — Show who is logged on and what they are doing

whatis whatis
whatis (1) — search the whatis database for complete words