DEV Community: Rashmiranjan Sahoo

Azure storage service - learning day 4

Rashmiranjan Sahoo — Sat, 21 Oct 2023 04:26:32 +0000

What is life cycle Management?

Lifecycle Management offers a rich rule-based policy which we can use to transition our data to the best access tier and to delete data at the end of its life cycle.

Security in azure

security is most important aspect in cloud.

Security mapped for all the services

Authentication

Storage account keys
Shared access signature (SAS)
Azure Active Directory (Azure AD)

Access Control

-Role based access control (RBAC)
-Access control list (ACL)

Network access

Firewall and virtual network

Data Protection

Data encryption in transit
Data encryption at rest

Difference between Iam and (RBAC, ACL)

In IAM we give the access on container level in blob and data lake.

But in RBAC, ACL we give the access on folder level, and it is only available for data lake gen2.

By click on container go to the directory structure,
which is HFS hierarchy file structure,
Right click on any directory.

select manage ACL
Add Principal Here principal means it can be a user, group or it can be an application service.

Data redundancy for storage

Redundancy mainly used when it comes to data lake for high availability and disaster recovery.

Types

LRS (locally redundant storage)

Three copies of data which is maintained within the same primary data center.

ZRS (Zone Redundant Storage)

-Three copies of your data replicated synchronously to 3 Azure availability zones in a primary region.

Zones are different physical locations on different data centers.

GRS (Geo-redundant Storage)
-This allows our data to be stored in different geographic areas of the country or world.

we get 3 copies within a primary region, but it goes one step further and places 3 additional asynchronous copies in another region.

RA-GRS (Read access Geo redundant storage)

This is Grs but adds a read only element that allows us to have read access for things like reporting.

GZRS (Geo zone redundant storage)

Copy our data synchronously over 3 primary region azure availability zones using ZRS.

Then asynchronously copies our data to a single physical location within the secondary region.

RAGZRS (Read Access Geo Zone redundant storage)

It adds a layer of readability to our secondaries.

Monitoring service

Azure monitoring service helps us to maximize the availability and performance of our applications and services.

This information helps us to understand how our applications are performing and proactively identifies issues affecting them and the resources they depend on.

The components of monitoring services are:

alerts
metrics
diagnostics
logs Analytics

Alerts

Alerts in azure monitor proactively notify us of critical conditions and potentially attempt to take corrective actions.

Creating Azure data lake gen2- learning day 3

Rashmiranjan Sahoo — Thu, 19 Oct 2023 16:10:08 +0000

Disclaimer: I am writing this for revisiting all the concepts in azure data engineering and clearing my fundamentals. If you find this article helpful to you, I would love it.

what is azure data lake gen2?

It is a data lake solution provided by azure.
It is a combination of blob storage and data lake gen1 to provide high scalable and secure data lake, for big data storage and analytics.

How to create a data lake?

To create a data lake in azure
click on create a storage account
fill all details
click on next
In advanced there is a section
- Data lake storage gen2
- Enable hierarchical namespace ☑
- click on this checkbox
By clicking this check box our storage account converts into azure data lake gen2 with hierarchical name space or directory which is not available in simple blob storage.
In the background , there is a data lake configured for system and the disk space and cluster enable for us.

when we enter into our storage account

click on container to create container
- give name
- create
when we click on the container that we created
we can see different options available on the top bar that is add directory a disk like structure

What is hierarchical name space?

Hierarchical namespace is based on Linux file storage and Hadoop file system i.e., HDFS file system.

It organizes files into a hierarchy of directories for efficient data access.

We can store everything on data lake gen2 as we know it can store 3 types of data classification data i.e., structured, semi-structured, Unstructured.
Data can be web-server log data, relational data, streamed data, etc.

We can process this data by synapse analytics, data bricks, stream analytics.

Access Tiers

An access tier refers to a data storage option in cloud-based storage services that helps us to manage the cost and performance of our data based on how frequently we access it.

Types of Access tiers:

Hot
It is optimized for storing data that is frequently accessed.
It offers low latency and suitable where data frequently read and written.
Cool
Optimized for storing data that is infrequently accessed and stored for at least 30 days.
There is an early deletion fee is charged for deletion.

What is early deletion fee?
If we feel that we don't need this data for 30 days on 15th day we want to delete.

There we will pay some fee.

This fee is called early deletion fee.

Archive Access Tier

Optimized for storing data that is rarely accessed and stored for at least 180 days.
It offers the lowest storage costs but can have the highest data retrieval times.

This tier is suitable for long-term storage, such as compliance data, backup archives, and historical records.

what is Rehydration?

To read data from archive storage we must first change the tier to hot or cool.

This process is called rehydration and can take hours to complete.

Standard priority:
The rehydration request will be processed in the order it was received and may take up to 15 hours.
High priority:
The rehydration request will be prioritized over standard requests and may finish under 1 hour for files under 10gb in size.

next article will be about all features in azure data lake gen2 like life cycle policy, security, etc.

Azure Storage Account service -learning day 2

Rashmiranjan Sahoo — Wed, 18 Oct 2023 15:59:01 +0000

Disclaimer: This is the note I am publishing as blogs for me to revisit all topics and clear my fundamentals. I would love if this article you found as helpful.

Azure Storage account is a Microsoft managed cloud service that provides storage that is highly available, secure, durable, scalable and redundant.

Within azure there are

2 types of storage accounts
4 types of storage
4 levels of data redundancy
3 tires for storage files

4 types of storage

Azure blobs
Azure files
Azure Queues
Azure Tables

Azure blob is important so we will explore it later.

Azure file Storage

Azure files is a shared network file storage service that provides administrator a way to access native smb file shared in the cloud.

Using this smb protocol we can integrate with our local machine to cloud azure.

Azure Queue Storage

Azure Queue Storage is a service that allows users to store high volumes of messages, process them asynchronously and consume them when needed.

Here we put a retention day of 7 days or a month as per required.

After that day the message will be disappear.

Azure table storage

Don't confuse with the name like table that don't mean structured data here.

Azure table storage is a scalable, no-SQL, key-value data storage system that can be used to store large amounts of data in the cloud.

This storage offering has a schema less design and each table has rows, that are composed of key value pairs.

In No-SQL there is a document dB where data stores as Json document file.

similar ex: Mongo dB

Azure Blob Storage

Extra importance for this storage service

Azure blob storage is Microsoft azure service for storing binary large object or blobs which are typically composed of unstructured data such as text, images and videos along with their metadata.

Blobs are stored in directory like structure called Containers.

Note:
All the 3 types of data classification we can store into this one place which is blob.

Those classifications are:

structured
semi-structured
unstructured

Benefits of blobs

optimized for storing massive amounts of unstructured data
Text or binary data
General purpose object storage
cost efficient
Provide multiple access tiers.

Note:
we can do all the tasks using this single blob storage service.

Types of storage account
At the time of creation of storage account there will be option

Standard
Premium

These two types are called types of storage account.

Standard: as from the name we can understand that it has les s I/O use case with low latency

Premium: This has a very low latency.

very good data movement from point A to point B.
High I/O use case.

In the time of creation, we have to very careful about the name, name of storage account.

Because it is going to create a URL through which we connect this account from other services or accounts to perform any operation.

Next article will be about azure data lake gen2.

Azure Storage services - Learning Day 1

Rashmiranjan Sahoo — Tue, 17 Oct 2023 16:20:34 +0000

Hi, here I am sharing my learning for my better practices and storing here so that I can read it whenever I need.

If it helps you to learn or recall the fundamental things, I would be happy.

cloud

The practice of using a network of remote servers hosted on the internet to store, manage and process data, rather than a local server or a personal computer.

In simple terms Hosting something on the internet and you are not managing on your local system.

Mainly 3 types of service provided by cloud.

1. IaaS (Infrastructure as a service)

IaaS provides the foundational infrastructure elements needed for running applications, such as virtual machines, storage, and networking.

Users can access and manage these resources remotely, eliminating the need for physical servers and data centers.

ex: azure, gcp, aws.

2. PaaS (Platform as a service)

PaaS offers a platform that includes not only infrastructure but also tools and services for developing, testing, and deploying applications.

It abstracts much of the underlying infrastructure, allowing developers to focus on writing code rather than managing servers.

ex: Microsoft Azure App Service, Azure Data factory, Data lake, etc.

3. SaaS (Software as a service)

SaaS delivers software applications over the internet on a subscription basis.

Users access the software through a web browser, and it is typically hosted and maintained by a third-party provider.

This model is user-centric, and users do not need to manage the underlying infrastructure or worry about updates and maintenance.

ex: Microsoft 365.

To store the data in azure there are a lot of service available. Anyone can see from the documentation.

But mostly we use,

Azure storage account

Azure SQL

Azure Data Lake

Azure cosmos DB

Why we need this?

To answer this question lets go back into 90's
when everything stored relational database where size of data is low.

At that time, we use rdbms systems for storing data and processing data.

But when we hit 20's there a lot of tech things come into the world.

The world explores different different technologies and slowly starts generating data in lots of amounts.

So, the big data arises here.

Then 3v's come into the place which describes the key characteristics or challenges associated with the BIG data.

Velocity --> 1sec, 1hr
Variety --> structured, semi structured, unstructured
Volume --> 5GB, 10Gb, 30Gb, 1TB

One of these individuals can't be called as Big Data.

By combining these 3

If 5gb,10gb or any large volume of different varieties of data is generating in every sec, hour, or day then we can call this as BIG DATA.

what is data classification (remember this vocabulary)?

1. structured data

These data have some schema like row, column.

Mostly in tabular format.
ex: SQL, csv, spreadsheet

2. semi-structured data

No-SQL, key-value pair, JSON.

3. Unstructured data

media files, office files, text files, log files.

Next article will be about azure storage services.