DEV Community: Beatrice Akaeme

AWS FARGATE OVERVIEW

Beatrice Akaeme — Thu, 28 Dec 2023 02:46:06 +0000

AWS Fargate is a scalable, serverless managed service by Amazon Web services which is a compute engine for Amazon Elastic Container Service (ECS) that can deploy and manage containerized applications in the cloud and ensures it is highly available. What is it mainly used for? To start with, you can run containers with it and you don't even need to create or manage any server or cluster which is good. Since it is a computational engine, as a customer you don't need to worry about managing EC2 instances at all or any infrastructure. So what are you required to do as a customer? You can start by paying for resources per application which you have specifically chosen. You get to design and develop your applications as well as package it in containers. You are free to work on the memory and CPU requirements, IAM policies too. Also works with Amazon Elastic Kubernetes Service (EKS), Amazon Simple Storage Service (S3) and Amazon CloudWatch. Another thing to note is that it is cost effective, you only pay for used resources.

FEATURES
1)Can manages cluster of servers for users as well as the EC2 instances

2)Clusters can be optimized

3)Easy scalability of applications

4)High availability of infrastructure which is needed to run application

5)Integration with ECS

6)You only need to pay for the amount of time the task makes use of the resources

7)AWS Fargate helps you to match the needs of your application to suit the size of the memory of your workloads.

8)Networking support for every Fargate task is which is completed in your virtual private cloud (VPC).

9)Monitoring and Logging: logs for your applications can be sent CloudWatch logs through AWS Fargate.

Components Of AWS Fargate

Amazon ECS: With the API which Amazon ECS provides to interact with AWS Fargate, the use of other services such as Amazon CloudWatch and CloudFormation is allowed, in order to manage and monitor your containers is possible which aids in managing your container as well as monitor it.

Clusters: You can group multiples of services which your tasks needs to run smoothly. AWS Fargate helps to manage your clusters.

Networking: AWS Fargate has networking capacities including Virtual private network (VPC) as well as security groups which it provides for your application containers to encourage communication between resources and containers in your VPC.

Task Definition: This is basically a file with text which gives a clear description of containers in your application including the the specifications for your application. This information is usually stored in a Jason formatted file.

Tasks: After creating the task definitions, the next step should be to initialize the tasks definition. In your cluster, You can decide how many tasks that will run. You can keep it simple and run few or run multiple tasks at the same time.

Services: With ECS service you can run tasks you have created in the ECS cluster which can be single or multiple at the same time.

Container Image: This is a software package created from a docker file that consists of everything needed to run your application

Compute resources: AWS Fargate provides the basic compute resources and scalability needed to run your application container. These include CPU and memory with specific amount which each container needs according to definition of tasks.

Benefits of AWS Fargate:
1)Simplicity: AWS Fargate has a simple infrastructure that can be accessed by any level of expertise. Getting started is easy without the need to worry about provisioning or scaling compute resources, or managing the operating system, networking, and storage configuration. You just need to focus on building and deploying your applications.
Most of the underlying infrastructure, is taken care of by AWS which allows developers to focus solely on applying development.

2)Scalability: When you choose to work with AWS Fargate, you don't need to worry about the scalability of your compute resources or scaling your containers since it will be done for you by AWS Fargate automatically.

3) Saves Money: AWS Fargate is very cost effective, with this service you only pay for what you use saving you money. As a serverless compute resource, it operates on a Pay-As-You-Go model.

4)Serverless: With AWS Fargate you do not need to worry about setting up servers as well as managing it for your containers about customizing and managing the servers; your containers will still run on them.

5) Security: One of the features of AWS Fargate is security. It comes with the service. There is isolation and safe communication between containers. When it also gets combined with the AWS IAM, this helps to make the security your applications even stronger. So, you don't need to worry about the security of your applications.

6)Monitor: It can be combined with Amazon CloudWatch to provide consistent monitoring for your applications.

LIMITATION:

Lack of Control: Since AWS Fargate does the work for us of managing our applications, it means you don't have much control over how your applications is run, whether is cluster sizing, or configurations, your own choice of a particular instance with which your containers will run on. As a result, you cannot customize it to your needs.

Cost: If you have lots of small services that are rarely used, or if your application requires a lot of resources, or even for simple tasks, it can be a lot cheaper to use ECS instead. You have to bring your own EC2, but it still does all the container management for you.

Availability: Fargate pods cannot be deployed to AWS region, AWS Wavelength, or AWS Local Zones.
Fixed regional availability
Fargate service is not available in all regions/zones. Also, far fewer EKS supporting regions support Fargate.

Lack of GPU Support: There is no support for GPU instances on AWS Fargate and so they are not currently available. Developers with GPU-dependent workloads might need to explore other solutions.
As a result of this, workloads that needs GPU acceleration has to be run on EC2 instances.

Portability: Fargate cannot be used by a users if for instance there is a need for them to move their workloads from one cloud provider to another cloud provider.

Not all workloads are supported: Some workloads that do require high-performance computing or specialized hardware are not by AWS Fargate If you have a workload that requires a lot of resources, such users are better off managing their own servers.

Lack of image caching: Image caching is not supported by AWS Fargate and as a result there is longer pod startup times as images are not cached on the node which affects efficiency of work being done.

USE CASES
1)AWS Fargate can be used to scale up light workloads which has high traffic during the day and scale down it down at night when the traffic is low to save money for workloads that requires higher infrastructure.

2)AWS Fargate is a better choice for running microservices. Developers can deploy each containerized microservice and the required resources will be managed automatically.

3)AWS Fargate can be used for batch processing, such as image and video processing, data analysis and ETL tasks. You can tailor your requirements according to the workload involved.

4)Deploying and running web applications without managing server is easy with Fargate.

5)You can Launch your containers without doing the work of to provisioning or managing EC2 instances using AWS Fargate. If you require large workloads of memory and do not want to make much of the work to manage the server or create one, then then Fargate is preferred.

6) AWS Fargate empowers AI the ML workloads, as well as assists in scaling up the development, deployment, and testing of a user's ML models.

AWS IAM POLICIES OVERVIEW

Beatrice Akaeme — Thu, 30 Nov 2023 22:43:00 +0000

AWS Identity and Access Management policies gives users appropriate permissions to resources in their AWS cloud account. I believe AWS services needs the IAM policies for support. These policies helps to keep the AWS services secured. It either gives or denies permissions to resources within a users AWS account. This will ensure the right person gains access to that account at the right time too.

These identities include:

Users; This can be a person with a unique identity and has security credentials such as password or access & secret key. It can also be a service or applications that interacts with resources in AWS environment. Each type of user is given permission to manage use of AWS resources. These resources can only be assigned through permissions or policies.

Group; A set of IAM users that do same thing and share policies and permission together. Each time a permission is assigned to a group with the use of access control policies , all the users in that group are automatically entitled to it. For example, placing a user into an Administrator or developer group will automatically assign the user any permissions given to the Administrator group or developer group. It means users can be moved between different groups and automatically shift permissions as groups change.

Role; These can be used to assign access to AWS resources and AWS Users temporarily rather than sharing credentials. This can give various users ability to take up different permissions for different tasks. As a result of this, passwords are not necessary here. Anyone can just come up and take a role they need for that moment or that particular tasks which is not permanent.

Policies; This is a document that defines different levels of permissions which are attached to users, groups or roles, Each of this cannot use any resource except the policies associated with it in the document says it can.

How Does IAM Work?
This includes the following elements:

A principal is an entity that can perform actions on an AWS resource. A user, a role or an application can be a principal.

Authentication; Is used to confirm who you claim to be each time you try to access the AWs resources.

Authorization; Is an act of verifying if someone can actually access a resource they are requesting to access and then go ahead to authorize and sometimes authenticate the request for the user before it can be granted by AWS. There must be a supporting policies.

Types of IAM policies

Identity-based policies: These are policies that can be attached to either a user, a group or a role stating what they are allowed to do with any of the AWS resources.

Customer managed; These are policies that you as an AWS customer can create and use in your AWS account. These policies are more specific or customized according to your preferences.

Resource based; These are policies that are attached to a particular resource such as Amazon S3 buckets or a VPC endpoint.

Managed policies; These are policies that are independent that can be attached to multiple users groups or roles.

Inline policies; These policies deals with one-to-one relationships and directly inserted into either a user, group or role. This type not always needed, so it is not a great choice.

AWS Networking Overview

Beatrice Akaeme — Tue, 31 Oct 2023 10:44:59 +0000

Amazon Web Services (AWS) provides an extensive set of networking services and features to help developers build, manage, and optimize their cloud infrastructure. AWS networking services are created to offer high availability, security, and scalability. Below is the list of the main AWS networking services and concepts:

Amazon Virtual Private Cloud (VPC): Amazon VPC is the foundational building block of networking in AWS. It allows you to create your own virtual network in the Amazon cloud. It is a safe and isolated virtual networks in the cloud. As a developer you can connect your on-premises resources to AWS infrastructure through this virtual private network thereby launching AWS resources such as EC2 instances, RDS databases, and Lambda functions. You have full control over your VPC, including IP address ranges, subnets, route tables, and network gateways from the day it is created till the day You delete it.

Subnet: A subnet or subnetwork is a segment of a network inside of your virtual private network with its own range of IP addresses in your VPC. You must create subnets in order to launch your EC2 instances. One subnet can be in only one AZ at same time. It is further divided into private subnet and public subnet. A public subnet You can create public and private subnets, and they are associated with Availability Zones (AZs) in a region. Public subnets are typically used for resources that require direct internet access, while private subnets are used for internal resources.

Internet Gateway: This is an attachment of your virtual private cloud to the internet. An Internet Gateway makes communication between your public subnets and the internet possible. The public subnet with public IP address makes use of a route table to direct traffic to the internet gateway. It is very important to create an internet gateway when creating your own VPC to ensure that resources can be accessed from the internet and your instances can communicate with the outside world. You can route internet traffic to your instances so long as you attach internet gateway to your VPC. You can attach internet gateway to one or several subnets in your VPC when it is created.

Virtual Private Network (VPN): This managed service which is offered by AWS creates a safe online connection with AWS Site-to-Site VPN and AWS Client VPN to establish secure connections between their on-premises network and your AWS VPC. It makes your browsing more safe and secure with your VPC encrypting your identity and online traffic thereby preventing your data from being stolen

Direct Connect: AWS Direct Connect is a dedicated network connection and it provides more reliable and consistent connection between the environment of AWS directly through to the environment in which the user is currently in. It is done through a private network connection between AWS and your on-premises data center or office helping to reduce network instability and overcrowding. We have two types such as: 1)Dedicated connections which are connections that is between your data center and AWS. 2)Hosted connections which connections done by an AWS Direct connect delivery partner for you.

Peering: This is a connection between one VPC and another or with several other VPCs. With VPC peering you can connect multiple VPCs within the same or different AWS regions, using private IPv4 or IPv6 addresses allowing resources in different VPCs to communicate with each other as if they were on the same network. This allows traffic to flow between the connected VPCs making it possible for applications or workloads in one VPC to access data or resources in another or to enable file sharing.

Transit Gateway: AWS Transit Gateway is a service that simplifies network connectivity between multiple VPCs enabling them to communicate with each other, allows VPCs to as well connect to onsite networks through VPNs, and or Direct Connect connections. At least with this you don't need to manage multiple VPC connections.

Elastic Load Balancing: Is a cloud service provided by AWS in which traffic from network is distributed uniformly, efficiently and automatically across multiple servers or resources such as EC2 instances, IP addresses, Lambda functions and Containers in either one availability zone or multiple availability zones within an AWS region. This improves availability and fault tolerance and only routes traffic to healthy targets.

Types of Elastic Load Balancer;

1)Classic Load Balancer - Is a traditional load balancer that Operates at both Open system interconnection (OSI) layer 4 and layer 7. It does not support certain features.

2)Application Load Balancer (ALB) – traffic is distributed based on the content of the request at the application layer (HTTP/HTTPs). Operates at OSI layer 7

3)Network Load Balancer (NLB) – Decision is routed at transport layer( OSI layer 4). Good for high-performance applications and can handle millions of requests per second. Recommended for unstable and spiky and workloads

4)Gateway Load Balancer (GLB) – Good for deploying, scaling and managing virtual appliances such as firewalls. Operates at network layer(OSI layer 3)

Route 53: Amazon Route 53 is a highly Scalable and available Domain Name Server from AWS's Cloud platform which developers and organizations can use to route end users to their web applications in reliably and at a low. It routes traffic to resources like Amazon S3, Elastic Load Balancing, or EC2 instances, domain name registration for your website, translates domain names into IP addresses so as to route traffic to your website and can perform health checks.

Security Groups and Network Access Control Lists (NACLs): These are used for controlling inbound and outbound traffic to your AWS resources. Security Groups is a stateful virtual firewall that are associated with EC2 instances, while NACLs act as a stateless firewall at the subnet level.

Elastic IPs: Elastic IP addresses provide static, unchanging public IP address IPv4 address for your cloud computing environment. You can choose to associate and allocate a fixed public IP address to your AWS resources like EC2 instances and NAT gateways. They are particularly useful for ensuring that the public-facing IP address of your instance remains the same after a stop/start.

VPC Flow Logs: These lets you capture and log information about the IP traffic going to and from network interfaces in your VPC as well as how your network resources are actually operating. This is can be very helpful for security monitoring and troubleshooting.

AWS Global Accelerator: AWS Global Accelerator is a networking service offered by Amazon Web Services which uses static IP addresses to route traffic to multiple AWS regions, EC2 instances and Application Load Balancers, improving the availability, security and performance of public applications. This runs across multiple regions.

VPC Endpoints: These enable a highly efficient, secured private connectivity from your VPC to specific and supported AWS services with use of private Link without requiring public IP address at a reduced data transfer costs. This ensures that your data remains within the AWS network, and lowers exposure to the public internet.

Elastic Network Interface (ENI): An ENI is a virtual network interface which you can attach to an EC2 instance in your VPC, authorizing it to communicate with various subnets and security groups. It can attach computers and servers over online without the traditional physical cabling and other hardware.

Basically, ENIs are virtual network cards you can join to your EC2 instances. They are used authorize network connection for your instances.

These are just some of the many networking services offered by AWS to cater to a wide range of use cases, from simple web applications to complex industrial architectures, to guarantee that users can build highly available, secure, and scalable cloud infrastructures.

AMAZON EC2

Beatrice Akaeme — Fri, 29 Sep 2023 20:17:47 +0000

Amazon EC2 means Elastic Compute Cloud: Is one of Amazon's Web based service which can be used by businesses to run applications in the public cloud.
It is a foundational service provided by Amazon Web Services (AWS) that allows users to spin up virtual servers which is called instances, in the cloud. EC2 instances provide scalable compute capacity that can be tailored to various workloads, from simple web hosting to complex data processing. Here are the key aspects of Amazon EC2:

BENEFITS OF AMAZON EC2:

SCALABILITY: Increases capacity automatically up or down within minutes depending on what the client wants and is done through auto scaling. Several instances can be scheduled to run at a particular time or according to the CPU utilization all done at the lowest possible cost. With Elastic Load Balancers (ELB), traffic can be distributed across numerous instances for enhanced availability and scalability.

FLEXIBILITY: With EC2, you are allowed to choose from numerous OS, software packages, instance types and storage.

RELIABLE: Amazon EC2 is highly reliable offering an availability of 99.99% for each Amazon availability region.

SECURITY: A very strong networking and security is provided by Amazon VPC with Amazon EC2 for the compute resources. You can control what should be exposed or not to the public cloud.
As a User you can control access to instances through AWS Identity and Access Management (IAM), security groups, and network access control lists (ACLs).

WELL CONTROLLED: You can totally control your instances, both the root access and even interacting with them the way you do normally with other machines. You are free to stop and restart your instance.

SAVES MONEY: Amazon EC2 is cost effective since you only pay for the compute capacity you consume. It allows you to select plans according to your compute needs thereby saving cost even as you fully utilize resources.
AWS provides a variety of pricing models, including On-Demand, Reserved Instances, and Spot Instances, to optimize costs.

INSTANCE:
Amazon EC2 enables users to create and manage virtual servers in the cloud, known as instances.
EC2 instances can be customized in terms of CPU, memory, storage, and networking to suit a wide range of applications and workloads.
EC2 instances are billed based on various factors, including instance type, region, usage hours, and any additional resources such as EBS volumes or data transfer

INSTANCE TYPES:

Amazon EC2 offers an array of instance types optimized for different use cases, such as compute-optimized, memory-optimized, storage-optimized, and GPU instances.
Customers can choose the instance type that matches their specific requirements the most.

GENERAL PURPOSE: This instance type is good for variety of workloads. It provides you with a balanced computation, memory and networking resources which can be used for numerous applications.

Types;

A1 instance is more suited for web servers and containerized microservices

M5 instance is suited for small-to-midsize databases, data processing tasks, and as a backend server for enterprise applications like SAP or SharePoint.

T3/ T3a instance is used for long-lasting application instances such as websites, web applications, and code repositories.

Use Cases; Used to provide a highly scalable platform for Web servers.
In Mobile development, developers can use it to build and deploy their applications.
Used in Game development delivered to users all over the world in the cloud.

COMPUTE OPTIMIZED: This is useful for compute intensive applications but balanced memory usage. High instances are delivered at a cost-effective prize.
Types;

C5/ C5n instance is for applications like online gaming, scientific modeling, media transcoding – which require raw computing power.

C6/ C6g instance is for highly intensive and advanced applications such as high-performance computing, video encoding, ad serving, and distributed analytics.

Use Cases;
Machine learning applications- Because it can handle heavy workloads
Gaming- Can deliver high quality gaming experience.

MEMORY OPTIMIZED INSTANCE:
This instance type is designed for workloads that require very quick performance to process large amounts of data sets to be processed in memory.

Types;

R6g/ R6gd instance is suitable for high-memory workloads, including relational databases example MySQL and in-memory caching such as Redis.

X1/ X1e instance is used for memory-rich applications, including SAP HANA, and Apache Spark, and high-performance computing

High memory instance is used to run high in-memory databases such
as the SAP HANA database.

R5/ R5a/ R5n instance is good for applications that has a high memory consumption like high-performance databases, real-time Big data analytics, and large in-memory cache applications

STORAGE OPTIMIZED INSTANCE TYPES
These instances are meant for applications requiring high storage such as big data, data warehousing, and log processing with systematic read and write access to huge datasets. Built for the delivery of applications with low latency.

Types;

D2 instance is designed for applications like Hadoop clusters, log clusters, and data warehousing

H1 instance is are most suitable for data-intensive applications such as MapReduce, and for applications that require high efficiency, fast and systematic access to large data volumes.

l3/ l3en instance is used for high-frequency OLTP systems, RDBMS, and caching for in-memory databases like Redis.

AMAZON MACHINE IMAGES (AMIs): These compute instances use hardware accelerators for greater performance in compute-intensive applications.

AMIs are pre-configured templates which includes an operating system and many times, application software.
Users can choose from a broad selection of publicly available AMIs or create custom AMIs according to their specific needs.

INSTANCE LAUNCH and TERMINATION:

Users can comfortably launch instances through the AWS Management Console, CLI, or SDKs.
When these Instances are no longer needed, then it can be terminated which helps to control costs.

STORAGE OPTIONS:

EC2 instances can be attached to different types of storage, which includes Amazon EBS volumes for block storage and Amazon S3 for object storage.
Instance store volumes provide temporary, high-performance storage directly attached to the instance.

NETWORKING:

Users can configure Virtual Private Cloud (VPC) networks to isolate and control network traffic for instances.
Elastic IP addresses allow instances to have static, public-facing IP addresses.

USE CASES:

EC2 is suitable for a wide range of workloads, such as web hosting, application hosting, data processing, machine learning, and scientific computing.

Monitoring and Management:

Amazon CloudWatch provides monitoring and logging capabilities for EC2 instances, helping users track performance and discover issues.

CONCLUSION:

Amazon EC2 forms the core of AWS's compute offerings, providing scalable and flexible compute resources to meet diverse application needs.
By understanding the capabilities and options provided by Amazon EC2, users can effectively provision and manage virtual servers in the cloud, optimizing their computing resources to match specific workloads while maintaining control over security and costs.

DATA STORAGE ON AWS

Beatrice Akaeme — Thu, 31 Aug 2023 03:43:07 +0000

In modern businesses and technological organizations, data storage plays a very important role in making sure organizations reach their full potential in terms of their information assets. With the emergence of digital data, ranging from customer information and transaction records to sensor data and multimedia content, storing data effectively is more critical than ever. Now lets talk about its importance:

"Data Storage in the Modern Business and Technological organizations'
Data storage stands as the foundation upon which organizations build their digital infrastructure in this digital age and data-driven world. The rapid evolution of technology has led to an extraordinary generation of data from various sources and formats. Data is used in following sectors; for e-commerce transactions, social media interactions, for industrial sensors and scientific research, data has become the core of business operations and decision-making.

Data storage serves as the storehouse for every valuable information, providing the means to capture, retain, and access data reliably and efficiently. Businesses use stored data to gain insights into customer behaviors, track operational performance, develop innovative products, and refine strategies for growth.

Types of AWS Data Storage Services

Amazon Web Services (AWS) offers a variety of data storage services to cater to different use cases and requirements. These services can be broadly categorized into four main types:

Object Storage Services

Amazon S3 (Simple Storage Service): Amazon S3 is a large highly scalable object storage service that allows you to store and retrieve large amounts of data, such as images, videos, backups, and logs. It offers different storage classes based on data access patterns and costs.

Components Of Amazon S3:
1.Buckets:A bucket is a high-level cloud-based data storage service container that holds objects (files) in Amazon S3. After creating a bucket in the region of your choice which is closest to you, give it a globally unique name and choose the storage. It acts as a logical unit for organizing and managing objects.
Naming: Bucket names are globally unique across all of Amazon S3.

Objects: Is used for storage on Amazon S3 which can be data files, including documents, photos, and videos. It is made up of data, key and metadata. Object is identified by a unique key.

Key: The key is a unique identifier for each object within a bucket. It represents the full path of the object inside the bucket. It can be further broken down into “prefix”(folders, that is, the pathname) and “object name”. Can be used for organizing, managing and retrieving objects within a bucket.

Object URLs:
Every object in Amazon S3 has a unique URL or web address that allows direct access to a specific bucket or object within the bucket over the internet. The URL typically follows a pattern like https://.s3..amazonaws.com/.

Amazon EBS (Elastic Block Store): Amazon EBS provides block-level local storage volumes that can be attached to and used by Amazon EC2 instances. You can use it to keep data persistently in a file, to host databases, and other types of persistent storage.

Aspects of EBS:
Volume Types: EBS volumes are virtual hard drives attached to EC2 instances, providing durable block storage. They can be attached to and detached from EC2 instances spontaneously thereby enabling data mobility and flexibility.
EBS Volumes can only be attached to a single EC2 instance at a time.

Types of volumes are;
1.General Purpose (SSD): Balances both price and performance for a wide range of workloads, suitable for most of the use cases either small or medium workloads. Workloads can be production and development applications.

2.Provisioned IOPS (SSD): Delivers predictable high performance with low-latency and consistent I/O operations. Use cases can be critical production applications and databases requiring high maximum performance storage.

Throughput Optimized (HDD): Meant for frequently accessed, large storage and throughput workloads. Good for data warehousing and log processing.

3.Cold HDD: Good for infrequently accessed data with a focus on cost-effectiveness as the main criteria.

Features of EBS:
Snapshots: It contains all the information you need to restore your data from EBS volume. You can take snapshots of a volume to capture the Point-in-time backups of the EBS volumes, ensuring data backup, recovery, and volume duplication that you can attach to another instance.

Encryption: Amazon Web Services also offers flawless data encryption for EBS Volumes. When you attach an encrypted EBS Volume to an instance then all the data, including data on the volume, disk I/O and even the snapshots created from that encrypted volume, are all encrypted. EBS volumes can be encrypted to protect data at rest.

Availability and durability
Amazon provides very high available and durable service for EBS at no extra charges by replicating the EBS Volume data across multiple servers in an availability zone so as to prevent the loss of data.

Lifecycle Management: EBS allows creating lifecycle policies to automate volume creation, retention, and deletion.

Amazon EFS (Elastic File System): This is a fully managed scalable cloud-based storage provided by AWS and accessed by multiple EC2 instances at once. It's designed for scenarios that require shared access to files and data.

Features of EFS
Fully managed: Amazon EFS is a fully managed file storage solution that simplifies the process of sharing files across multiple instances. You as a developer are not required to manage file servers or storage, update hardware, configure software, or perform backups.

Highly available and durable:
Has a durability of 99.999999999 percent and availability of up to 99.99 percent. AWS EFS stores every directory, file, and link redundantly across multiple Availability Zones, for file systems that use Standard storage classes .

Scalability: Offers automatic scaling to accommodate your workloads or documents as they increase. This is done while minimizing time spent on administration.

Security and Access Control:
AWS private virtual cloud provides a secure, managed cloud environment that can be used to run workloads inside AWS.
EFS integrates with AWS Identity and Access Management (IAM) for centralized access control.

Use Cases:
1.Content Sharing: EFS is suitable for storing and sharing multimedia assets, documents, and other files in content-rich applications in a very secured, fast and easy manner.
This ensures consistency across the system.

2.Web Hosting applications: It's a great choice for web servers that require shared file storage for code, configuration files, and uploaded content.

3.Modernize application development: You can exchange the data from the AWS container resources such as ECS, EKS, as well as serverless web applications efficiently and without needing extra management.

4.Big Data and Analytics: EFS can be used to store data for analytical use by Amazon EMR and other big data services.

5.Machine Learning and AI Workloads: EFS is suitable for large data AI applications where same data will be accessed by multiple instances and containers to improve collaboration and reducing data duplication.

Integration with Services:
EFS can be integrated with various AWS services, including EC2, Lambda, and ECS, allowing these services to access the shared file storage.

Cost Considerations:
EFS pricing is based on the amount of data stored and accessed.
It's important to understand how usage patterns affect costs and choose the appropriate performance mode.

Amazon Glacier: Amazon Glacier is a low cost back-up and archival storage solution used for long-term durable data retention. It is appropriate for data that doesn’t need to be accessed frequently or immediately like backups, archives, cold storage.
The above examples each have specific features and use cases, making them fit for different storage use case.
Amazon Glacier is tailored for data archival, backup, and compliance use cases where data is retained for a very long time from few months to 10 or more years.
It offers a very costs-effective storage compared to other AWS storage services but has a much lower retrieval times.

Key features of Amazon Glacier

Vault Creation
Vault:The fundamental storage container in Amazon Glacier where you store your data just like buckets in Amazon S3.

Access policy: Controls who can access the data in a vault and what they can do with it. We can use AWS IAM to safely enter the management console and also safeguard the S3 Glacier data.
It is possible to create multiple users for IAM and specify separate user policies for each individual to limit or allow access to certain parts

Archives: all existing data is stored in archives. You can store different types of data in an archive, including photo images, video, audio, documents and is a top unit of storage in Glacier. We have unlimited storage capacity on AWS Glacier in the cloud and therefore, can store an infinite number of archives in a vault.

Data Retrieval Options:
Expedited Retrieval: Among the 3, this retrieval option Provides the fastest access to data, here the retrievals can happen within 1 – 5 minutes. This mode of data retrieval is needed when you have an urgent need to access data quickly from a group of archives.
Standard Retrieval: The retrieval time for data in this case takes several hours(3 - 5 hours) and is suitable for most use cases. This retrieval method is the most common. It be used for any size of data, full or partial archive.
*Bulk Retrieval: * Designed for data that is infrequently accessed. This is the most cost-effective retrieval with retrieval time which is within 5-12 hours. This data retrieval is suggested for bulk retrieval with access to significant portions of data(petabytes of data), cost-effectively.

Data Lifecycle Management: Data in Amazon Glacier can be managed using data lifecycle policies that changes objects from one storage class to another.
Leading to optimization of storage costs based on the data's access patterns over time.

Vault Lock:
Vault Lock is a feature that lets developers set retention policies on individual vault to implement data retention and stability for compliance purposes.
Once data is locked, it cannot be deleted or changed until the specified retention period expires. For example, WORM (Write Once Read Many) policies can be used to prevent additional edits after uploading.

Data Security:
Data is encrypted at rest by Amazon Glacier using the AES-256 encryption algorithm while supporting secure data transit with SSL.
Developers can choose to manage the encryption keys themselves or use AWS Key Management Service (KMS) for added security.
Data stored in Amazon Glacier is immutable, which means that after an archive is created it cannot be updated.
Gaining entry to Glacier requires credentials which AWS can use to authenticate your requests and it must have permissions to access Glacier vaults or S3 buckets.
Every requests is required by Glacier to be signed for authentication protection.

USE CASES
Data Archival: Amazon Glacier is suitable for storing historical records, financial data, medical records, and legal documents.
Backup and Recovery: Industries can use Glacier to store backups of critical systems and data.

Digital preservation. For example Government agencies can use Glacier for digital preservation efforts. Libraries can use it to preserve various data as well.

Applications such as Facebook and Instagram with large amount of multimedia data coming in, can use Amazon Glacier to store data.

Compliance: It helps organizations meet up with the regulatory requirements for data retention. Glacier enables financial services, healthcare institutions and other industries to keep data which serves as a regulatory and compliance archives over extended periods of time.

Integration with Other Services:
Amazon S3 Integration: lifecycle policies can be set up to automatically transition objects from S3 to Glacier based on access patterns.
AWS Data Transfer Services: Snowball and Snowmobile can be Used to transfer large amounts of data to Glacier.

AURORA DATABASE ENGINE FOR STARTUPS

Beatrice Akaeme — Mon, 31 Jul 2023 04:24:42 +0000

What is an AWS Database Engine?
AWs offers a array of services which uses different database engines according to the needs of the customer.
AWS database services offers fully managed service that is scalable, reliable and has a very high performance. It takes care of the administrative tasks such as setting up of the database, patching, back-up and scalability.
Data is created, read, updated and deleted from your database with the use of Database Engine.
Amazon web services supports 7 different engines such as MySQL, MariaDB, PostgreSQL, Aurora, Oracle, and Microsoft SQL

1)What is Amazon Aurora?
Amazon Aurora is an AWS cloud-based relational database engine which fully managed and developed in RDS. Amazon Aurora has a lot of uses and features which includes storing your data and provides response really fast and efficiently processing according to the database workloads. Aurora is also cost-effectiveness and simple open-source databases that supports all open-source MySQL functionality and runs only on AWS infrastructure.

What are the features of Amazon Aurora?
1)With the use of Amazon Aurora your database is secured.

2)It is cost effective. With Amazon Aurora You only pay for the resources you use.

3)Your DB instances is highly available since it can be replicated in multiple availability zones giving your organization availability of 99.99%.

4)Amazon Aurora is highly scalable going up and down to meet the demands of your organization changing in size according to your storage needs.

5)AWS fully takes care of Updating, monitoring, upgrading, or provisioning of database so that developers don't need to worry about that.

WHAT MAKES AMAZON AURORA A GOOD CHOICE FOR STARUPS?
Amazon Aurora can be an excellent choice for startups for the following reasons:

Easy To Setup: Amazon Aurora is fully managed service and very easy to setup which gives startups time to focus on building their applications and products without worrying about database administration.

Cost-Effective: With Amazon Aurora, startups can take advantage of its pay-as-you-go pricing model offering, paying for only what they use, this can be very cost-effective for startups. As a result of this, startups don't need to invest in costly hardware upfront.

High Performance: Amazon Aurora has a high performing SSD-backed storage and architecture which is used for applications for various workloads in your organization. Aurora can handle rapid growth and provide low-latency responses for startups.

Scalability: Aurora serverless can scale your startup's compute capacity automatically raising it up and down by adjusting instance size and adding read replicas to ensures that the database can grow along with the business whose workload can be unpredictable.

Compatibility: If your startup can easily migrate existing database to Amazon Aurora with minimal changes if it is already using MySQL or PostgreSQL since they are compatible.

Security and Reliability: Your company's data is secure and highly available because Aurora provides features like encryption at rest and in transit, automatic backups, and multi-region replication

Global Reach: If your startups has a global user base, then Amazon Aurora Global Database will make replication across multiple AWS regions possible, thereby ensuring low-latency access to data for users around the world.

Managed Backups and Recovery: Continuous backups of Aurora is automatically , this allows startups recovery of their databases up to a point in case of data loss or corruption.

Fully Managed Replication: Amazon Aurora supports both automated and manual read replicas, allowing your organization to unload read traffic and expand read throughput when this is done data consistency is not compromised.

Focus on Core Business: Startups can focus on building and improving their core products and services and leave managing database infrastructure to AWS taking advantage of Amazon Aurora's managed service,

However, it's important to consider your business specific use case, all the requirements and budget as you choose a database solution for your startup. It is advisable to always perform a careful evaluation of your startups technical requirements and growth forecast to arrive at the best decision for your startup's database architecture.

AWS DATABASE ENGINES

Beatrice Akaeme — Wed, 28 Jun 2023 04:26:35 +0000

WHAT IS A DATABASE ENGINE
A database engine is an fundamental software that a database management system uses to function by creating, reading, updating, and deleting data from the database. The database engine does the real work of organizing the information and allows you to work with tables. Database engine represents a fundamental building block of your database.

AMAZON RELATIONAL DATABASE SERVICE(Amazon RDS)
It is a managed Relational Database Service provided by Amazon Web Services (AWS) that supports the most popular varieties of database engines. It is a type of database that stores data in tables with rows and columns. It also stores and organizes large amount of data that applications can draw upon to help them perform various tasks. Relational database is provided by Amazon RDS is cost-effective with resizable capability. Amazon RDS is very useful for some ICT startups that have a functional database infrastructure already in use and are interested in the cloud technologies.

DATABASE ENGINES SUPPORTED BY AMAZON RDS
These include:
MySQL: It is a free, popular and open-source relational database engine that is widely used in web applications. MySQL is one of the most popular database engines in the world. It is free and open-source. It’s commonly fast, very sturdy and can coexist with many different operating systems.
Generally, the solutions obtained from MySQL are used for databases which requires only a straightforward read-write activities as well as strong performance.

Aurora: It is one of the engines supported by Amazon RDS that is fully managed relational database engines which works well with MySQL and PostgreSQL. It is designed to be highly scalable and available delivering fast performance with automatic backups, and failover. Unlike Amazon RDS, Amazon Aurora is designed specifically for the cloud. It combines the performance and availability of customary operational databases, with the simplicity, personalized, and cost-effectiveness of open source databases.

MariaDB: It is a fast, popular open-source relational database management system which is a scalable relational database management system that’s also an improved version of MySQL created by the original developers of MySQL. The performance is enhanced with augmented security, and more storage engines.

PostgreSQL: This is a powerful advanced enterprise highly stable open-source object-relational database system that gives support to SQL. Its broad features makes it a good fit for web applications, data warehousing, and business intelligence which has earned it a strong reputation for reliability, feature robustness, and performance.

Oracle: It is a universally used commercial relational database management system (RDBMS) engine from the American software and hardware. Various editions of Oracle, such as Oracle Standard Edition One, Oracle Standard Edition Two, and Oracle Enterprise Edition are supported by Oracles.

Microsoft SQL Server: It is a commercial relational database engine developed by Microsoft which is widely used in enterprise applications for managing vast amounts of data. Amazon RDS for SQL Server makes it easy to set up, operate, and scale SQL Server deployments in the cloud. SQL server is Microsoft’s owned database engine that supports a broad variety of transactions. As with Oracle, it’s generally best suited for larger companies with massive fixed datasets who already use Microsoft SQL as a core part of their business but not a good choice for startups.

USE CASES OF AMAZON RDS
Some of the most common use cases of AWS RDS include the following:

Web applications: Amazon Relational Database can be used to power web applications to fulfill the needs of very demanding applications, giving them a scalable and highly accessible database backend with high throughput, massive storage scalability, and further high availability. These applications are built on platforms like WordPress, Drupal, Joomla, and many others.

Web and mobile applications: Amazon RDS offer a scalable and secure database solution for web and mobile apps, making it the perfect choice for businesses looking to launch apps fast and effectively.

Mobile applications: Amazon RDS Supports developers of mobile applications that has need for a database platform with high throughput and availability so that they can endlessly update these applications providing them a highly available, scalable database service for applications that need to store and retrieve data in real time.
Mobile application companies can use Amazon RDS to provide a better performance, cost-effective, highly secure, and increased availability for their social and mobile games.

Disaster recovery: It can be used to design, test, set up and manage standby databases and read replicas in different regions or availability zones, providing a dependable disaster recovery solution for critical applications in order to maintain business continuity.

E-commerce: Amazon Relational Database System offers small, medium and large e-commerce businesses a flexible, safe, highly scalable, and cost-effective database solution for online sales and retailing. It enables the company's database to store and manage product catalogs, order histories, and customer data for e-commerce websites and applications. Using Amazon RDS can simplify administrative database tasks thereby saving time and money.
With Amazon RDS companies can add millions of new items to their database every month leading to overall customer satisfaction and sales increase.

Analytics and business intelligence: It can store and manage huge volumes of data for data analysis and business intelligence applications thereby ensuring that there is a fast and reliable access to data used for reporting and analysis in business.

DevOps: It can be combined with other AWS services to automate database distribution and management as part of the DevOps process.

Travel applications: Amazon RDS offers applications such as Airbnb a simplified time-consuming database administrative tasks and automated database replication. Mobile apps like Airbnb also take advantage of RDS' scalable storage capability.

Streaming applications: Applications like Netflix works with RDS because of its storage scalability , and availability of Amazon RDS, which allows them to handle high demand daily.

Data warehousing: Amazon RDS offers a scalable and safe data warehousing solution to businesses that needs to store and analyze very large volumes of data in the cloud.

AWS DATABASES

Beatrice Akaeme — Fri, 02 Dec 2022 04:36:19 +0000

WHAT IS A DATABASE?
A database is an organized systematic collection of structured logically related information which is electronically stored in a computer system so that it can be accessed easily for management and updating. This information can come in any form such as words, numbers, images, videos, sales transactions, customer data, financials and product information. Databases management system also known as a Database, which is used for storing, maintaining and accessing any sort of information as mentioned above or more. Most databases use structured query language (SQL) for writing and querying data.

IMPORTANCE OF DATABASE
**Data Integrity: Databases often have built-in rules and conditions to makes sure that the data is consistent and ensures integrity by making sure it is accurate even if multiple users are working in the system at once.

When there is a conflict in the data, the database automatically chooses one version over another so that only correct information is stored.

Data Security: Privacy and compliance requirements associated with any data is supported in a Database. Databases have various methods to ensure security of data. Your data is secure when it’s kept in a database. Different users might have different levels of access, such as read-only which authorizes certain individuals access, this prevents unauthorized users from gaining access to your sensitive data.
There are also user logins required before accessing a database and various access specifiers allowing only authorized users to access the database

Managing Data Efficiently: Databases allow many massive corporations to easily manage large volume of data on a daily basis, scaling to millions, billions, and more. It is only possible to store this quantity of digital data with a database. This cannot work using spreadsheets. The database helps manage information more effectively than paper-based filing systems because they can store a greater volume of data.
It also allows to addition, deletion, or modification of information quickly and easily thereby keeping data up-to-date and accurate.

Organization of Data: keeps data organized systematically. It organizes and catalogs information otherwise you’d have to manually search for data in multiple different files and documents
Databases makes it easier to find what you’re looking for and keep related data together the form of tables, records, files, rows and columns to allow easy access to the specific information you need at any point in time. Databases allow businesses and organizations to keep track of their company assets, which is important for maintaining accurate records

Easy Research And Update: It is very easy to access and research data in a database. This is done using Data Query Languages (DQL) which allow searching of any data in the database and performing computations on it. In a database, it is easy to update data using various Data Manipulation languages (DML) available. One of these languages is SQL.

Ease Of Use And Reliable
Databases are easy to use and accessed from any computer using internet connection allowing cooperation to quickly and easily get the information they need when it is needed.
Databases store data consistently and reliably. This helps to prevent the data from becoming corrupted.
If the data is stored in a database, it can be easily recovered, even if it is corrupted.
By using a database, you can save a lot of time in your organization and complete projects in less time than before.
Very helpful when you need to access large amounts of data quickly and easily. In addition, a database can help you automate tasks which save you time and money in the long run

For Business Performance
Databases is necessary for any business to improve performance when is well managed. It is needed to keep track of your business operations, making it easier to analyze data. This guides you to make better decisions based on accuracy of information.

Additionally, a database can help you improve customer service, target marketing campaigns, and understand how well your business is performing.

TYPES OF AWS DATABASES
Amazon Relational Database Service: is a fully managed service provided by AWS and is used when you have a finite amount of structured data and want to form links and joins between different tables. It allows users to set up, manage and monitor a database in the cloud. It stores data in tables with columns and rows, with each row representing a single record, differentiated with a unique ID known as a key, and each column containing data related to that record. It is not itself a database. Instead, it gives you the ability to run different relational databases as you want to.

Amazon DynamoDB: Amazon DynamoDB is a NoSQL database service fully managed by the AWS cloud which supports document-based data, managed and automated. It provides fast, predictable, and seamlessly scalable performance and it automatically handles the traffic of data on the multiple servers. It is a fully serverless service by AWS, so you don’t have to worry about capacity planning and scaling. It aids businesses by offloading operating and scaling burdens in a distributed database.

Amazon ElastiCache: Amazon Elasticache is an easy-to-setup, managed, scalable, in-memory data caching, web service provided by AWS used to reduce the requests on your database, providing very low latency and high-performance to the user which is cost-effective, and also scalable caching solutions. Gives two caching engines, namely, Memcached and Redis.

Amazon Aurora: It is another Relational Database Engine built for the cloud, fully managed by AWS which provides support for MySQL and PostgreSQL. It means that code, applications, and drivers are used in the database. It has better performance capabilities than most commercial databases and it can auto-scale up to 64 TB per database instance. It provides better performance in terms of availability, scalability, and read replicas. Since Amazon Aurora is fully managed by RDS, all administrative tasks such as database setup, patching, backups, and more are fully automated.

Amazon Redshift: Amazon Redshift is a fast and fully managed data warehouse service in the cloud designed to help us analyze and query available data set interpreting the data to gain more insight into our business and customers as well. It is a relational database, based on PostgreSQL. It offers query performance using SQL-based tools and business intelligence applications. The Redshift data warehouse can be scaled up to petabyte or more as per the requirements.

Amazon Neptune: It is different from the SQL database and is used for the visual representation of data using graphs. The data are connected and are correlated to each other and have some connection between them. You can run custom analysis on different sets of datasets and show results on the dashboards in the form of graphs. The main purpose of developing a graph database is network security, fraud detection, to understand the drug discovery, and so on. It provides multi-zone availability, read replication, encryption, backup, and recovery. To ensure data availability, it continuously replicates all its data to amazon S3 and other availability zones for purposes of security and high availability as well.

AWS MONGODB AND YOUR BUSINESS NEEDS

Beatrice Akaeme — Sun, 23 Oct 2022 16:47:44 +0000

WHAT IS MONGODB
This is an open source NoSQL database management application used for storing data. Replaces traditional relational databases such as tables and rows, with collections and documents. NoSQL databases are quite useful for working with high volume data storage. It supports various forms of data and enables you to store unstructured data.

HOW IT WORKS
To start with, MongoDB is a database server and the data is stored in these databases. That is to say that it provides an environment which gives you a server that you can start and then create multiple databases on. MongoDB does this by making use of collections made up of documents containing data structure since it is a NoSQL database. The documents use a variant called Binary JSON (BSON), a Binary representation of JSON documents, as a result, data is stored in the MongoDB in the format of BSON documents. These documents are created using the fields. These fields are similar to the columns in a relational database. You are allowed to create multiple databases and multiple collections. Nested data can be stored in MongoDB documents as well as a variety of data types, and arrays of documents.

WHY USE MONGODB
MongoDB stores data in documents making it easy for developers to store structured or unstructured data. MongoDB’s query capabilities are designed specifically for manipulating both structured documents and unstructured data, which makes it easy for users to use. MongoDB was built for people building internet and business applications and so it is flexible and adaptable to real business world situations.

MongoDB has a JSON-like document format which is very flexible and adaptable, making it a natural choice for developers. It has a very flexible data store that stores data in a document. This takes place in a database which holds one or more collections of documents.

MongoDB can also handle high volume of Data and can scale both vertically or horizontally to accommodate large data loads.

Another feature of MongoDB is that it offers an efficient way to search data with text, geospatial, or time-series dimensions
MongoDB supports searching by field, range queries, and regular expression searches
Indexes can be created to improve the performance of searches within MongoDB. Any field in a MongoDB document can be indexed.

In addition, MongoDB includes features to analyze data, including support for numerous simultaneous queries, indexing, and aggregation
Applications should handle increasing numbers of users that use the applications at the same time.

Scaling horizontally means adding more servers to distribute the load across multiple nodes.
Scaling the database horizontally can be achieved in MongoDB through the sharding and replica set features
MongoDB is built on a horizontal scale-out architecture and uses the concept of sharding for distributing data to scale horizontally by splitting data across multiple MongoDB instances.
These scalability features also have benefits for fault tolerance in a MongoDB database deployment.

MONGODB FEATURES
MongoDB is made up of a very flexible database that makes use of collections holding different types of documents in it making it a Schema-less Database. The multitudes of unique documents found in a collection is made up of several numbers of fields, content, and size. The size and content of each document can be different from each other. Data is stored in fields instead of rows and columns which make the data much more flexible compared to RDBMS. The document structure is a clear structure with key-value pairs.

Each field in the documents is indexed with primary and secondary indices this makes easier and takes less time to get or search data from a set of data.
MongoDB allows representation of hierarchical relationships, to store arrays, and other more complex structures more easily.
MongoDB provides great scalability distributing data on multiple servers. It is very flexibility to databases making it perform really high as well as provides high availability.
MongoDB can also perform operations on grouped data and get a single result or computed result.

RUNNING MONGODB ON AWS
Amazon Web Services provides us with various services which we need to build applications in the cloud. These services are reliable, efficient, scalable, flexible, cost-effective, highly available and secure.
With Amazon you can set up infrastructure to support the deployment of MongoDB on AWS Cloud in a scalable, efficient, flexible and cost-effective way.
The best way to run MongoDB on AWS is with MongoDB Atlas, a fully-managed cloud database service that handles and manages the deployments of your modern applications on the cloud. Saves you a lot of time so that you can focus on other business related issues for the growth of your organization.

In order for you to start using MongoDB Atlas, you’ll need to do the following:

Create a MongoDB Cloud account.
Create a MongoDB Atlas cluster.
Configure network access and create a cluster user.
Connect to the cluster.

MongoDB Atlas has a free tier, so there is no need of any payment or credit card information.

Quick Start is used to launch a fully managed MongoDB Atlas deployment on the AWS Cloud. It uses AWS CloudFormation templates for these deployments.

You are only responsible for the cost of the AWS services used while running this Quick Start reference deployment to launch MongoDB

AWS Services
Quick Start uses the following AWS Services:

Amazon EC2: The Amazon Elastic Compute Cloud (Amazon EC2) service is a compute service which developers can use to develop and deploy applications easily and faster on the AWS cloud.
Amazon EC2 enables business subscribers to run application programs in the computing environment

Amazon VPC: Atlas supports network peering which establishing a private connection between your Atlas VPC and your AWS's VPC. With Amazon Virtual Private Cloud (Amazon VPC) service you can provide a private, isolated section of the AWS cloud where you can launch AWS services and other resources in a virtual network of your choice.

Amazon EBS: Amazon Elastic Block Store (Amazon EBS) provides a consistent block level storage volumes which can be used with EC2 instances in the AWS Cloud. Each EBS volume automatically replicates within its Availability Zone in order to protect your applications from failure, it is highly available and durable.
This means that if the EC2 instance on which MongoDB is running fails, then the EBS volume and the data it contains is recoverable.

AWS CloudFormation: AWS CloudFormation is an infrastructure-as-code (IaC) service that provides you with an easy method to create and manage collections of connected AWS resources, and provision and update them so that you can spend less time managing those resources directly from templates and concentrate on your applications that run in AWS.

IAM : AWS Identity and Access Management (IAM) enables you to securely control access to AWS services and resources for your users. The MONGODB-AWS authentication mechanism uses your Amazon Web Services Identity and Access Management (AWS IAM) credentials to authenticate your user. With IAM, you can manage users, security credentials such as access keys, and permissions that control which AWS resources users can access, from a central location

AWS MONGODB AND BUSINESE/STARTUPS
Starting a new business is very challenging and businesses in early stage needs to make decisions that will impact their business long term. MongoDB and Amazon Web Services (AWS) have been collaborating together for over a decade to produce developer tools that simplify the process of building software, while helping startups get going with minimum resources.

MongoDB for Startups helps companies at infancy-stage get started with MongoDB by ensuring they have access to MongoDB’s cloud developer data platform as well as credits and support on MongoDB Atlas, Atlas Search, MongoDB Realm, among others. MongoDB Atlas launched on AWS in 2016 and is now available in over 20 regions across the Earth. By providing a combined set of database and data services and a unified developer experience, MongoDB Atlas on AWS allows companies at all levels build applications which are highly available, successful at international scale, and cooperative with the most demanding security and privacy standards.
Furthermore, MongoDB provides unwavering technical advice, shared marketing opportunities, and partnership opportunities to help startups grow their business organization as well as their customer base.

AWS AND APACHE SPARK

Beatrice Akaeme — Fri, 09 Sep 2022 14:08:41 +0000

WHAT IS APACHE SPARK
Apache Spark is an open source data processing framework which can execute processing analytical tasks on very huge data sets which it allocates across multiple computers. It can handle both batch and real-time analytics and data processing workloads and run in Hadoop clusters through YARN or Spark's standalone mode . It is a very fast since it runs on RAM and is a general engine for a massive data processing. Apache Spark supports various widely used programming languages such as Python, Java, Scala, and R. It can be used for multiple things such as batch applications, iterative algorithms, queries, SQL and streaming.

COMPONENTS
Apache Spark Core – This is the fundamental widespread execution engine for the Spark platform that all other services are built upon.
Spark SQL – is a component which is at the top of Apache Spark’s core which enables users to query data stored in contrasting applications using the common SQL language and it also provides SQL language support.
Spark Streaming – Performs streaming analytics with Spark Core's fast scheduling capability. Data can be ingested in mini-batches from many sources like Kafka, Flume, and HDFS.
MLlib (Machine Learning Library) – This is an Apache Spark's library which contains a wide array of machine learning and statistical algorithms.
GraphX –This is part of Spark's library which is used to manipulate graph databases and perform computations. It is a distributed graph-processing framework on top of Apache Spark

Spark languages
Spark is mainly written in Scala, it is the native language for interacting with Spark core engine . Scala can be a good fit for those who have prior knowledge of Java. It enables developers to write the clean designs of spark applications. The Scala shell can be accessed through spark-shell

Python ; Of all the languages supporting Python is a simple, open-source, general-purpose language and is very easy to learn and understand. Used for creating schemas, calling REST API is much easier to perform with python while working in spark.

R programming language can also be downloaded and run in Spark. This enables users to run the popular desktop data science language on large-scale distributed data sets in Spark in order to use it to build applications which supports machine learning algorithms.

FEATURES

Apache Spark has following features.

Speed − Spark has 100 times faster in memory to help in running an application in Hadoop cluster, also 10 times faster when running on disk. How is this possible? by reducing number of read/write operations to disk. Big data is distinguished by volume, variety, velocity, and veracity which needs to be processed at a very high speed. Spark fast in performance, utilizes memory and disk, and has built-in fault tolerance

Uses multiple programming languages − Spark supports use of multiple programming languages such as Java, Scala, or Python. Therefore, you can write applications in different languages.

Advanced Analytics − Spark supports ‘Map’ and ‘reduce’ as well as SQL queries, Streaming data, Machine learning (ML), and Graph algorithms.

Flexibility – Apache Spark supports numerous languages and allows the developers to write applications in Scala, Java, R, or Python.

In-memory computing – Spark stores the data in the RAM of servers which permits very fast access and in turn increases the speed of analytics.

Real-time processing – Spark is able to process real-time streaming data. Unlike MapReduce which processes only stored data, Spark is able to process real-time data and is, therefore, able to produce instant outcomes.

Better analytics –Apache Spark is made up of a rich set of SQL queries, machine learning algorithms, complex analytics, etc. With all these components, analytics can be performed in a better way with the help of Spark.

Use Case For Spark: It is used in healthcare sector making data available to health workers. It is used in the financial sector for financial products recommendation and also investment banking. It is used in manufacturing sector and retail sector for a more efficient business.

AWS AND APACHE SPARK
On Amazon EMR
Where is the best place to run Apache Spark? You can run Apache Spark on Amazon EMR . With the AWS Management Console, AWS CLI, or the Amazon EMR API You can rapidly and easily create managed Spark clusters. With Amazon EMR you get a managed Hadoop framework that which makes it easy, fast, and cost-effective to process extensive amounts of data using EC2 instances.

On Amazon Glue
AWS Glue is a fully managed extract, transform, and load (ETL) service that is presently being provided by AWS to prepare and load data for analytics . In the AWS management Console You can easily create and run an ETL job. It is a very good data pipeline tool which can automatically creates partitions to make queries more efficient.

Apache Spark on EC2
To install Apache Spark on EC2 instances you will need to to go a set of standard configurations. The first thing we need is an AWS EC2 instance. First create an AWS account and sign into the console with it. In the console click services, then EC2, then Launch instance. After that install Spark.

Spark on Kubernetes
Kubernetes is an open-source container orchestration system initially developed at Google. Spark can run on clusters managed by Kubernetes and this has been growing in popularity. A Kubernetes cluster includes a set of nodes on which you can run containerized Apache Spark applications.

Apache Spark in the Cloud
To deploy Apache Spark in the cloud, Amazon EMR is the best way to go. It allows you to launch Spark clusters in minutes without needing to do node provisioning, cluster setup, Spark configuration, or cluster tuning. EMR enables you to provision one, hundreds, or thousands of compute instances in minutes.

Amazon Redshift and Apache Spark
Amazon Redshift is a part of AWS, it is an analytical database. A Cloud-based Data Warehouse service fully managed by Amazon to handle large raw through a process called ETL. Amazon makes it easy to discover new insights from the data. Amazon Redshift is fast, easy and cheap to use. Spark is used for real-time stream processing, while for Redshift its a near real-time batch operations. You can build an application with Spark, and then use Redshift both as a source and a destination for data

SEVERLESS DATA LAKE ON AWS

Beatrice Akaeme — Wed, 24 Aug 2022 14:20:51 +0000

** WHAT IS DATA LAKE**

A data lake is commonly considered as a storage system where data can be stored in a natural form. It allows organizations to store all their structured and unstructured data at any scale. When you store it using the structured or unstructured, it means that You can get and store your data the way it is, without having to first structure the data, and run different types of analytics. Data lake exist because we all use one form of data or the other and we have got systems of record, we have got streaming data, we have got batch data internal, external data. And its really a combination of theses different kinds of data sources that leads us to get powerful insights about what our users are doing about the way the world is working around us and leads us to develop more intelligent applications.

*WHY DATA LAKE *
AGILITY: A data lake gives the company agility, the ability to work super fast and access data quickly.
It provides a repository where consumers can quickly find the data they need and use it in their business projects.
A data lake lets you break down data silos and combine different types of analytics to gain insights and guide better business decisions.

ORGANIZATION OF DATA: Helps to organize, manage and store all categories of data especially in large cooperation that works with thousands of data.
This cooperation doesn't have to worry about having their data all over the place.

PROCESS INVOLVED IN BUILDING DATA LAKE
DATA INGESTION: The ability to ingest data from various sources.

DATA STORAGE: S3 is used to ensure security and durability of your data at low cost.

DATA CATALOGUE: This is an application that contains your data or all data in your organization. It makes finding your data more easy.

**DATA DISCOVERY/SEARCHING.
**This is a data discovery tool for collecting and evaluating several data sources.

                   **DATA LAKE ON AWS**

This means organizations prefer to create their data lakes on public cloud such as Amazon Web Services(AWS). For the following reasons:

Pay As You Go: When businesses uses public cloud, they get to pay for only the period/capacity they actually use the services for their data. AWS scaling capacity makes equivalent resources available at much lower cost compared to personal and small quantity purchase by every private owned business. Certain services such as, database, table, column and tag-based access controls, and cross-account sharing, are provided by AWS Lake Formation at no charge.

Upfront capital investment: With AWS, there is a No Upfront, Partial Upfront, or All Upfront available. AWS provides these three reserved instance payment options for you choose to choose from based only on your usage. No upfront investment is required in this case.

Fast Development: The required tools needed for development are just a click away on a cloud. Build your Data lake and deploy faster on AWS cloud.

Great tools for your Data Lake: With "AWS Lake Formation", you can set up one of the most simplistic data lake on the cloud

AWS Security: AWS provides you with secure infrastructure for your projects and access to the most complete platform for Big Data.

Higher availability On AWS: You can build and deploy your Data Lake on AWS set to multiple data centers to make it highly available and ensures quicker disaster recovery.

Faster scalability: AWS provides a great choice for Data Lake Scalability such as Amazon S3. It can cost-effectively build and scale data lake of different sizes in a very secure environment.

Reliable and Secure Storage: With Amazon S3, you can be sure of a durable, reliable and secure storage for your Data Lake. In Amazon S3 your data is protected by 99.999999999% (11 9s) of durability.

            **SERVERLESS DATA LAKE.**

AWS Serverless Architecture helps us build and run our Data Lakes easily.
How can we build data lake without having to manage any servers? What AWS services can we use in order to build it? Building your Data Lake on AWS means you as a client building and deploying
your Data Lake without having to manage any servers yourself. Aside from this benefit of not having to manage servers, you only pay for what you consume and this architecture scales as your data grows. It is more scalable, faster to release, and flexible, at a lower cost because users only pay for what they use.

Why serverless?

Storage: A serverless data lake architecture offers a highly available, scalable, secure and durable amount of storage with low latency. A serverless storage solution like S3 is suited for that due to its lower cost. You can store various kinds of data from mobile apps, business apps, websites etc.

For computing: Example of AWS serverless compute services are Glue, Lambda and Fargate used in different use cases. These computing resources are used to run a code on-demand

Management: The basic infrastructure is completely owned and operated by AWS, so the client needs not worry about maintenance other than focusing on ant monitoring the day to day activities of the service.

High Availability: AWS ensure that the architecture is highly availability. The data is replicate across regions automatically.

AWS SERVERLESS AS MICROSERVICE

Beatrice Akaeme — Fri, 22 Jul 2022 09:29:00 +0000

AWS SERVERLESS ARCHITECTURE In AWS, we can build and run applications and services without having to manage infrastructure. This means that your application runs on AWS server rather than yours. You don't need to do much such as provisioning, scaling, and maintaining any servers to run your applications, databases, and storage systems, AWS manages all of that for you making everything so easy for you. You as a developer just need to focus on writing the code needed to run the application. AWS has what is called "Function As A Service" which is a compute platform for serverless to run your functions. Lambda is an example of "Function As A Service". You don't need to configure or maintain servers only pay for the amount of time your code runs on the server. Serverless Functions makes servers so much easier to manage.

Types of Serverless
Backend-as -a -Service (BaaS):This is a cloud-computing model where you outsource computing activities, such as hosting or data storage.
Function-as-a-Service (FaaS): These are units of logic that get incorporated and executed in response to configured events, like HTTP requests.

REASON FOR SERVERLESS
Serverless is very important for any developer because it gives the developer time to concentrate more on writing the codes which he needs for running his business. The developer doesn't need to be concerned about the configuration and management of servers. So that the developer will have time to create products for clients and run his business with ease. It helps scale infrastructure and can adjust quickly with business demands.

WHAT ARE MICROSERVICESS?
According to AWS, Microservices are an architectural and organizational approach to software development where software is composed of small independent services that communicate over well-defined APIs. These services are owned by small, self-contained teams. It takes every application function and puts it in its on service that runs in a container. It is a pattern for organizing computer systems into services that can scale with demand. This means that programmers can use different development tools for every function and are not limited by only one programming language. On the plus side, each microservice can be developed and scaled separately, which helps to improve programmers performance.

REASONS FOR USING MICROSERVICES
microservices can easily scale up the development cycles and can do that separately per component.
It can be deployed independently and can update easily incase if the code doesn't work since every microservice runs on its own.
Programmers understand the code because of its simplicity and independence.
These microservices are often owned and run by small autonomous teams makes it easy for every team to works at its own pace independently of one another.
Microservices has reusable functionality, since it runs as separate peace of code, it can be used in multiple products. Meaning you can write a code to be used for a function and still use it for another feature.
Because the microservices are loosely coupled, it increase the applications resistance to failure.

Types of Microservices
Stateless Microservices: This type of microservices is the type which never keeps any existing data with it.

Stateful Microservices: This type of microservice can maintain a record in a database which making it easier for the developer to program with it very efficiently.

AWS SERVERLESS MICROSERVICES FOR CLOUD COMPUTING
Serverless Functions can be a great deployment model for Microservices because they offer the fastest way to production, the lowest cost, and very high elasticity.
Serverless microservices are cloud-based services which makes use of serverless functions to perform very high and specific roles within an application. Serverless functions can also be easily combined with a range of managed services, which reduces the overhead that is often associated with other microservice execution. At the launch of a new app it may be convenient to choose Serverless Functions, since they are really fast to create and needs very little upfront investment.
AWS is Suited for Serverless Microservices because it offers a wide range of solutions that assist deploying microservices-based software. WHICH includes SDK packages as well as PaaS, SaaS, and IaaS solutions.
An example of an offer by AWS to a host of services which aids Serverless Microservices architecture is Amazon Lambda.

AWS LAMBDA AS A MICROSERVICE
Amazon Lambda: AWS Lambda is a Serverless platform which runs code in response to configured events, like HTTP requests and manages its computing resources while balancing the load. Lambda is firmly integrated with API Gateway. API Gateway makes synchronous calls to Lambda making the application function as serverless. AWS Lambda is a service that allows you to run your functions in the AWS cloud Server eliminating the operational complexity for you and your team. You write and upload your code to Lambda, and it takes care of everything needed to run and manage the execution of your codes. Lambda supports several programming languages, so you can choose the most suitable for your code to run.
It integrates with the API gateway, allows you to invoke functions with the API calls, and makes your architecture completely Serverless. With AWS Lambda, you upload your code and let Lambda take care of everything required to run and scale the implementation to meet your actual demand curve with high availability. Lambda can be invoked from other AWS services or be called directly from any web or mobile application. In microservices architecture, each of the application components is loosely coupled, deployed and ran independently. An API created with Amazon API Gateway, and then AWS Lambda-initiated functions is all you need to build a microservice.

BENEFITS OF SERVERLESS MICROSERVICES
Serverless microservices has serverless architecture obviously, it has less overhead cost, improved cost efficiency, you and your team can easily combine serverless functions with other managed services. Functions can be integrated with databases, message queues, and API management tools in similar ways for different use cases making your code reusable for other functions as well. This helps in reducing the amount of initial code that you have to write as you scale out your application.

Serverless microservices are a good fit for complex, advancing applications, as their modularity makes them able to manage and scale up easily. Both microservices and serverless architecture follow the same structural principles. They both resist traditional, monolithic development approaches, prioritizing scalability, and flexibility instead. Amazon API Gateway provides programmatically generated client SDKs in a number of popular programming languages to lower integration overhead.

SHORTCOMING OF SERVERLESS MICROSERVICES
Teams may find it challenging to define the scope of each function within their microservices.

Repeated overhead for creating each new microservice to be created by the team.

Problems of optimizing server density and usage of resources

Complexity of running multiple versions of multiple microservices at the same time.

client-side code requirements to integrate with many services same time.