DEV Community: fvgm-spec

Building a basic path finder app with help of AI in a few minutes

fvgm-spec — Sun, 19 Jan 2025 13:29:48 +0000

This is a submission for the GitHub Copilot Challenge : Fresh Starts

What I Built

If you are concerned about saving our natural environment, it would be useful for you to have an application always available where you can find the closer paths to recycling containers. I live in a little country in Latin America between Brazil and Argentina, you my guess which one it is, here the programs to protect our natural environment by recycling paper, glass, plastic and other renewable trash, it not exactly as developed like in countries such as Germany or any other in the north of Europe.

People who really cares about this issue, at least here in my country, try to do as many as little things to avoid environmental climate change, just storing the recycling trash in the places where it has to be done. Having an efficient way to do that would be beneficial for every body, so this is the idea that I had: asking AI to help me to build an app to achieve that efficiently.

Demo

Repo

To review what I did link to this repo

Copilot Experience

The first challenge that I faced was starting to use GitHub Copilot in VSCode, is just as easy as creating a directory where you are going to build your code + ctrl + I, and there you go...

Once I received the first answer from my AI prompt, I knew that I was going in the right "path", I wanted to ask for help and build a path-finder App that would help me to get the more efficient routes near to my house to take my recycling trash.

As I wanted to build an ASAP (as simple as possible) app in Python I just asked so to Copilot:

Once I got the suggestions from the AI I started to apply them in my code:

The result of my prompt was clear, a path-finder app, just as simple that I could start working on improving it using AI models provided by Github Copilot.

GitHub Models

I used the Claude 3.5 Sonnet model from Anthropic, as I have previously used this model with other AI editors, such as claude.ai. It is reliable for helping with code and also provides very complete code suggestions.

Conclusion

This demo test using GitHub Copilot Clearly shows how a simple project can be started from scratch, just by having a simple idea in mind. Then it can be improved using creativity and GitHub Copilot features.

Designing Cost-Optimized AWS Architectures: A Comprehensive Guide

fvgm-spec — Thu, 02 Nov 2023 16:49:57 +0000

In this tutorial, we will delve into the art and science of designing cost-optimized AWS architectures. Cost optimization is a crucial aspect of AWS cloud computing, and this guide will provide a blend of theoretical knowledge and practical exercises to help you create architectures that not only meet your performance requirements but also optimize your AWS bill.

Introduction to Cost Optimization in AWS
Understanding AWS Pricing Models
Design Principles for Cost Optimization
Architecting for Scalability and Elasticity
Use Case: Cost Optimizing Client's Architecture
Conclusion

1. Introduction to Cost Optimization in AWS

Understanding the importance of cost optimization is the first step towards building cost-effective AWS architectures. We'll introduce the concept and highlight its significance in cloud computing.

2. Understanding AWS Pricing Models

Before diving into cost optimization, it's crucial to comprehend AWS pricing models. AWS and most of the cloud based services, offers the pay-as-you-go approach, that allows users pay only for the individual services they need, without requiring long-term contracts or complex licensing. So they can only pay for the services they consume, and once they stop using them, there are no additional costs or termination fees.

In the image below are shown the basic pricing principles of AWS pricing model, every single user is charged only for the resources they consume, so there are no upfront fees and no minimum commitment or long-term contracts required.

3. Design Principles for Cost Optimization

One of the Pillars of an AWS Well Architected Framework, is Cost Optimization. The design principles of this Well Architected Framework Pillar includes:

Adopting an efficient consumption model, design principle, is directly related to the pay-as-you-go pricing model previously described, where users pay only for the computing resources they consume, and increase or decrease it depending on business the requirements.

Another remarkable design principle is Measuring the overall efficiency of your workloads and the costs associated with deliveries, then the results of these analyses can be used to understand the gains you make from increasing output, increasing functionality, and overall reducing cost.

Another principle is referred to as Stop spending money on undifferentiated heavy lifting, as AWS does the heavy lifting of data center operations like racking, stacking, and powering servers. It also removes the operational burden of managing operating systems and applications that traditionally were done in managed services.

Last but not least, AWS offers the opportunity to Analyze and attribute expenditure, by making it easier to accurately identify the cost and usage of workloads, which then allows transparent attribution of IT costs to revenue streams and individual workload owners.

If you feel like exploring further, you can always review the Official Documentation

4. Architecting for Scalability and Elasticity

Scaling your architecture based on demand is a fundamental part of cost optimization. Solution Architects should ideally design robust and scalable applications, having always in mind a must-have building principle, which is "build today with tomorrow in mind", it means that apps need to cater to current scale requirements as well as the anticipated growth of the solution. This growth can be either the organic growth of a solution or it could be as a result of a merger and acquisition or a similar scenario.

This topic is closely related to the AWS Reliability Pillar, which states that an application designed under this principle should achieve reliability in the context of scale, modularity, and constructed based on multiple and loosely coupled building blocks (functions).

The image below shows how should be built highly scalable and reliable application using a microservices architecture.

Scalable modular applications

5. Use Case: Cost Optimizing Client's Architecture

So let's assume we working for a client that is facing an issue related to storage and needs to migrate their warehouse comprised mainly of images and scanned documents. They need a solution that guarantees elasticity, scalability, and durability.

Their amount of data goes from 60 to terabytes of data, so they we want to shift to cloud store and their first task is to determine what is the best service to use and secondly, how they'd go about transferring these objects to the cloud.

We can offer our client a few options that could cover the main prerequisites they talked about before (elasticity, scalability, and durability) they can perfectly be Amazon RDS, DynamoDB or perhaps Amazon Aurora?

So far so good, so now the requirement is to determine the nature of the objects to be stored if they are structured or unstructured. objects of various sizes and formats. Our client needs to store all sorts of image formats, including TIFF, JPEG and PNG, in this case Amazon S3 comes up as an ideal option for these purposes, as it provides highly durable, highly available elastic object storage. If you are at least familiar with this versatile and commonly used AWS service, chances are that you have seen this table before:

Another remarkable feature, is that we won't need to provision the Amazon S3 storage level first and the services will scale upwards to accommodate any future volumes of data that we need to provide. That means our clients, could whenever they want store and retrieve any type of object in Amazon S3, no matter if they are images, files, videos, documents, or archives. It scales quickly and dynamically so you don't need to pre-provision space for objects if you store them.

There are three storage classes for Amazon S3, as seen in the table above, and each of them provides different levels of availability, performance, and cost which makes it a highly flexible solution.

Compared to choosing a relational database like RDS Service for storing objects with Amazon S3, would be unlikely to be efficient and the cost might outweigh the benefit of using a relational database on a cloud service.

Now that the implementation is almost done, we can start to look at ways of optimizing our storage further, starting with Amazon S3 Standard Class, which is the most widely used class (used for what is called, "hot data"), as it provides the maximum durability and availability combination of the storage classes, and in fact, it's hard to find a use case that doesn't suit this class at all!

The first point is determining the right class that will support our data, now we should look at how to import the data. As AWS Solution Architect, may think about using a device called AWS Snowball

The reason for choosing it and how this device works, and its main features are a little out of the scope of this article, if you want to explore further just check the very first source of truth, AWS Official Documentation

Infrequent Access Storage Class, is the second storage class analyzed, if Standard Class is best suited for hot data (data which is accessed very frequently), then Infrequent Access Class is ideal for warm data, which can be understood as data that is not accessed so frequently, or no so frequent as the hot data :) We still would need the durability, but the availability in retrieval time is not a critical mission. So if the requirement is to retrieve assets regularly, Standard Class would end up being probably more efficient than Infrequent Access.

Now the third storage class is Amazon Glacier, it is essentially (as its name suggests) the one referred to "cold storage" or cold data.

Glacier suits archives and taped library replacements and anything where we may need to keep a version of a backup or an archive to meet compliance requirements.

So if accessing data using Standard Class it's very, very fast and highly available, with Infrequent Access Class, it means less availability. With Glacier, it can take two to five hours to request an archive be retrieved. So it's perfect for anything where we don't have a time constraint. Objects stored in Glacier have a relatively long return time.

Right at this point, is when Lifecycle Rules come into play, to shift assets from Standard or Infrequent Access storage classes to Amazon Glacier. If we have older backups and taped archives, we might wanna consider importing those directly to Glacier and also setting up rules to shift older assets to Glacier after a period of time.

There is another cool Amazon S3 tool that can help which will help you observe your data or access patterns over time and gather information to help you improve the lifecycle management of your Standard and Standard IA storage classes, it is called Storage Class Analysis Tool. This analysis tool will observe the access patterns of your filtered data sets for 30 days or longer to gather information for analysis before giving you a result, then the analysis continues to run after the initial result and updates the result as the access patterns change, which is a really useful feature.

6. Conclusion

In this comprehensive guide on designing cost-optimized AWS architectures, we've covered the principles, practices, and practical exercises to create architectures that not only meet your performance needs but also save on costs.

As you continue your journey with AWS, remember that cost optimization is an integral part of building efficient and sustainable cloud solutions. Happy architecting!

References

Amazon OpenSearch Service Essentials

fvgm-spec — Wed, 20 Sep 2023 00:27:22 +0000

In this comprehensive tutorial, we will dive deep into Amazon OpenSearch Service, a managed service that makes it easy to deploy, operate, and scale OpenSearch clusters for search and analytics in the AWS Cloud. Whether you're new to Elasticsearch or want to harness its power in the AWS cloud, this guide will provide a blend of theoretical knowledge and hands-on exercises to get you started.

Introduction to Amazon OpenSearch Service
Main Features of Amazon OpenSearch
Creating an Amazon OperSearch Service Domain
Visualizing and Analyzing Data with OpenSearch Dashboards
Conclusion

1. Introduction to Amazon OpenSearch Service

Amazon OpenSearch Service is a fully managed service that simplifies the deployment, operation, and scaling of OpenSearch clusters. Is a community-driven, open-source search and analytics suite derived from open-source Elasticsearch 7.10.2 and Kibana 7.10.2, which allows you to search, analyze, and visualize data in real-time, making it ideal for log and event data analysis, full-text search, and more.

2. Main Features of Amazon OpenSearch

OpenSearch Service includes the following features:

Scale

Numerous configurations of CPU, memory, and storage capacity known as instance types, including cost-effective Graviton instances
Up to 3 PB of attached storage
Cost-effective UltraWarm and cold storage for read-only data

Security

AWS Identity and Access Management (IAM) access control
Easy integration with Amazon VPC and VPC security groups
Encryption of data at rest and node-to-node encryption
Amazon Cognito, HTTP basic, or SAML authentication for OpenSearch Dashboards
Index-level, document-level, and field-level security
Audit logs
Dashboards multi-tenancy

Stability

Numerous geographical locations for your resources, known as Regions and Availability Zones
Node allocation across two or three Availability Zones in the same AWS Region, known as Multi-AZ
Dedicated master nodes to offload cluster management tasks
Automated snapshots to back up and restore OpenSearch Service domains

Flexibility

SQL support for integration with business intelligence (BI) applications
Custom packages to improve search results

Integration with popular services

Data visualization using OpenSearch Dashboards
Integration with Amazon CloudWatch for monitoring OpenSearch Service domain metrics and setting alarms
Integration with AWS CloudTrail for auditing configuration API calls to OpenSearch Service domains
Integration with Amazon S3, Amazon Kinesis, and Amazon DynamoDB for loading streaming data into OpenSearch Service
Alerts from Amazon SNS when your data exceeds certain thresholds

3. Creating an Amazon OpenSearch Service domain

An OpenSearch Service domain, which is the same as an OpenSearch cluster, can be created using the OpenSearch Service console or by using the AWS CLI with the create-domain command. Domains are clusters with the settings, instance types, instance counts, and storage resources you specify.

Practical Exercise: Creating an Amazon OpenSearch Domain

Log in to the Amazon OpenSearch Console, and click "Create Domain".

Configure your domain with settings like version, instance types, and storage.

In this step, you will need to enter the name of your domain, following the suggested naming convention, and you will also need to set a username and master password for this Domain. As the data that we will store in the domain is not sensitive at all and this is a testing exercise, setting Network Public access, will be enough.

For the domain to be available, you will need to wait a range of time from 10 to 15 minutes, so the next step will be to upload some sample data in your OpenSearch domain.

Uploading data to your OpenSearch Domain.

Data is loaded into OpenSearch domains as JSON documents, and you can do it through the command line using cURL or through OpenSearch UI. I feel like a Data Engineer very used to work with CLIs, so I ran into some issues sending requests to my recently created OpenSearch endpoint, this time I preferred to use the UI.

In order to load data using the UI, you will need to log into your Domain by clicking in Amazon OpenSearch Service Domains UI OpenSearch Dashboards URL, which is simply your Domain endpoint + /_dashboards, and then you will be prompted to introduce the master_username and master_password you previously set when creating the Domain.

Once there click in Dev tools.

In Dev tools console you will need to perform a PUT request to the API in order to upload some data to your OpenSearch Domain:

PUT movies/_doc/1
{
  "director": "Burton, Tim",
  "genre": ["Comedy","Sci-Fi"],
  "year": 1996,
  "actor": ["Jack Nicholson","Pierce Brosnan","Sarah Jessica Parker"],
  "title": "Mars Attacks!"
}

Then Click to send request and your first index will be created with the name movies

Querying data from OpenSearch UI.

Once documents are successfully loaded as indexes in your OpenSearch Domain, you can be able to check that the data is loaded by querying it. You can find more sample data to load into your domain in the repo that I created as a companion to this tutorial.

So far I have loaded more data to the endpoint by uploading the following document through the API:

POST movies/_doc/3
{
    "director": "Baird, Stuart", 
    "genre": ["Action", "Crime", "Thriller"], 
    "year": 1998, 
    "actor": ["Downey Jr., Robert", "Jones, Tommy Lee", "Snipes, Wesley", "Pantoliano, Joe", "Jacob, Ir\u00e8ne", "Nelligan, Kate", "Roebuck, Daniel", "Malahide, Patrick", "Richardson, LaTanya", "Wood, Tom", "Kosik, Thomas", "Stellate, Nick", "Minkoff, Robert", "Brown, Spitfire", "Foster, Reese", "Spielbauer, Bruce", "Mukherji, Kevin", "Cray, Ed", "Fordham, David", "Jett, Charlie"], 
    "title": "U.S. Marshals"
}

You can find how to load indexes to OpenSearch in the official documentation.

Now you need to query your data by passing a single query to the API that contains a string of the recently uploaded data, it is similar to the like statement in SQL GET movies/_search?q=U.S.&pretty=true. YOu will get the following output:

4. Visualizing and Analyzing Data with OpenSearch Dashboards

OpenSearch Dashboards is an open-source data visualization tool designed to work with OpenSearch Service Domains. OpenSearch Dashboards gives you data visualization tools to improve and automate business intelligence and support data-driven decision-making and strategic planning.

You can access OpenSearch Dashboards from OpenSearch Domains UI in AWS Console, or simply accessing to

{your_domain_endpoint}/_dashboards/app/home#/

Once there you can add some sample data:

There you have a couple of sample data sources to choose from:

And here is the Dashboard:

This data that you are visualizing through the Dashboard can also be queried in the Dev Tools Console, guess you need to search in the sample_data_flights all flights related to Warsaw, so we will need to query:

GET opensearch_dashboards_sample_data_flights/_search?q=Warsaw&pretty=true

to get all flights with OriginCityName = Warsaw:

5. Conclusion

In this Amazon OpenSearch Service article, we've covered everything from fundamental concepts to practical exercises for creating, uploading and querying data in OpenSearch Service Domain, and and also visualizing data through OpenSearch Dashboards. Amazon OpenSearch Service is a powerful tool for various use cases, from log analysis to real-time data exploration.

With this knowledge, you can confidently leverage Amazon OpenSearch Service to search, analyze, and visualize your data, unlocking valuable insights for your organization. Happy searching!

Amazon DynamoDB Unleashed: Complete tutorial for beginners

fvgm-spec — Tue, 05 Sep 2023 14:33:17 +0000

In this tutorial, we'll dive deep into Amazon DynamoDB, a fast and fully managed NoSQL database service designed for seamless scalability and low-latency performance. You'll gain a solid understanding of both the theoretical concepts and practical aspects of working with DynamoDB, allowing you to leverage its power for your data storage needs.

DynamoDB is the preferred choice when it comes to applications that need low-latency data access. Let's start to learn and practice some of the fundamentals on this amazing fully managed NoSQL database service that AWS offers. If you’re into building scalable, serverless and high-performance applications, this is right for you!

What is Amazon DynamoDB
Main Components
Creating DynamoDB Tables
Working with Data
Data Model and Schema
Why DynamoDB?
DynamoDB vs. other DB Services
Conclusion
Additional Resources

1. What is Amazon DynamoDB

Amazon DynamoDB is a fully managed NoSQL database service offered by Amazon Web Services (AWS). It is designed for developers who need a fast, scalable, and highly available database for modern applications. In this section, we'll explore why DynamoDB is a popular choice and its key features.

For those of you who are new to this terminology, NoSQL databases are known as tabular databases and store data differently than relational tables. They come in a variety of types based on their data model and DynamoDB here works on key-value pairs and other data structure documents provided by Amazon.

DynamoDB only requires a primary key and doesn't require a schema to create tables. Hence it can store any amount of data and serve any amount of traffic, therefore, you can expect a good performance even when it scales up. It's pretty simple to learn and a small API that follows a key-value method to store access and perform advanced data retrieval.

2.Main components

DynamoDB comprises three fundamental units:

Attributes: This is the simplest element in DynamoDB that stores data without any further division. Each attribute has a name and a value. DynamoDB supports various data types for attributes, including strings, numbers, binary data, lists, and maps. Attributes are used to store the actual data in your items.

Items: Items are individual data records within a DynamoDB table. Each item is uniquely identified by a primary key, which can consist of one or two attributes: a partition key (mandatory) and an optional sort key. Items can also have additional attributes that provide data for each record.

Tables: Tables are the highest-level data structures in DynamoDB. They are where you store your data. Each table consists of items, and each item represents an individual data record. Tables are schema-less, meaning that items within a table do not need to have the same attributes, allowing flexibility in your data modeling.

Key Features

Managed Service: DynamoDB is a fully managed service, which means AWS takes care of the operational aspects like server provisioning, scaling, and maintenance. This allows developers to focus on building applications instead of managing infrastructure.

Scalability: DynamoDB can automatically scale to handle high-traffic workloads without the need for manual intervention. It can handle millions of requests per second, making it suitable for applications with variable workloads.

Performance: It offers single-digit millisecond latency for read and write operations, making it an excellent choice for applications that require low-latency access to data.

High Availability: DynamoDB is designed for high availability with built-in data replication and automatic failover across multiple Availability Zones. Your data is always accessible, even in the event of hardware failures.

Security: It provides robust security features, including encryption at rest and in transit, fine-grained access control with AWS Identity and Access Management (IAM), and VPC (Virtual Private Cloud) integration for network isolation.

Serverless Triggers: You can integrate DynamoDB with AWS Lambda to create serverless workflows that respond to changes in your data, enabling real-time processing and automation.

Global Reach: DynamoDB offers multi-region and multi-master capabilities, allowing you to deploy databases globally with low-latency access for users worldwide.

Pay-as-You-Go Pricing: DynamoDB uses a pay-as-you-go pricing model, where you only pay for the read and write capacity you consume and the storage you use, with no upfront costs or long-term commitments.

Use Cases

DynamoDB is well-suited for a wide range of use cases, including:

Real-time applications that require low-latency access to data.
Internet of Things (IoT) applications for managing device data.
Gaming applications for user profiles, leaderboards, and in-game items.

Session management and user authentication in web and mobile apps.

Content management systems and catalogs.

Ad tech platforms for tracking user behavior and ad impressions.

3. Creating DynamoDB Tables: (Practical Exercise) Creating a Sample Data Model

Let's go through the steps to create a DynamoDB table using the AWS Management Console:

Log in to the AWS Management Console.
Navigate to the DynamoDB Dashboard.
Click "Create table" and specify the table name and primary key attributes.
Configure the provisioned throughput or choose on-demand capacity.
Create the table.

Let's create a sample data model for a hypothetical e-commerce application using DynamoDB.

{
  "TableName": "EcommerceProducts",
  "KeySchema": [
    { "AttributeName": "ProductID", "KeyType": "HASH" },
    { "AttributeName": "Category", "KeyType": "RANGE" }
  ],
  "AttributeDefinitions": [
    { "AttributeName": "ProductID", "AttributeType": "N" },
    { "AttributeName": "Category", "AttributeType": "S" }
  ],
  "ProvisionedThroughput": {
    "ReadCapacityUnits": 5,
    "WriteCapacityUnits": 5
  }
}

This schema defines a DynamoDB table for storing e-commerce product data.

4. Working with Data

Now let's populate with some sample data our EcommerceProcucts table, so once the table is created, you will need to click in the table, then Explore table items button in the top right, and then, Create item button. The easy way is to add the Items manually by assigning the Name, Value and Type, I prefer the JSON View, you can put these values in that field, and then Create item

{
  "ProductID": {
    "N": "101"
  },
  "Category": {
    "S": "Electronics"
  },
  "Price": {
    "N": "699.99"
  },
  "ProductName": {
    "S": "Smartphone"
  }
}

Easy peasy isn't it!!

You can use the AWS SDKs or AWS CLI to interact with DynamoDB programmatically. Here's an example of adding data to our "EcommerceProducts" table:

aws dynamodb put-item --table-name EcommerceProducts --item '{
  "ProductID": {"N": "101"},
  "Category": {"S": "Electronics"},
  "ProductName": {"S": "Smartphone"}
}'

This command will add the Item to the table, just as we did before in AWS Console.

5. Data Model and Schema

DynamoDB is schema-less, and what does schema-less mean? it means that DynamoDB doesn't require a schema to create a table, allowing you to define your data structure dynamically. It utilizes the concept of items (records) and attributes (fields) to store and retrieve data.

The structure of a DynamoDB table is also comprised of Primary Keys, Partition Keys, Sort Keys and Partitions, you can deep dive into it in AWS official documentation. Basically keys work the same as it happens in JSON documents, but wait...so DynamoDB is basically comprised of JSON documents? Yes that's exactly what happens.

In the following table structure in JSON for a sample DynamoDB table called Employees, you can find Items and Attributes, where Items are similar to the concept of rows and records in relational DBs systems, each Item represents a different Employee in the table. Each item is composed of one or more attributes, so in this sample table each employee has attributes such as EmployeeID, Company, Username and so on.

Employees table

{
    'EmployeeID': 101,
    'Company': 'Rose-Hill',
    'Username': 'ahenry',
    'Name': 'Antonio Henry',
    'Sex': 'M'
},
{
    'EmployeeID': 102,
    'Company': 'Lowe, Johnson and Flynn',
    'Username': 'fwashington',
    'Name': 'Frank Washington',
    'Sex': 'M',
    'Address': {
                 'Street': '470 David Ports Apt.281',
                 'City': 'Chapmanchester', 
                 'ZIPCode': 'MP 12525'
                }
},
{
    'EmployeeID': 103,
    'Company': 'Cook-Crawford',
    'Username': 'lhawkins',
    'Name': 'Lisa Hawkins',
    'Sex': 'F',
    'Mail': 'hyoung@hotmail.com',
    'Birthdate': datetime.date(2002, 11, 21)}
}

Note the following features of the Primary Key in the Employees table:

Each item in the table has a primary key as we know them from the relational model, that sets a unique identifier from all of the others in the table. In the Employees table, the primary key consists of one attribute (EmployeeID).
Some of the items have a nested attribute, which is Address. DynamoDB allows this kind of attribute to have up to 32 levels deep.

Now let's understand a little bit more about how primary keys and sort keys works in DynamoDB, by loading a table called Music, that has some of my favourite Artists on it:

Music table

{
  "ArtistID": {"N": "50"},
  "ArtistName": {"S": "Metallica"},
  "AlbumID": {"N": "1"},
  "AlbumName": {"S": "...And Justice For All"},
  "SongTitle": {"S": "Dyers Eve"},
  "Genre": {"S": "Metal"}
}

{
  "ArtistID": {"N": "1"},
  "ArtistName": {"S": "AC/DC"},
  "AlbumID": {"N": "4"},
  "AlbumName": {"S": "Let There Be Rock"},
  "SongTitle": {"S": "Whole Lotta Rosie"},
  "Genre": {"S": "Rock"}
}

{
  "ArtistID": {"N": "132"},
  "ArtistName": {"S": "Soundgarden"},
  "AlbumID": {"N": "7"},
  "AlbumName": {"S": "Superunknown"},
  "SongTitle": {"S": "Black Hole Sun"}
  "Genre": {"S": "Rock"}
}

Note that the primary key for the table Music consists of two attributes (ArtistID and ArtistName). Each item in the table must have these two attributes. The combination of ArtistID and ArtistName distinguishes each item in the table from all of the others.

DynamoDB also supports two different kinds of primary keys:

One of them is the Partition key, which is basically a primary key, composed of one attribute known as the partition key.

In a table that has only a partition key, no two items can have the same partition key value.

The other one is the composite primary key, which is the combination of the Partition key and the Sort key. This type of key is composed of two attributes. The first attribute is the partition key, and the second attribute is the sort key.

DynamoDB uses the partition key value as input to an internal hash function. The output from the hash function determines the partition (physical storage internal to DynamoDB) in which the item will be stored. All items with the same partition key value are stored together, in sorted order by sort key value.

In a table that has a partition key and a sort key, it's possible for multiple items to have the same partition key value. However, those items must have different sort key values.

In our Music table, such a thing would happen if we add to the table another item that has the same ArtistID for Soundgarden, in this case the Composite Primary Key will be the ArtistID and SongTitle, and ArtistID identified with 132 in the table will be stored in the same partition.

{
  "ArtistID": {"N": "132"},
  "AlbumID": {"N": "5"},
  "AlbumName": {"S": "Badmotorfinger"},
  "ArtistName": {"S": "Soundgarden"},
  "Genre": {"S": "Rock"},
  "SongTitle": {"S": "Face Pollution"}
}

You can read and learn more from the DynamoDB Core Components in AWS official documentation.

6. Why DynamoDB?

When data no longer fits on a single MySQL server or when a single machine can no longer handle the query load, some strategy for sharding and replication is required. The pitch behind most NoSQL databases is that because they were designed from the ground up to be distributed and to handle large data volumes, they provide some combination of benefits that a simple relational database can't easily offer.

So NoSQL databases allow us to model data close to what the application requires. The relational database management system's way of thinking is to usually force fit every domain model into a structure of tables and columns, and this has led to a plethora of artifacts that try to solve the impedance mismatch problem.

What you'd need in this case, is something that can scale up to that number first and accept various different data types in such a case something like Dynamo DB is easy to set up and the scaling would be extremely smooth hence companies such as Expedia and next in use Amazon Dynamo DB as the primary database to leverage its features to deliver steady latency and a stable application experience to their customers.

7. DynamoDB vs other DB Services

Moving on we shall discuss DynamoDB features compared with other sister services from AWS RDS, each of these services is slightly different in terms of Storage, Maintenance and Scaling features, and sometimes can be pretty difficult to determine which best fits your needs.

In fact, in many situations, multiple databases are a part of a single solution. So let's go through an overview of each of them to help inform your decision further.

8. Conclusion

In this tutorial, we've delved into the world of Amazon DynamoDB, from its data model and schema to creating tables, and working with data.

As you continue your journey with DynamoDB, consider exploring advanced topics such as best practices for optimizing query performance and features like Global Tables and security controls.

Now equipped with a solid foundation, you can confidently harness the scalability and speed of Amazon DynamoDB for your data storage needs.

9. Additional Resources

[AWS DynamoDB Documentation](https://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Introduction.html)
[AWS DynamoDB Tutorial](https://www.youtube.com/watch?v=k0fcbRj_pZE)
[AWS DynamoDB Schema Design](https://www.youtube.com/watch?v=XvD2FrS5yYM)

Happy data modeling and querying!

AWS Storage Services Tutorial: Amazon S3 Basics

fvgm-spec — Mon, 28 Aug 2023 13:57:55 +0000

AWS Storage Services Tutorial: Amazon S3 Basics

In this tutorial, we will delve into one of the foundational AWS storage services: Amazon S3 (Simple Storage Service). Amazon S3 is a highly scalable, durable, and secure object storage service that is designed to store and retrieve vast amounts of data from anywhere on the web. This tutorial will provide you with both theoretical insights and practical hands-on exercises to get you started with Amazon S3.

Introduction to Amazon S3
Creating an S3 Bucket
Uploading and Managing Objects
Pros and Cons from both approaches
Access Control and Permissions
Data Management Features
Conclusion

1. Introduction to Amazon S3

Amazon S3 is an object storage service that provides developers and IT teams with a simple and scalable storage solution. It's designed to store and retrieve vast amounts of data, such as images, videos, backups, and logs. Each piece of data is stored as an object, consisting of the actual data, a unique key (filename), and metadata.

2. Creating an S3 Bucket

An S3 bucket is a container for storing objects. Your first S3 bucket can be created using AWS CLI or through the AWS Console (User Interface), so we will perform both approaches.

Creating bucket from console

Creating a Bucket from the Console is a very straightforward process, you just need to log in to AWS Console and search S3 in the search bar, once there click the button Create bucket

The following screen will appear, these options coresponds to your Bucket name, the AWS Region where you wish to store your data, and some other configurations that generally, are chosen as default, unless you need to set customized and advanced settings before clicking Create bucket in the end of the page.

Once these settings are done, you will have set your first bucket in the S3 storage service.

Now let's do the same but using AWS CLI!

Creating bucket from AWS CLI

This step assumes that you have configured your AWS CLI locally, if this is not the case you can follow this link in order to configure your AWS Command Line Interface.

You can run the line above in your favorite command line interface (CLI) or the one installed by default in your laptop aws s3api create-bucket --bucket your-unique-bucket-name --region your-preferred-region --create-bucket-configuration LocationConstraint=your-preferred-region, please just replace your-unique-bucket-name with a globally unique name and your-preferred-region with the AWS region of your choice after the parameters --region and --create-bucket-configuration LocationConstraint. In the case of your bucket name this link will guide you on the naming rules for buckets Bucket naming rules.

aws s3api create-bucket --bucket your-unique-bucket-name --region your-preferred-region

You will receive the following output in your command line:

3. Uploading and Managing Objects

Uploading data to your bucket is a process that can also be done from AWS Console and from CLI, so let's show how to do it in both cases

Uploading data from AWS Console

Once you created your bucket, you just need to fill it with data coming from multiple sources (local flat files, SQL connections, API requests, among others) and on different formats (text, images, etc...). You just need to get in the bucket, and click the button Upload

There you can add files or folders, by clicking on one or the other button, or dragging and dropping any object in the selected area

Then click on the Upload button at the end of the page.

Uploading files from AWS CLI

Now, let's upload a file to your S3 bucket using the CLI.

aws s3 cp your-file.txt s3://your-unique-bucket-name/

I just uploaded an excel file that I had stored locally by running the line above

I have written a tutorial previously that covers how to do the same loading data actions from CLI but using an incredible python library called awswrangler

You can read it and put it into action from my blog

3. Pros and Cons: Creating S3 Buckets using AWS CLI vs. AWS Console

When it comes to creating Amazon S3 buckets, you have the choice between using the AWS Command Line Interface (CLI) or the AWS Management Console (User Interface). Each option comes with its own set of advantages and disadvantages.

AWS CLI

Pros:

Automation and Scripting: The CLI allows for automation of bucket creation and management tasks through scripts, making it easy to repeat tasks and maintain consistency.
Efficiency: For users comfortable with command-line interfaces, the CLI can be faster and more efficient, especially for bulk operations.
Integration: It's easy to integrate AWS CLI commands into your custom applications or scripts for seamless integration with other processes.

Cons:

Learning Curve: For users unfamiliar with command-line interfaces, there might be a learning curve to understand and use CLI commands effectively.
Potential Errors: Mistakes in commands could lead to unintended actions, potentially affecting your resources or data if not used carefully.

AWS Console (User Interface)

Pros:

Visual Interface: The UI provides a visual representation of your resources, making it easier for users who prefer a graphical interface.
Ease of Use: Creating buckets through the UI is straightforward, requiring minimal technical expertise or familiarity with command-line tools.
Validation: The UI often includes validation checks and prompts, reducing the likelihood of errors during the bucket creation process.

Cons:

Manual Process: Creating buckets through the UI can be time-consuming, especially for repetitive tasks or when managing multiple buckets.
Limited Automation: While the UI can be used to perform manual tasks, it lacks the automation capabilities that the CLI provides, which might be necessary for certain use cases.
Limited Scripting: If you need to integrate bucket creation into your custom scripts or applications, the UI might not be as flexible as the CLI.

In summary, choosing between the AWS CLI and the AWS Console for creating S3 buckets depends on your familiarity with command-line interfaces, the level of automation you require, and your preference for a visual or scripted approach. Both options have their merits, so consider your workflow and needs before deciding which one to use.

5. Access Control and Permissions

S3 offers fine-grained access control. You can control who can access your bucket and objects. By default, all newly created buckets and objects are private.

Making an object public, can be also done from AWS Console by editing the Block public access in the Permissions page shown above, I will show how to do it from CLI

aws s3api put-object-acl --bucket your-unique-bucket-name --key your-file.txt --acl public-read

6. Data Management Features

Amazon S3 provides various features for managing your data:

Versioning: Enable versioning on your bucket to keep multiple versions of an object.
Lifecycle Policies: Define rules to automatically transition objects to different storage classes or delete them after a specified period.
Cross-Region Replication: Replicate objects across different regions for disaster recovery.

7. Conclusion

In this tutorial, we've covered the basics of Amazon S3, from creating buckets to uploading objects and managing access. Amazon S3's scalability, durability, and rich feature set make it a fundamental service for various cloud-based applications.

To explore further, you can learn about advanced features such as S3 event notifications, data encryption, and using S3 with other AWS services.

Remember, this is just the beginning of your journey into AWS storage services. Happy exploring!

Feel free to check the official documentation for more in-depth information and advanced features.

Stay tuned for more tutorials on AWS storage services!

Interacting with Amazon S3 using AWS Data Wrangler (awswrangler) SDK for Pandas: A Comprehensive Guide

fvgm-spec — Sun, 20 Aug 2023 19:09:24 +0000

Introduction

Amazon S3 is a widely used cloud storage service for storing and retrieving data. AWS Data Wrangler (awswrangler) is a Python library that simplifies the process of interacting with various AWS services, including Amazon S3, especially in combination with Pandas DataFrames. In this article, I will guide you into the process of how to effectively use the awswrangler library to interact with Amazon S3, focusing on data manipulation with Pandas DataFrames.

Introduction to AWS Data Wrangler
- What is AWS Data Wrangler?
- Key Features and Benefits
Prerequisites
- Set Up Your AWS Account
- Install Required Libraries
Connecting to Amazon S3
- Configuring AWS Credentials
- Creating a Connection
Uploading and Downloading Data
- Loading Data to S3
- Reading Data from S3
Conclusion
- Leveraging awswrangler for S3 Data Operations
- Further Learning Resources

1. Introduction to AWS Data Wrangler

What is AWS Data Wrangler?

AWS Data Wrangler is a Python library that simplifies the process of interacting with various AWS services, built on top of some useful data tools and open-source projects such as Pandas, Apache Arrow and Boto3. It offers streamlined functions to connect to, retrieve, transform, and load data from AWS services, with a strong focus on Amazon S3.

Key Features and Benefits

Seamless Integration: Integrate AWS services with Pandas DataFrames using familiar methods.
Efficient Data Manipulation: Perform data transformations efficiently using optimized Pandas methods.
Simplified Connection: Easily configure AWS credentials and establish connections to AWS services.
Error Handling: Built-in error handling and logging mechanisms for improved reliability.

2. Prerequisites

Set Up Your AWS Account

Before you start, ensure you have an AWS account setup with the necessary IAM (Identity and Access Management) user credentials with S3 access permissions, and have the AWS CLI configured locally.

Install Required Libraries

It is a generally well know good practice to work in isolated environments, specially when you are trying some new pythn libraries, so if you are a conda user, you should first create a conda environment where you will after that install awswrangler.

First create your conda environment by running:

conda create -n data-wrangling python=3.11

Then activate the environment running:

conda activate data-wrangling

Now is time to install the required libraries inside the environment using the following command:

pip install awswrangler pandas boto3

3. Connecting to Amazon S3

Creating S3 bucket

In order to create the S3 bucket we will use AWS CLI, and if you followed the previous guidelines on Setting up your AWS account, your access keys should be stored in your C:\Users\%USERPROFILE%.aws directory

And then you will be able to create the bucket from command line by running:

aws s3api create-bucket --bucket aws-sdk-pandas72023 --region us-east-2 --create-bucket-configuration LocationConstraint=us-east-2

I have called the bucket aws-skd-pandas72023 but you can name it whatever you like as long as it follows the naming rules for S3 buckets Bucket naming rules

Then you will be receiving the following output in the command line:

And you will be able to visualize the newly created bucket in your AWS user:

Creating a Connection

awswrangler library internally handles Sessions and AWS credentials using boto3 in order to connect to your bucket using your AWS credentials, so besides importing awswrangler you should also:

import boto3

All the packages you would need to import are the following:

#Importing required libraries
import awswrangler as wr
import yfinance as yf
import boto3
import pandas as pd
import datetime as dt
from datetime import date, timedelta

You can also git clone the repository that has the code used in this tutorial.

If you have cloned the repository, you have noticed that we are using the library yfinance to extract stocks data from the API, and then store it in a pandas dataframe, so we can write the extracted data (dataframes) in the previously created S3 bucket using awswrangler.

awswrangler API is able to read and write data in and from a huge kind of file formats and to a numeroius number of AWS services, refer to this list for more information.

4. Loading and Downloading Data from S3 buckets

Covering all the services available on the API to read and write data from and to AWS, would make this tutorial quite long, so we will be working with a commonly used data source storage in data projects, such as S3.

Uploading Data to S3

In the repository shared above I have written a function that writes the dataframe extracted with the get_data_from_api function to the S3 bucket previously created:

def write_data_to_bucket(file_name:str, mode:str):
    """
    Parameters:
    ----------
    mode(str): Available write modes are 'append', 'overwrite' and 'overwrite_partitions'
    """

    path = f"s3://{bucket}/raw-data/{file_name}"
    #Sending dataframe of corresponding ticker to bucket
    wr.s3.to_csv(
        df=df,
        path=path,
        index=True,
        dataset=True,
        mode=mode
    )

So let's put the get_data_from_api into action by passing the stock symbol of NVDA stock and then we will load it to the bucket using write_data_to_bucket

If we go to the S3 bucket, we'll notice that there will be a new folder inside the bucket with the name of the stock, and a CSV file inside of it:

You can also pass multiple dataframes to the function so they will be created in the bucket:

Reading Data from S3

Reading data from an S3 bucket using awswrangler, is a very straightforward task, as you only need to pass the S3 path where your files are stored and the path_suffix corresponding to the method you are using, in this case read_csv. Find more parameters available to use with this method in awswrangler API reference.

In my tutorial I have written a function that takes the folder name where the CSV file was stored when we wrote the data coming from the API in the S3 bucket:

def read_csv_from_bucket(folder_name:str) -> pd.DataFrame:
   df=wr.s3.read_csv(path=f"s3://{bucket}/rawdata/{folder_name}/", 
   path_suffix = ".csv")

   return df

5. Conclusion

There are a lot of methods available in the API Reference page inorder to interact with S3 and multiple other AWS services.

I encourage you to keep testing the once you find useful to integrate with your ETLs and pipelines.

AWS Data Wrangler simplifies the interaction with Amazon S3, providing a seamless experience for performing data operations using Pandas DataFrames. This tutorial covered the fundamental concepts of connecting to S3, uploading, downloading, and transforming data, as well as advanced interactions. By leveraging AWS Data Wrangler, you can streamline your data workflows and focus on deriving insights from your data.

Cheers and happy coding!!

Resources for Further Learning

AWS Data Wrangler Documentation: https://aws-data-wrangler.readthedocs.io/
AWS Data Wrangler GitHub Repository: https://github.com/awslabs/aws-data-wrangler
AWS SDK for Python (Boto3) Documentation: https://boto3.amazonaws.com/v1/documentation/api/latest/index.html

AWS in Plain English

fvgm-spec — Mon, 11 Apr 2022 11:58:32 +0000

This is supposed to be my very first article regarding my learning path on AWS ecosystem.

This time I will write about a useful tool I found in one of my readings, it is AWS in Plain English, of course is not complete documentation of each AWS service like we can find in the official documentation, but it is a compiled guide that tell us, for example:

What Should we have called that service (EC2 = Amazon Virtual Servers)
How we usually use this service ÇIt´s self explanatory)
It´s like: A short description of what this service is similar to other we for sure have used in the past (VPC = VLAN networks).

As a summary, it has a lot of short description of AWS services that we already know and some others that we maybe not.

For the link visit https://expeditedsecurity.com/

DEV Community: fvgm-spec

Building a basic path finder app with help of AI in a few minutes

What I Built

Demo

Repo

Copilot Experience

GitHub Models

Conclusion

Designing Cost-Optimized AWS Architectures: A Comprehensive Guide

Table of Contents

1. Introduction to Cost Optimization in AWS

2. Understanding AWS Pricing Models

3. Design Principles for Cost Optimization

4. Architecting for Scalability and Elasticity

5. Use Case: Cost Optimizing Client's Architecture

6. Conclusion

References

Amazon OpenSearch Service Essentials

Table of Contents

1. Introduction to Amazon OpenSearch Service

2. Main Features of Amazon OpenSearch

3. Creating an Amazon OpenSearch Service domain

Practical Exercise: Creating an Amazon OpenSearch Domain

4. Visualizing and Analyzing Data with OpenSearch Dashboards

5. Conclusion

Amazon DynamoDB Unleashed: Complete tutorial for beginners

Table of Contents

1. What is Amazon DynamoDB

2.Main components

Key Features

Use Cases

3. Creating DynamoDB Tables: (Practical Exercise) Creating a Sample Data Model

4. Working with Data

5. Data Model and Schema

6. Why DynamoDB?

7. DynamoDB vs other DB Services

8. Conclusion

9. Additional Resources

AWS Storage Services Tutorial: Amazon S3 Basics

AWS Storage Services Tutorial: Amazon S3 Basics

Table of Contents

1. Introduction to Amazon S3

2. Creating an S3 Bucket

Creating bucket from console

Creating bucket from AWS CLI

3. Uploading and Managing Objects

Uploading data from AWS Console

Uploading files from AWS CLI

3. Pros and Cons: Creating S3 Buckets using AWS CLI vs. AWS Console

AWS CLI

AWS Console (User Interface)

5. Access Control and Permissions

6. Data Management Features

7. Conclusion

Interacting with Amazon S3 using AWS Data Wrangler (awswrangler) SDK for Pandas: A Comprehensive Guide

Introduction

Table of Contents

1. Introduction to AWS Data Wrangler

What is AWS Data Wrangler?

Key Features and Benefits

2. Prerequisites

Set Up Your AWS Account

Install Required Libraries

3. Connecting to Amazon S3

Creating S3 bucket

Creating a Connection

4. Loading and Downloading Data from S3 buckets

Uploading Data to S3

Reading Data from S3

5. Conclusion

Resources for Further Learning

AWS in Plain English