DEV Community: Carlos Moreno

An experience with Database Modernization: From Oracle for Aurora PostgreSQL – A Cloud Perspective

Carlos Moreno — Fri, 02 Aug 2024 18:17:32 +0000

Hey cloud folks! Recently, As an Solution Architect i had the chance to work on a pretty cool project with a customer. They were running a whopping several Oracle databases on Amazon RDS for their core services, and they wanted to switch things up to Amazon Aurora PostgreSQL. Why? Cost savings, better performance, and the flexibility of open source – the usual suspects!

Now, migrating code and schemas from Oracle PL/SQL to PostgreSQL isn't exactly a walk in the park. It's like translating between two languages with different dialects, data types, and grammar rules. We had to tackle stored procedures, packages (with functions and SPs), functions, and even some quirky Oracle-specific stuff. Automatic conversion tools like AWS SCT are lifesavers, but there's always that "last mile" that needs some hands-on attention.

The Approach: A Mix of Automation and Expertise

We rolled up our sleeves and put together a crack team of a mix of Database specialists and AWS experts who knew their way around both AWS and Oracle/PostgreSQL. We used a phased approach, starting with a deep dive into databases landscape. We mapped out dependencies, prioritized schemas, and even did some code spelunking to see what we were up against.

For the actual migration, we leaned on AWS DMS (Database Migration Service) and SCT (Schema Conversion Tool) to automate as much as possible (almost 50%). But we also had to get our hands dirty with some custom scripts and even used some fancy GenAI agents to help us out with code conversion suggestions. Think of them as our AI-powered code whisperers!

Phase 1: The Scouting Mission – Detailed Evaluation and Planning

Before we dove headfirst into migrating client's databases, we took a step back and planned our attack. We mapped out a detailed migration plan, complete with a realistic timeline (remember, this is a time & materials gig, so flexibility is the key!), assigned responsibilities, and a backup plan in case things got bumpy. We wanted to make sure everyone was on the same page and knew their role.

We also did a deep dive into client's database ecosystem. We figured out how the databases were connected to each other and to the applications they supported. We sorted the databases by how important they were and how tricky they might be to migrate. We even peeked under the hood of some custom code to get a sense of the manual conversion effort we were in for. Think of it as a pre-flight checklist before takeoff!

Phase 2: Gearing Up – Preparation for Database Modernization

With our plan in hand, we started setting up our toolkit. We configured AWS DMS and SCT, but we also wrote some custom scripts to handle the more complex stuff.

We even had some AI buddies helping us out with code conversion suggestions – hey, we're not afraid to embrace new tech!

To make sure our plan was solid, we ran a pilot migration on one of the less complex schemas. This was our chance to test our tools, our manual conversion strategy, and get feedback from client's team. It's like a test drive before buying a new car, right?

A very Goog Manual Conversion Strategy:

Alright, let's talk about the nitty-gritty of manual code conversion. Since the automated tools couldn't handle everything, we had to roll up our sleeves and dive into the Oracle code ourselves. Here's how we tackled it:

- Analyzing and Planning

First, we put on our detective hats and really got to know the Oracle code. We combed through every line, looking for clues about the business logic, dependencies, and any Oracle-specific quirks. We talked to the developers and users to get the inside scoop, and we even drew up flowcharts to visualize how everything worked together. It was like piecing together a puzzle!

- Converting Functions, Procedures, and More

Next, we started translating the Oracle code into PostgreSQL. This meant replacing things like implicit cursors with explicit loops or SELECT statements, and adapting exception handling to PostgreSQL's syntax. We also had to carefully check data types and make sure they were compatible. It was like learning a new language, but with code instead of words!

- Documentation and Optimization

Once we had the code converted, we meticulously documented all the changes we made. We also looked for ways to optimize the code and make it run even smoother on PostgreSQL. Think of it as fine-tuning a race car for peak performance!

This manual conversion process was definitely a challenge, but it was also super rewarding. It's a bit like restoring a classic car – it takes time and effort, but the end result is a thing of beauty that runs like a dream.

Phase 3: Full Speed Ahead – Database Modernization at Scale

Once we had the green light from the pilot, we cranked up the migration to full speed. We migrated the databases in batches, taking advantage of AWS's scalability to get things done faster. We also kept a close eye on performance and made sure everything was running smoothly.

After the migration, we did a final round of testing and optimization. We wanted to squeeze every ounce of performance out of Aurora PostgreSQL and make sure client's team was happy with the results. We even gave them some training on how to manage their shiny new database environment.

Lessons Learned

One thing we learned is that schema complexity matters. All of Lala's databases were labeled "Very Complex" by the tools, so we knew we'd have our work cut out for us. We also found that about 55% of code objects needed manual conversion – that's a lot of coffee-fueled coding sessions!

The Payoff: Happy Cows, Happy Devs

In the end, the hard work paid off. Client getting those sweet cost savings they were after, and their databases are humming along on Aurora PostgreSQL. Plus, they've got a more modern, scalable platform that's ready for whatever the future holds.

If you're thinking about a similar database modernization project, my advice is to plan carefully, use the right tools, and don't be afraid to get your hands dirty with some manual code wrangling. Oh, and maybe keep a coffee pot handy!

Transforming Knowledge Access in Financial Services with a Generative AI-Powered Chatbot

Carlos Moreno — Fri, 02 Aug 2024 17:19:43 +0000

In the dynamic and complex world of financial services, staying ahead of the curve requires quick and accurate access to information. Leading financial institutions are turning to innovative solutions to empower their employees with the knowledge they need to make informed decisions. I want to show you how can deploy a cutting-edge AI-powered chatbot, built on AWS, that is transforming how financial professionals access and leverage critical business knowledge.

The Challenge

Financial services organizations face a constant influx of data, including market trends, regulatory updates, internal procedures, and client information. Employees often struggle to find the right information at the right time, leading to delays, inconsistencies, and missed opportunities.

The Solution: Your own AI-Powered Knowledge Assistant

This AI-powered chatbot acts as a virtual expert advisor, available 24/7 to provide accurate and contextualized answers to employee queries. Leveraging the power of AWS's Generative AI, this chatbot is designed to understand natural language and deliver relevant information on a wide range of financial topics, including:

Regulatory Compliance: Staying up-to-date with the latest regulations and their implications.
Investment Strategies: Accessing research, market analysis, and portfolio recommendations.
Risk Management: Understanding risk factors, mitigation strategies, and compliance procedures.
Client Services: Quickly finding answers to client questions and resolving issues.
Internal Operations: Streamlining onboarding, accessing company policies, and understanding internal processes.
Technical Underpinnings: The Power of AWS

The chatbot is built on a robust AWS architecture, utilizing the following key components:

Natural Language Processing (NLP) Engine: For understanding and interpreting user queries.
Foundation Models (FMs): Powerful language models that generate text and understand meaning.
Knowledge Bases: Integrate with internal data sources to create a comprehensive repository of information.
Vector Databases: Store information in a way that enables efficient semantic search.
Collaboration Platform Integration: Seamlessly integrates with popular communication tools like Microsoft Teams.

Key element of this Solution: Retrieval Augmented Generation (RAG)

The chatbot goes beyond simple question-and-answer interactions. It employs a Retrieval Augmented Generation (RAG) approach, which combines:

Semantic Search: User queries are transformed into numerical representations that capture their meaning. These representations are used to search for relevant information in the knowledge base.
Contextual Generation: The most relevant information snippets are retrieved and used to generate accurate and contextually relevant responses.

Architecture Diagram

The initial release of the chatbot is a Minimum Viable Product (MVP). While it offers core functionalities, i envision a future where it evolves into a more comprehensive solution with advanced features like:

Personalized Recommendations: Tailoring responses based on user roles and preferences.
Advanced Analytics: Providing insights into user behavior and knowledge gaps.
Multimodal Capabilities: Understanding and responding to images and other media.

Core AWS Services

Amazon Lex: The core of the chatbot. It handles natural language understanding (NLU), enabling the bot to comprehend user queries and determine the appropriate response.
Amazon Bedrock: Provides access to a variety of powerful foundation models (FMs), including Amazon Titan for creating embeddings (numerical representations of text) and other FMs (like Anthropic) for text generation.
Knowledge Bases for Amazon Bedrock: This service allows you to connect your internal data sources (e.g., documents, manuals) to the foundation models. It manages the retrieval of relevant information from these sources based on user queries.
Amazon OpenSearch Serverless: Acts as a vector database. It stores the embeddings (numerical representations) of your knowledge base content. This allows for efficient semantic search, finding documents that are most similar in meaning to the user's question.
AWS Lambda: Serverless compute service that can be used for custom logic or integration with other systems. In this case, it might handle preprocessing of user queries or post-processing of responses.
Amazon DynamoDB: A NoSQL database that can store conversation history, user preferences, or other data needed for the chatbot's operation.

How to Interact with the Chatbot

User Interaction: The user interacts with the chatbot through a frontend interface (in this case, Microsoft Teams).
Query Processing: Amazon Lex receives the user's query and uses natural language understanding to determine the intent.
Retrieval Augmented Generation (RAG):
Embedding Generation: Amazon Bedrock's Titan model converts the user's query into an embedding (numerical representation).
Semantic Search: OpenSearch Serverless compares the query embedding with the embeddings stored in its vector database to find the most semantically relevant documents.
Response Generation: The relevant documents are passed to another foundation model in Bedrock (e.g., Anthropic) along with the original query. This model generates a natural language response based on the retrieved information.
Response Delivery: Amazon Lex sends the generated response back to the user through the interface.

Key Points

This architecture is designed as an MVP, focusing on core functionality to get the chatbot up and running quickly.
The diagram indicates a connection to a data lake. This could be a source of additional data that can be integrated into the knowledge base over time.
AWS services are inherently scalable, allowing the chatbot to handle increasing traffic and data volumes as needed.

From Challenge to Savings: Successful Migration Business Case

Carlos Moreno — Tue, 27 Feb 2024 13:35:02 +0000

To fully leverage the benefits of public cloud, it is essential to build a solid business case.

It all starts with the Business Case. A well-defined business case should:

Identify migration objectives: What is the goal of migrating to the cloud?
Evaluate costs and benefits: How much will migration cost, and what benefits are expected?
Develop a migration plan: How will the migration be carried out?
Measure success: How will the success of the migration be measured?

Investing in a solid Business Case is an investment in the success of the migration.

Migration to Cloud can be a complex and costly project. Without a well-defined plan, there's a risk that the migration may not meet expectations, generate unexpected costs, and even negatively impact business operations.

A Business Case acts as a guiding force towards migration success. By investing time in its development, the following benefits are gained:

Clarity and justification:

Defines specific migration objectives, providing clarity to stakeholders.
Justifies the investment in migration by demonstrating the expected business value.

Planning and strategy:

Establishes an action plan with the necessary stages and activities for migration.
Defines the most suitable migration strategy, considering factors such as application types, environment complexity, and business needs.

Evaluation and measurement:

Sets success metrics to evaluate the impact of migration on the business.
Enables measurement of Return on Investment (ROI) and demonstrates the value of migration.

Alignment and commitment:

Aligns stakeholders around migration objectives and benefits.
Generates commitment to the project, ensuring collaboration and support necessary for its success.

A solid Business Case for migration to the public cloud:

Reduces project-associated risks.
Optimizes the use of financial and technical resources.
Maximizes the chances of migration success.
Demonstrates the business value of cloud investment.

Without a well-defined Business Case, migration to the public cloud can be a costly, complex process with a high risk of failure.

Common challenges faced without a cloud migration strategy:

Lack of planning and organization
Technical incompatibility
Unexpected costs
Negative impact on operations
Difficulties in management and control

Examples of savings due to successful migration to public cloud:

Time and resource savings through automation and agility: Calculate the time and resources saved by comparing with processes prior to migration. For example, automation of deployment processes can significantly reduce the time required to provision resources, leading to faster application delivery and cost savings in labor hours.
Cost comparison of cloud security solutions versus on-premise solutions: Evaluate the cost of cloud security solutions compared to on-premise solutions, including hardware, software licenses, maintenance, and staffing. Calculate the savings in investments and costs of preventing attacks. Cloud security solutions often offer pay-as-you-go models, allowing for cost savings through flexible pricing and eliminating upfront hardware costs.
Subscription models for software licenses: The cloud offers subscription models that allow you to avoid paying for on-premise software licenses. This can result in significant cost savings by eliminating upfront licensing fees and reducing the total cost of ownership over time.
Automation and operational efficiency: Automation and operational efficiency can reduce the need for dedicated personnel to manage infrastructure. With cloud-native tools and services, tasks such as scaling, monitoring, and maintenance can be automated, leading to cost savings in labor costs and increased productivity.
Dynamic environment provisioning: Easily create and tear down development and testing environments in the cloud, reducing costs and optimizing resource utilization. With on-demand provisioning of resources, you only pay for what you use, eliminating the need for over-provisioning and reducing infrastructure costs.
High availability and reduced downtime: The high availability of cloud infrastructure reduces downtime, meaning less time spent troubleshooting issues and more time for innovation. By leveraging cloud-native features such as load balancing, auto-scaling, and geographic redundancy, organizations can minimize downtime and ensure continuous availability of their applications and services.

Essential tools for building a strong Business Case and Migration Plan to AWS:

Cost analysis tools:

AWS Cost Calculator: https://calculator.aws/

Risk assessment tools:
Risk Assessment for Cloud Adoption

Discovery and assessment tools:
AWS Migration Hub: https://aws.amazon.com/migration-hub/
Migration Evaluator: https://aws.amazon.com/migration-evaluator/

Benefits analysis framework:
Cloud Economics: https://aws.amazon.com/economics/resources/

The source of the animated gifs is https://www.flaticon.com/

DynamoDB Data Modeling: An Effective way to start...

Carlos Moreno — Mon, 22 Jan 2024 22:35:11 +0000

Effective database design is crucial for the performance and scalability of our applications but also to perform re-architect tasks for App Modernization Journey.

DynamoDB, AWS's fully managed NoSQL database service, offers great flexibility in data modeling. In this article, i will explore the fundamental and some advanced concepts of data modeling in DynamoDB, applying them to a practical context: Business Rules for Retails Stores.

Embarking on the journey of DynamoDB data modeling can be both exciting and challenging. i'll try to demystify the intricacies of data modeling in DynamoDB. Whether you're a beginner or an intermediate user, join us as we explore key concepts, practical exercises and examples.

Tenets of NoSQL Data Modeling

The tenets of NoSQL data modeling focus on leveraging the features and flexibilities offered by NoSQL databases. Here are some mentioned:

Dynamic Schema: NoSQL databases allow dynamic schemas, meaning each record can have different attributes without requiring a fixed structure. This provides flexibility to adapt to changes in data without modifying the database schema.

Denormalization: Unlike relational databases, where normalization is favored to reduce redundancy, NoSQL commonly embraces denormalization. Incorporating redundant data in a single document or record facilitates more efficient queries and reduces the need for multiple queries to obtain complete information.

Query-Centric Modeling: Data design in NoSQL often relies on the most common query patterns. Instead of designing the data structure for all possible operations, priority is given to queries that are performed most frequently, thus optimizing performance for specific use cases.

Eventual Consistency: Many NoSQL databases adopt the eventual consistency model instead of immediate consistency. This implies that after a write, eventual consistency will be achieved, but not immediately. This approach favors availability and partition tolerance in distributed systems.

Partitioning: Partitioning distributes data among nodes or servers to improve efficiency and scalability. Designing data models with partitioning in mind helps distribute the load evenly and minimizes bottlenecks.

Strategic Index Usage: Instead of relying solely on secondary indexes, NoSQL databases often favor the creation of strategic indexes aligned with the most frequent queries. These indexes can be composite and customized to optimize performance.

How we can start?

Define the use case:
This is the first step and involves identifying and clarifying the purpose and objectives of the system or application you are designing. This sets the foundation for making informed decisions during the data modeling design.

Identify access patterns:
Examine how data will be accessed and queried in your application. Identifying these patterns helps design a data model that optimizes query efficiency and operations.

Read/Write workloads:
Understand the relative proportions of read and write operations in your system. This influences the design of your database to optimize performance based on the most common operations.

Query dimensions:
Identify the different ways in which data will be queried and aim to optimize the data model to efficiently support these queries.

Aggregations:
Consider how data should be aggregated and summarized to support efficient aggregation operations such as sums, averages, or item counting.

Design the data model that best suits your needs. This may involve creating tables, defining primary and secondary keys, and establishing relationships but also following simple best practices:

Avoid relational design patterns:
In NoSQL databases like DynamoDB, it is often beneficial to avoid traditional relational database design patterns. This includes extreme normalization.

Start with one table, but use as many as required:
Begin with a simple design using one table, but do not hesitate to create additional tables as needed for your application. DynamoDB favors a polyglot approach to adapt to different use cases.

Understanding Tables in DynamoDB

DynamoDB's table structure, diverging from traditional relational databases, revolves around primary keys, sort keys, and attributes.

Primary Keys: The primary key is the cornerstone of a DynamoDB table, serving as the identifier for each item within it.

Components:

Partition Key: Often referred to as the hash key, it determines the partition or storage location of the item based on its value.
Sort Key: Also known as the range key, it comes into play when items share the same partition key, facilitating efficient sorting and querying within that partition.

Identifying Primary Keys:
Primary keys in DynamoDB are fundamental to table design. They are divided into two components: the partition key and the sort key. The partition key determines the partition in which the item will be stored, while the sort key organizes items within the partition. Properly identifying primary keys is crucial for query performance and efficiency.

How are Inserts and Reads?
In DynamoDB, the performance of write and read operations is directly related to the table design and primary keys. DynamoDB is highly scalable and distributed, but optimal performance is achieved by evenly distributing operations across partitions and keys. Inserts and reads benefit from designing primary keys that effectively distribute the load.

Avoid Overloading Items into Partitions:
DynamoDB distributes data across partitions, and the service's efficiency is optimized when items are evenly distributed among partitions. Overloading items in a single partition can create bottlenecks and impact performance. It is essential to design primary keys to evenly distribute the load and avoid overloading a single partition. This is known as a "hot partition" and should be avoided to ensure balanced performance.

Sort Keys:

Functionality: While the partition key is fundamental for item retrieval, the sort key adds an extra layer of sophistication by allowing items with the same partition key to be distinguished and organized.
Use Cases: Sorting items based on attributes such as timestamps or numerical values becomes seamless with the sort key, enabling targeted queries within specific partitions.

Attributes:

Nature: DynamoDB items are essentially collections of attributes, each of which holds a specific piece of data.
Flexibility: Attributes accommodate various data types, from simple strings and numbers to complex structures like lists or maps, offering flexibility in data representation.

Key Distinctions from Relational Databases:

Schema-less Nature: DynamoDB operates on a schema-less or schema-flexible model, allowing each item in a table to have different attributes. This contrasts sharply with the rigid structure of relational databases.
Indexing Approach: Traditional relational databases heavily rely on indexes for efficient querying. In DynamoDB, the primary key itself acts as a natural index, streamlining access patterns without the need for additional indexing mechanisms.
Scalability: DynamoDB's architecture, particularly its partitioning mechanism, enhances scalability. The partition key distributes data across multiple nodes, enabling the system to handle varying workloads seamlessly.

Data Access: Read and Write Patterns
Efficient data access is paramount. Learn common read and write patterns to optimize latency and speed. Uncover strategies that DynamoDB offers to handle various workloads effectively.

Building Queries: In DynamoDB, queries are constructed using the Query operation. You can perform queries using the partition key and, optionally, the sort key to retrieve a specific set of items. Queries are efficient in DynamoDB and benefit from a well-structured primary key design.
Sort Key Condition vs. Filter Expressions: When conducting queries in DynamoDB, you can specify conditions on both the partition key and the sort key using KeyConditionExpression. Additionally, you can apply filter expressions using FilterExpression. The main difference lies in that KeyConditionExpression operates on the keys directly and is more efficient, while FilterExpression filters the results after they have been retrieved.
Composite Keys:Composite keys, or composite primary keys, are an essential feature in DynamoDB. They enable the creation of more complex data models by combining a partition key and a sort key. This allows for efficient queries and logical organization of data. A well-designed composite key is crucial to fully leverage DynamoDB's query capabilities.

Now, go with some exercises...

Exercise #1 – Discount Control in Stores

Suppose you are designing a database to control the rules regarding discounts that a retail store in a specific area of Monterrey, Mexico, can apply at a point of sale (POS) in a single day.

Requirements:

Each store must have a unique identifier.
It is necessary to establish a maximum limit for discounts that a store can apply in a day.
Record when a store has exceeded the daily discount limit.
Information about the store's location and the specific area within a specific city is required.

Questions to Consider:

Ideal Primary and Secondary Keys:

What would be the ideal primary and secondary keys for this scenario?
How can you ensure a unique identifier for each store while efficiently organizing data for queries?

Modeling Discount Rules:

How would you model the information regarding discount rules in the database?
Should you create separate tables for store details and discount rules, or can they be effectively managed within a single table?

Attributes Required to Meet Requirements:

What attributes are necessary to fulfill the specified requirements?
Consider the data points needed for tracking daily discounts, location information, and identifying when a store surpasses the discount limit.

Possible Solution to Exercise #1 - Primary Keys:

Partition Key:

Unique Store Identifier.
Sort Key: Can be the date of the day to control daily limits.

Attributes:

Maximum Daily Discounts Limit for the Store.
Discounts Applied in a Day.
Location Information: Specific Zone within a Specific City.

We can test the functionality of the table by creating a Lambda function with code similar to the following



import boto3

def query_discount_rules_for_store(store_id, date):
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table('RetailStoreRules')

    filter_expression = Key('StoreID').eq(store_id) & Key('Date').eq(date)

    response = table.query(
        KeyConditionExpression=filter_expression
    )

    items = response.get('Items', [])

    return items

In this example:

We use the boto3 library.
The dynamodb.Table method is used to load the DynamoDB table named 'RetailStoreRules'.
The KeyConditionExpression is constructed using the eq method for equality comparisons.
The table.query method is called with the constructed filter expression, and the results are retrieved from the response.
Make sure to install the boto3 library before running the Python code

Now, go with another exercise but using Composite Keys...

Exercise #2 – Composite sort key for discount limits in stores

Let's say you are designing a table called RetailStoreLimits to track the daily discount limits that a retail store in a specific area of Monterrey, Mexico, can apply at a point of sale (POS).

Requirements:

Each store must have a unique identifier.
Record the daily discount limits that a store can apply.
Enable the retrieval of daily discount limits for a specific store within a specified date range.

Possible Solution to Exercise #2 - Composite Keys:

Identify Relevant Attributes:

StoreID: Unique identifier for the store. -Date: Date for the daily discount limit.
MaxDailyDiscountLimit: The maximum daily discount limit allowed for the store.

Define the Structure of the Composite Sort Key:

Combine StoreID and Date to form the composite sort key.
Composite Sort Key: "StoreID#Date"

Configure the Sort Key in DynamoDB:

When creating or updating the RetailStoreLimits table in DynamoDB, specify the composite sort key as "StoreID#Date" Ensure that your application also inserts data according to this structure.

Insert Example Data:

Insert example data into the table to represent daily discount limits for different stores and dates.



{
  "StoreID": "Store001",
  "Date": "2024-01-25",
  "MaxDailyDiscountLimit": 1000
}

Query Daily Discount Limits for a Store and Date Range:

Use Query with KeyConditionExpression to query the daily discount limits for a specific store within a date range.



from boto3.dynamodb.conditions import Key

def query_discount_limits_for_store(store_id, start_date, end_date):
    dynamodb = boto3.resource('dynamodb')
    table = dynamodb.Table('RetailStoreLimits')

    filter_expression = Key('StoreID').eq(store_id) & Key('Date').between(start_date, end_date)

    response = table.query(
        KeyConditionExpression=filter_expression
    )

    items = response.get('Items', [])

    return items

In this Python version:

We use the boto3 library.
The dynamodb.Table method is used to load the DynamoDB table named 'RetailStoreLimits'.
The KeyConditionExpression is constructed using the eq method for equality comparison and the between method for date range.
The table.query method is called with the constructed filter expression, and the results are retrieved from the response.

Now let's talk abot LSI and GSI.

In the context of NoSQL databases like DynamoDB, a Local Secondary Index (LSI) is an index associated with a table that shares the same partition key as the main table but has a different sort key. This means that data is organized differently in the index compared to the main table, enabling efficient queries based on the sort key of the LSI.

On the other hand, a Global Secondary Index (GSI) is an independent index of the main table, with its own partition key and, optionally, a different sort key. Unlike LSIs, a GSI does not share the partition key of the main table.

Exercise #3 – Products in Retail store system:

Let's suppose we are designing a NoSQL database for a retail store system. In this scenario, our goal is to manage products and, specifically, perform efficient queries on products based on their category and popularity.

Specific Requirements:

Record information about products, including their ID, name, category, stock quantity, and the number of times they have been viewed by users.
Enable the query of products by category and the retrieval of the most popular products overall.

Possible Solution to Exercise #3:



Table Name: RetailProducts
Primary Key: ProductID (Partition Key)
Attributes:
ProductName
Category
StockQuantity
ViewCount

To facilitate efficient queries:

Local Secondary Index (LSI):

Index Name: CategoryIndex
Partition Key: Category
Sort Key:ProductID

Global Secondary Index (GSI):

Index Name: PopularityIndex
Partition Key: ViewCount
Sort Key: ProductID

With this structure, you can efficiently query products by category using the LSI and retrieve the most popular products using the GSI.

We will be sharing an upcoming part to delve further into other concepts and techniques of data modeling for NoSQL. Additionally, we will present simple exercises to put these concepts into practice.

Being a Leader of a POD in an AWS Partner, I have assumed it as I understand Soccer game

Carlos Moreno — Mon, 27 Mar 2023 17:33:17 +0000

"A coach generates an idea, then he has to convince that this idea is the one that will accompany him to seek efficiency; then he has to find in the player the commitment that when adversity comes the idea is not betrayed. They are the three premises of a coach" (Cesar Luis Menotti - former Argentinian Soccer player and Coach)

Being a fan and admirer of the best player in history, I did not understand this phrase until recently:

"Messi still hasn't given up... we're waiting for him to win a match and the match is won by the team, not by a footballer." (Cesar Luis Menotti)

As a former Argentinian soccer player and coach, César Luis Menotti (El Flaco), have experienced firsthand the impact of teamwork on success. In any field, whether it is sports or business, effective teamwork is essential to achieving a common goal. In this article, I will discuss the importance of teamwork in an architecture team for AWS Partner, drawing upon my insights from the soccer field as a spectator

Amazon Web Services (AWS) has revolutionized the IT industry with its cloud computing services. As an AWS Partner, it is crucial to develop an efficient and collaborative team to provide the best solutions for clients. An architecture team plays a pivotal role in designing, implementing, and maintaining the infrastructure that powers these solutions. Think they are the midfield of a soccer team :)

In soccer, individual talent is important, but it is the collective effort that makes the difference. A team of players who understand and complement each other's strengths can outperform a group of talented individuals who don't work well together. Similarly, in an architecture team, synergy is vital. Each team member brings unique skills and perspectives to the table, and by combining these strengths, the team can develop more innovative and robust solutions for clients.

On the soccer field, effective communication is essential to coordinate and execute plays, and sometimes communication is based on looks, signs, words or sounds. Likewise, in an architecture team, clear and concise communication ensures that everyone is on the same page. Open channels for discussions and feedback create a supportive environment where team members can collaborate, share ideas, and resolve issues promptly.

In soccer, the ability to adapt to changing circumstances is crucial for success. In an architecture team, embracing change is equally important. The technology landscape is constantly evolving, and AWS Partners must be agile to stay ahead. A cohesive team can quickly respond to new challenges and opportunities, ensuring that their solutions remain cutting-edge.

A successful soccer team is built on trust, with each player confident that their teammates will fulfill their responsibilities. In an architecture team, trust and accountability are just as important. Team members must be able to rely on each other to deliver high-quality work on time. When trust is established, team members feel empowered to take ownership of their tasks and contribute their best efforts.

In soccer, a team with a shared vision is more likely to achieve their goals. Similarly, an architecture team with a common understanding of the project's objectives and the client's needs will be better equipped to deliver successful solutions. Aligning the team's goals fosters a sense of unity and commitment, motivating members to collaborate and support one another.

“You can stop running, or stop playing for long minutes; the only thing that cannot be stopped from doing is thinking.” (Cesar Luis Menotti)

Private connectivity to Amazon S3

Carlos Moreno — Sun, 26 Mar 2023 18:30:58 +0000

I'm excited to share with you about Amazon S3's new capability for simplifying private connectivity from on-premises networks: https://aws.amazon.com/about-aws/whats-new/2023/03/amazon-s3-private-connectivity-on-premises-networks/

Source image: https://docs.aws.amazon.com/AmazonS3/latest/userguide/privatelink-interface-endpoints.html

Virtual Private Cloud (VPC) interface endpoints for Amazon S3 now offer private DNS options that can help you more easily route S3 requests to the lowest-cost endpoint in your VPC.

With this new feature, your on-premises applications can use AWS PrivateLink to access S3 over an interface endpoint, while requests from your in-VPC applications access S3 using gateway endpoints. This helps you take advantage of the lowest-cost private network path without having to make code or configuration changes to your clients.

Imagine you work for a financial institution that has a hybrid cloud environment. Your organization has on-premises applications that need to access data stored in Amazon S3. However, you want to ensure that these requests are routed through a private network path to improve security and reduce data transfer costs.

With the new private DNS option for S3 interface endpoints, you can easily create an inbound resolver endpoint in your VPC and point your on-premises resolver to it. Then, you can enable private DNS for S3 interface endpoints and select "Enable private DNS only for inbound endpoint." This will ensure that requests from your on-premises applications are automatically routed to the lowest-cost endpoint over a private network path using AWS PrivateLink.

By using this capability, your organization can improve security by ensuring that requests to S3 are routed through a private network path rather than over the public internet. Additionally, you can save money on data transfer costs by automatically routing requests to the lowest-cost endpoint.

In summary, this new Amazon S3 capability is a great solution for organizations that have on-premises applications that need to access data stored in S3. By using private DNS for S3 interface endpoints, you can improve security, reduce data transfer costs, and ensure that requests are routed through a private network path.

This new capability has many potential use cases, including:

Hybrid Cloud: Organizations with on-premises applications can now more easily access S3 resources using AWS PrivateLink, while taking advantage of the lowest-cost private network path.
Cost Optimization: By automatically routing requests to the lowest-cost endpoint, organizations can save money on data transfer costs.
Security: Using private DNS for S3 interface endpoints improves security by ensuring that requests are routed through private network paths rather than over the public internet.

Overall, this new capability for Amazon S3 simplifies private connectivity from on-premises networks and offers several benefits to organizations. It's available now in all AWS Commercial Regions, and you can enable it using the AWS Management Console, AWS CLI, SDK, or AWS CloudFormation. To learn more, read the Amazon S3 documentation.

YES, the world as we know it is already changing

Carlos Moreno — Fri, 24 Mar 2023 18:45:44 +0000

In a world that continues to change after COVID-19, Global Banking Crisis is possible, Chat GPT releases its most powerful version - GPT4, Github releases Github Copilot, Google releases Google Bard and enters the generative AI competition.

In 2019 it seemed far away, today it is already simple and necessary.

As a Cloud Solution Architect I am wondering what I should do so that the wave does not wash over me.

I would like to share my perspective on how generative AI might impact our role as a Cloud Solutions Architect. Overall, I believe the role will evolve rather than be threatened, as AI can complement and enhance the tasks performed by a Solutions Architect. But it is a fact that the change will be disruptive.

We need to learn about AI, and learn as quickly as possible to increase our chances.

Here are some recommendations on how Solutions Architects can improve their role with the help of AI:

Embrace automation: AI can automate repetitive tasks, such as infrastructure provisioning and monitoring. By integrating AI-powered tools into your workflow, you can increase efficiency and focus more on higher-value tasks like solution design and optimization.
Upskill and reskill: Stay updated on the latest advancements in AI, cloud technologies, and data science. Enhance your skills in areas where AI can be integrated, such as machine learning, big data analytics, and natural language processing.
Collaborate with AI: Use AI-driven tools to augment your decision-making process. AI can help you identify patterns, anomalies, and correlations in complex data sets that are otherwise difficult for humans to process. This can enable you to make better decisions for cloud architecture design and optimization.
Leverage AI in optimization: AI can be used to optimize cloud resource usage, cost, and performance. By integrating AI-powered tools in your workflow, you can provide more effective cloud solutions that meet the specific needs of your clients.
Innovate with AI: As a Solutions Architect, you can be at the forefront of incorporating AI into the services and products you design. This can help you stay ahead of the competition and create more value for your clients.
Adapt your communication skills: As AI becomes more prevalent, it's essential to communicate effectively with both technical and non-technical stakeholders. Help them understand how AI can be leveraged in cloud solutions and how it can benefit their business.

By embracing AI and staying updated with the latest developments, Cloud Solutions Architects can ensure that their role evolves and becomes even more valuable in the future.

I even think that becoming an AI Specialist focused on robots that build Cloud Architectures and Solutions can be a viable way for a Solutions Architect to evolve their role, as it allows them to combine their existing expertise with emerging AI technologies. This specialization could involve the following aspects:

Develop AI-driven tools: As an AI Specialist, you can work on developing AI algorithms and tools that can automate the process of designing, deploying, and managing cloud architectures. This can include resource allocation, cost optimization, and monitoring.
Collaborate with robotics: Some cloud architecture tasks might involve physical infrastructure components, like data centers. If robots are utilized in these environments, a Solutions Architect with AI specialization can help design and implement robotic systems that work seamlessly with cloud infrastructure.
Improve decision-making: By focusing on AI, a Solutions Architect can enhance their decision-making skills by leveraging AI-driven insights to make better-informed choices in designing and optimizing cloud architectures.
Training and fine-tuning AI models: In this role, you'll be responsible for training AI models that can understand and optimize cloud architectures. You'll also need to ensure that these models are continuously updated and fine-tuned to accommodate changes in the technology landscape.
Enhance human-robot collaboration: As an AI Specialist, you'll be at the forefront of designing systems that enable smooth collaboration between humans and robots. This can be particularly useful in complex cloud environments where both human expertise and automation are needed.
Consult and advise: As an expert in AI-driven cloud architecture, you'll be well-positioned to consult and advise businesses on how to leverage AI and robotics for building efficient and scalable cloud infrastructure.

Scoring a Win for Cloud Transformation: Coaching a Financial Sector Company to Establish a Cloud Center of Excellence

Carlos Moreno — Fri, 24 Mar 2023 16:37:56 +0000

Hi my name is Carlos Enrique Moreno Alvarez, AWS Cloud Solutions Architect from Escala 24x7 Inc., and i like very much Baseball :)

"Baseball is 90% mental and the other half is physical".(Yogi Berra)

"In theory there is no difference between theory and practice - in practice there is" (Yogi Berra)

Reading these phrases from dear Yogi Berra, I remember what I experienced as a small part of the team that managed to establish a Cloud Center of Excellence in a company in the financial sector

First of all, I want to recommend this AWS prescriptive guide: Building a CCOE to transform the entire enterprise: https://docs.aws.amazon.com/whitepapers/latest/public-sector-cloud-transformation/building-a-cloud-center-of-excellence-ccoe-to-transform-the-entire-enterprise.html

I extract a paragraph from the guide as the basis of this article

"Building a CCoE is an iterative and continuous process that requires commitment and dedication from the organization. By following these recommendations and tailoring them to the specific needs of your organization, you can establish an effective and successful CCoE."

In the world of sports, the greatest teams continually adapt to stay ahead of the competition. Similarly, a company in the financial sector with a strong legacy culture needed to embrace change to remain competitive. I had the opportunity to start an advising a FSI company through the process of transforming its IT organization by establishing a Cloud Center of Excellence (CCoE).

This is the play-by-play account of our game plan for achieving cloud transformation success.

Before starting any game, a coach must first understand the strengths and weaknesses of their team. We analyzed the company's existing organization structure (with certain focus in IT) and identified areas that could benefit from the agility and scalability of cloud technology.

To get the team onboard, we presented the benefits of cloud adoption as a winning strategy. We envisioned a future where the company could respond more quickly to market changes, reduce costs, and foster innovation.

Just like recruiting star athletes (scouting), we gathered a group of dedicated Cloud Champions or Cloud Tiger Team committed to leading the company towards cloud transformation. These individuals were ready to embrace new technology and drive change within the organization.

To ensure our CCoE's success, we created a solid playbook outlining best practices, guidelines, and governance structures. This playbook would serve as the foundation for our team's cloud transformation journey.

During the early stages of the game, we had to maintain a balance between the old and the new. By implementing a hybrid infrastructure, we allowed the company to continue its operations while gradually embracing cloud technology. This was like playing with a mixed team of experienced veterans and promising rookies, ensuring a smooth transition to the new game plan.

Training and education were key components of our strategy. We provided comprehensive coaching sessions to the IT staff, equipping them with the knowledge and skills necessary to effectively manage the new cloud environment. This was like teaching athletes new techniques and strategies to improve their game.

To measure our progress and ensure we were on the path to victory, we established Key Performance Indicators (KPIs). These allowed us to monitor the success of our CCoE, identify areas for improvement, and celebrate achievements.

In any game, there will be challenges and setbacks. We faced resistance from individuals hesitant to embrace change. By addressing their concerns and demonstrating the benefits of cloud adoption, we gradually won over even the most reluctant team members.

As the company increasingly adopted cloud technologies, we began to see the results of our hard work. Processes became more efficient, costs were reduced, and the company's ability to innovate skyrocketed. It was like witnessing a struggling team transform into champions, ready to take on any challenge.