DEV Community: Luis Parraguez

Do You need an AI ASSISTANT? Let’s build it using Amazon Q! (Part 2)

Luis Parraguez — Sat, 03 Feb 2024 23:31:08 +0000

In my previous post, I shared how to implement an AI assistant using the features available in the recently launched Amazon Q for Business service.

Overall scope was to build that AI assistant to take advantage of a Knowledge Base including information from the internet, document repositories and uploaded relevant documents.

Until that moment, we had our AI Assistant already operating for internal use. We need now to take the next step to DEPLOY the AI Assistant to our organization so we can have multiple users leveraging its capabilities with the required security measures in place.

Let’s see now the process and lessons learned during the deployment of an AI Assistant using Amazon Q for Business!

Integrating our AI Assistant with an Identity Provider

The first step that we need to follow to deploy the Amazon Q web experience, web user interface that we are using to interact with our AI Assistant, is to integrate it with a system/service that will authenticate and authorize the access of users in our organization.

Similarly to the various alternatives that are available to integrate with data sources, Amazon Q offers various alternatives of an identity provider (IdP) that’s compliant with SAML 2.0.

The identity providers currently supported by Amazon Q include AWS IAM Identity Center, Okta, Microsoft EntraID, and PingIdentity.

Amazon Q acts as a service provider (SP) that requests user authentication and authorization from an identity provider (IdP). The IdP authenticates the user’s identity and provides attributes about the user to Amazon Q. Amazon Q then authorizes the user’s session based on these attributes.

Security Assertion Markup Language (SAML) is used to transfer user identity data between the IdP and Amazon Q in a standardized way. Some key points about integrating an IdP with Amazon Q:

Authentication confirms a user’s identity by verifying they are who they say they are;
Authorization allows users certain permissions or access to resources;
The IdP stores, manages and verifies user identities for applications like Amazon Q;
Amazon Q uses service-initiated single sign-on (SSO) to authenticate users. IdP-initiated SSO is not supported.

For our use case, I decided to use AWS IAM Identity Center as our Identity Provider. We need to follow preparation steps to proceed with the integration.

Enabling an IAM Identity Center instance

We need to enable IAM Identity Center using the service console:

At this point there is an important decision to be taken related to the type of instance for IAM Identity Center: organization instances or account instances. Let’s briefly understand the characteristics of each one:

Organization instances of IAM Identity Center: When we enable IAM Identity Center in conjunction with AWS Organizations, we are creating an organization instance of IAM Identity Center. An organization instance is the primary method of enabling IAM Identity Center as it provides support for all features of IAM Identity Center including managing permissions for multiple AWS accounts in your organization and assigning access to customer managed applications.
Account instances of IAM Identity Center: Account instances are bound to a single AWS account and are used only to manage user and group access for supported applications in the same account and AWS Region. Supported applications are AWS managed applications and OIDC-based customer managed applications. OpenID Connect (OIDC) is a standard for identity federation.

Looking at a summary of the capabilities available for each instance type:

Based on the previous definitions, we need to select an organization instance of IAM Identity Center due to, at least, two reasons: (1) We need to be able to integrate with Customer-managed applications (that is the case of our Amazon Q for business application) and (2) We need that the authentication and authorization process uses the SAML standard. Both requisites are not supported by an Account instance of IAM Identity Center.

If you hadn’t created an AWS organization before, the process to enable an organization instance will automatically create an AWS Organization in the background assigning the account that we are using as management account.

After the creation of the organization instance, we need to confirm our identity source. The identity source is where we administer users and groups, and it is the service that authenticates our users. By default, IAM Identity Center creates an Identity Center directory. For our use case we will use this default option.

Note in the previous image that there is an AWS access portal URL being shown here. It’s important to highlight that our AI Assistant will appear in that portal, but we won’t be able to access it using the portal. This is because IdP-initiated SSO is not supported by Amazon Q. We will see later how we will access our AI Assistant.

Moving forward, we need to create at least one valid user with a valid e-mail address and, optionally but highly recommended, create groups and assign users to them. Users and/or groups will be used later to enable access to our AI assistant.

For our use case, we created one group with one user to test the access, including the activation of MFA as a security measure already available as part of the process:

At this moment, we are ready to begin the integration of our AI Assistant with IAM Identity Center. To start the process, we need to come back to the Amazon Q console and “Edit” the Web Experience settings:

Following the steps detailed in Setting up Amazon Q with IAM Identity Center as identity provider — Amazon Q, we will complete the configuration of both Amazon Q and its integration with IAM Identity Center, executing carefully a series of steps in both services console.

At the end of this process, in IAM Identity Center, we will have our AI Assistant application configured with the assignment of authorized group/user, as you can see in the following image:

On the side of Amazon Q, in the Web experience settings section we will have the information about the service role created to authorize Amazon Q to provision the resources required for the deployment and mainly the Deployed URL that is the URL that the users will use to access our AI assistant application!!

Now we are ready to test the access of our user to our AI assistant application. To do that, we can click on the Deployed URL and start the process. As you can see in the following image, the integration with IAM Identity Center will be actioned and we will need to sign-in into the application:

And take advantage of MFA to enhance the security to our application:

Completing this process, we will get authorized access to our AI assistant and the user is READY TO GO to explore use cases that we described in the previous post!!

At this point, we have accomplished the objective to deploy our AI assistant using a WEB EXPERIENCE user interface with access enabled for our authorized user.

As from here, defining what will be our actual team of users, we can extend the configuration of IAM Identity Center and, through the proper communication channels and in compliance with the corporate policies, release access to our AI assistant for a broader group.

In case your organization has different identity providers, like Microsoft Entra ID (formerly Azure Active Directory) or others, you can also perform the required configuration as needed.

As in the case of data, I understand that these integration capabilities of Amazon Q with different Identity providers helps to mitigate adoption barriers and contributes to fulfill the strong expectation of organizations to have a secure access to the AI assistant that will provide access to very valuable corporate information and documentation.

Now that we have completed this deployment step, what could be the next step?

What about integrating our corporate applications and/or platforms with the AI assistant not through a web interface but through APIs?

This could be a very interesting example of the utilization of our AI assistant’s insights to enhance the execution of our Business/IT processes and our team’s collaboration, don’t you agree?

Let’s meet again in our next post and please feel free to share your feedback and comments!

Do You need an AI ASSISTANT? Let's build it using Amazon Q! (Part 1)

Luis Parraguez — Tue, 23 Jan 2024 00:30:28 +0000

Generative AI is a branch of artificial intelligence that can create new content or data from scratch, such as text, images, audio, or video. It is powered by deep learning models that learn from large amounts of data and generate novel outputs based on a given input or context. One of the most exciting applications of generative AI is building AI assistants that can interact with customers or employees in natural language, provide personalized information or recommendations, and automate tasks or processes.

We are already seeing in the market the launch of AI assistants, using a chat bot experience, that are built to answer questions using natural language using as source of information our own personal or company documents, extending the possibility of use the internet as the information source. In this way, we can have specialized/customized answers for our questions applying a personal/business context allowing us to get more precise and specific insights.

To implement an AI Assistant there are key functional and technical steps that we need to follow:

Define the use case and the target audience: The first step is to identify the specific problem or opportunity that the AI Assistant will address, and the characteristics and needs of the customers or employees who will use it. Let’s define as sample use case that our need is to have an AI Assistant that will be able to provide us with information about two domains: FSI industry trends and AWS services.

Collect and prepare the data: The second step is to gather and process the data that will be used to enrich the answers and evaluate the generative AI Assistant. The data should be relevant, diverse, and high-quality, and should reflect the domains and the context of our use case. For our sample use case, let’s assume that the requirement is to include information from the internet, document repositories and to be able to add ad-hoc relevant documents.

Choose and deploy the generative AI platform: The third step is to select and implement the generative AI platform that will enable the creation and management of the AI Assistant. The platform should be scalable, secure, and easy to use, and should provide tools and features for data ingestion, testing, and deployment. For our use case, we selected Amazon Q for Business as the platform to build our AI Assistant.

Design, build and test the AI Assistant: The fourth step is to design, build and test the AI assistant, ensuring that it meets the functional and non-functional requirements, such as performance, accuracy, reliability, usability, and ethics. The AI assistant should be evaluated and validated by real users, and feedback should be collected and incorporated.

Let’s see now the process and lessons learned during the creation of an AI Assistant using Amazon Q for Business.

First, what is Amazon Q for Business?

Amazon Q for Business is a fully managed, generative AI-powered enterprise chat assistant created by Amazon. It allows organizations to deploy an AI agent within their company to enhance employee productivity. Amazon Q for Business is tailored specifically for organizational use by allowing administrators to connect internal systems and limit access based on user permissions and groups. This ensures employees only get information relevant to their roles from trusted sources within the company.

Let’s start then with the creation of our AI Assistant!

I will show the AI Assistant creation process using the AWS Console, and that assumes that you have an AWS account and an IAM user with administration privileges required for the provision of the resources along the process. Alternatively, it is also possible to execute this process using the AWS CLI.

To create our AI Assistant, we need to create first an Amazon Q Application.

Let’s sign into the AWS Management Console and open the Amazon Q console using the administrative user previously created.

We need first to configure the Amazon Q Application initial settings:

Note in the image above:

Service Access Role: IAM role for Amazon Q to allow it to access the AWS resources it needs to create our application. We can choose to use an existing role or create a new role. The policy associated with this role will also allow Amazon Q to publish information to CloudWatch Logs that will allow us to monitor the data ingestion processes.
KMS Encryption Key: Amazon Q encrypts our data by default using AWS managed KMS keys, option selected for this use case. Alternatively, you can use a customer-managed key.
Not shown above but requested during the creation process, it is highly recommended to include Application Tags to identify all the resources linked to our Amazon Q application. As an example, in our case, we used the combination “Created_By | bbold-amazonq-app” for all the resources created.

Next, we need to create and select a Retriever for our Amazon Q application. An Amazon Q retriever determines where an Amazon Q conversational agent will search for answers to users' questions. When creating an Amazon Q application, a retriever must be selected. The retriever connects the application to external data sources containing information that can be used to respond to questions. There are two different types of retrievers that can be chosen:

Native retriever: Allows connecting directly to data sources like knowledge bases, documentation repositories, or databases using Amazon Q data connectors.
Amazon Kendra retriever: Connects to an existing Amazon Kendra index to query its data. Amazon Kendra is an intelligent search service powered by machine learning. Amazon Kendra allows organizations to index documents from multiple sources and provide a unified search experience for internal information.

Let’s see the key Retriever settings:

Note in the image above:

Retriever: We chose "Use native retriever" to build an Amazon Q retriever for our Amazon Q application. When we select the native retriever for an Amazon Q application, Amazon Q will create an index to connect to and organize the data sources configured for the application. While the native retriever index is not an Amazon Kendra index, both serve a similar purpose of housing and organizing content for retrieval. The main difference is the native retriever index is managed internally by Amazon Q, whereas a Kendra index would be a separate service integration.
Index provisioning: When creating an Amazon Q application, the index provisioning number of units refers to the storage capacity allocated for the application's index. Each unit in the index corresponds to 20,000 documents that can be stored. Each storage unit includes 100 hours of connector usage per month. The first storage unit is available at no charge for the lesser of 750 hours or 31 days. We selected 1 storage unit to try out Amazon Q without incurring charges during the initial evaluation period.
Similarly, we included Retriever Tags and Index Tags, using the same combination of the Application tags.

Now, we are ready to connect with our data sources! Amazon Q application data sources are repositories of information that can be connected to an Amazon Q application to power the conversational agent's responses. When a user asks a question through the Amazon Q chat interface, the system will search across connected data sources to find relevant answers and responses.

Using the Amazon Q retriever, we can select several cloud and on-premises data sources, as we can see in the following images:

From my point of view, this a VERY RELEVANT CAPABILITY of Amazon Q for Business, as the power of our AI Assistant will depend on the variety and quality of the information sources that we will connect, and for that it is very critical that we allow the organizations to leverage their information with security where that information is currently available. This will also contribute a lot to eliminating adoption barriers and reduce implementation time avoiding non-priority information migration efforts.

When we are talking, for example, about business, technical and other types of business documents, we will find them frequently stored in repositories like S3, SharePoint, OneDrive, or Google Drive. If we talk about technical/code repositories, they will typically be stored in a GitHub repository. The GOOD NEWS is that ALL these data sources are already covered by Amazon Q!

Remembering the requirements for our use case: “…include information from the internet, document repositories and to be able to add ad-hoc relevant documents.”, I was able to support this demand using the “Most Popular” data sources shown above. Let’s look at each one of them:

REQ #1: Including information from the Internet

To include information from the web, we will add a data source of type WEB CRAWLER. An Amazon Q Web Crawler connector crawls and indexes either public facing websites or internal company websites that use HTTPS. When selecting websites to index, we must adhere to Amazon Acceptable Use Policy and crawl only our own web pages or web pages we have the authorization to index.

Now let’s follow the typical process to add a data source: We start by specifying the name of our data source and the source of the URLs that we want Amazon Q to index. As you can see in the image below there are multiple options, from specifying an URLs list directly in the console to specify Sitemaps (Sitemaps contain lists of URLs that are available for crawling on a site to help crawlers comprehensively retrieve and index content).

For our use case, I decided to use the option to provide a Source URLs file stored in an S3 Bucket, as we will use the same bucket also as complementary data source later. The Source URLs file is just a text file that includes one URL per line. We can include up to 100 starting point URLs in the file. As example, we included in the file a list of URLs from our own website:

After we define the list of URLs, we need to configure the security aspects of our connection to them. For that Amazon Q offers several authentication alternatives. For this example, we will use “No Authentication” as we are crawling a public website.

In case you need to index internal websites, it also offers the option to use a Web Proxy and configure the authentication credentials using AWS Secret Manager.

In terms of security also, for selected data source types, Amazon Q provides optionally the alternative to configure a VPC and security group that will be used by Amazon Q data source connector to access your information source (Examples: access an S3 bucker or a database through a specific VPC). Since our data source is accessible from the public internet, we didn't need to enable the Amazon VPC feature.

Next, we need to configure the IAM Role for Amazon Q to access our data source repository credentials and application content. Being the first time that we are creating this data source, the recommendation is to create a new service Role and provide a name for it:

In the background, it will create an IAM role and a Customer Managed Policy to allow Amazon Q to access the S3 bucket where the source URLs file is stored, access authentication credentials stored in AWS Secret Manager (if applicable), manage the processing of the documents to be ingested and store information related to groups and users access to those documents.

Moving on, we need to define the Synchronization Scope for the documents to be ingested. We can define a sync domain range as seen in the following image:

For our use case, we selected the option “Sync domains with subdomains only” to prevent the scenario of indexing other third-party websites potentially linked from our company website.

Additionally, Amazon Q also offers very interesting additional options regarding the scope of documents to be indexed (“Additional Configuration”). I would like to highlight some aspects of them:

“Scope settings | Crawl depth”: The number of levels from the seed URL that Amazon Q should crawl. This parameter is important to ensure that all the webpages that we need to be indexed are effectively included. Recommendation here is to review the structure of the websites that you are planning to index and see how many levels you need to include. For example, a crawl depth of “3” means that the search engine will crawl up to 3 levels deep from the seed URL. For example: Seed URL (level 1), Pages directly linked from the seed URL (level 2) and Pages directly linked from level 2 pages (level 3).
“Scope settings | Maximum file size”: Where you can define the maximum file size of a webpage or attachment to crawl. You need to calibrate this parameter based on your knowledge of the documents sizes that are in your knowledge base.
“Include files that web pages link to”: When this option is selected in Amazon Q, it means that the crawler will index not only the content of the web pages specified in the seed URLs, but also any files that are linked from those web pages. This allows the full content being referenced from the web pages to be searchable (Some examples of files that may be linked from web pages include documents like PDFs, images, videos, audio files, and other attachments).
“Crawl URL Patterns” and “URL Pattern to Index”: Both parameters will help you to filter the scope of information to crawl and then to index. Amazon Q will index the URLs that it crawled based on the crawl configuration. The “crawl URL patterns” specify which URLs should be crawled to the specified crawl depth starting from the seed URLs. The “URL patterns to index” configuration can further target which of the crawled URLs should be indexed and searchable.

Once we are done with defining the sync scope and filters, we need to work with the configurations related to the synchronization jobs execution. For that we need to configure how and when we need those processes to run:

As you can see above, for the how to sync (Sync Mode) we can choose from make a full synchronization or only sync the updates. We selected the second option for our use case. In terms of the schedule (Sync run schedule), you select the frequency for the synchronization, from hourly to monthly, a custom time or On Demand. For our testing, we selected this last option.

Similarly to what we did for the Amazon Q Application, Retriever and Index, we can also specify Tags for the Data Source. Again, it’s highly recommended that you include tags to identify all resources that are related to your Amazon Q application for cost management purposes.

Finally, Amazon Q shows the Fields Mapping section. Amazon Q crawls data source document attributes or metadata and maps them to fields in your Amazon Q index. Amazon Q has reserved fields that it uses when querying your application. It shows the list of default attributes mapped for both web pages and attachments (they can be customized after data source creation):

With this, you can finish the configuration of the new WEBCRAWLER data source with the “Add Data Source” option. Having done that, we need to execute the first synchronization job that will run in a Full sync mode independently of your configuration. After you complete this process you can see the Details, Sync History, Settings, and Tags of your Amazon Q Data Source in the console:

In the image above, would like to highlight the Sync run history section where you can see the synchronization jobs results in terms of total items scanned, added/modified, deleted, and failed where you have quantitative information to evaluate if the crawling/indexing process have processed all what you expected based on your configuration. At this point, Amazon Q also offers the possibility to retrieve log information from CloudWatch Logs using a link in the “Details” column:

As you can see above, Amazon Q creates a Log Group related to the Amazon Q application using the Application ID (“99fd2bd8-bc98-4d34-8153-54a2a3b189b3”) as identifier and creates a Log Stream for each execution using the Data source ID (“69374b88-5f35-4b3b-a0cc-0bfb2742c23a”) as identifier.

Amazon Q already creates the Query that we can execute using Log Insights, and running it, you will get the logs related to the synchronization job where you will be able to see details about the URLs processed and eventually errors that must be fixed.

REQ #2: Including information from document repositories

Having completed the configuration of a WEBCRAWLER data source, we can move forward to our second requirement where we will leverage S3 buckets with documentation to be used by our AI Assistant. We need to configure an S3 data source for each bucket we plan to index.

The configuration of this data source includes very similar steps to the ones we already saw for the WEBCRAWLER, including data source name, configuration of utilization of a VPC or not, IAM role creation, Sync scope, mode and run schedule, tags and finally field mapping.

Same as in the case of the WEBCRAWLER data source, in this case you can also use Log Insights to get the logs related to the synchronization job where you will be able to see details about the documents processed and eventually errors that must be fixed.

REQ #3: Including information from add ad-hoc relevant documents

While I recommend storing and organizing your documentation in an S3 bucket for durability, protection, and security purposes, we may need also to upload specific ad-hoc files to expand the knowledge base of our AI Assistant. For those cases we can use the FILE UPLOADER data source:

As you can see in the image above, you will be able to see in the console the list of uploaded files that will be indexed and be part of the knowledge base of our AI Assistant.

Once we have created all our required data sources, completed the initial synchronization jobs, and verified that they have finished successfully and indexed our documentation, we will be ready to TEST our AI assistant! For that AWS has already built a WEB EXPERIENCE which is a web interface of our AI Assistant / CHAT BOT application that we can use to start interacting with it.

We can access the WEB EXPERIENCE, through the console:

And open our AI Assistant / CHAT BOT application interface:

As you can see above, we can configure some attributes like the Title (Name of the AI Assistant), Subtitle (Objective of the AI Assistant) and Welcome Message. Our suggestion is to include in the Welcome Message what are the domains or subjects that the AI Assistant will be prepared to answer questions about using the information that you provided through Data Sources. This is very important to manage expectations from the potential users of the solution.

Now, let’s see some examples of our AI Assistant in action!

SAMPLE PROMPT # 1 – Asking a question about industry trends (looking to use documents in the FILE UPLOADER): What are the key trends in 2024 for Corporate Banking?

Note that the answer includes the reference to the information source used by the Assistant to prepare the response and allows you to provide feedback regarding the quality of the response.

SAMPLE PROMPT # 2 – Asking a question about AWS services (looking to use documents in the S3 BUCKET): “Please prepare a summary of how AWS is supporting Banking Clients to improve their customers experience, including reference to customer examples and which AWS services are being used”

SAMPLE PROMPT # 3 – Asking a question about DEVOPS (looking to use documents in a WEBPAGE): “Please prepare a summary about how we can help our customers to accelerate the implementation of DEVOPS culture in their organizations”

As you can see, using Amazon Q for Business we have been able to implement an AI Assistant and as from here we can continue enriching the KNOWLEDGE BASE adding more documents to the data sources and/or adding more data sources as well as preparing our PROMPTS LIBRARY with templates that we can reuse to improve our productivity and maximize the quality of the responses obtained from the AI Assistant.

Important to highlight that those will be ON-GOING activities as we want to have our Assistant always UP TO DATE with our latest and verified documentation and we want also our team improving their productivity, knowing when and how to use the AI Assistant.

As with any other Cloud technology adoption process, it’s very critical to include a proper Organizational Change Management initiative to make sure that the team is properly engaged, communicated, and trained on this technology making sure that they understand that is a valuable tool at their disposal to gain productivity and efficiency that DOES NOT ELIMINATE the need for the “human in the loop” evaluation of the quality and applicability of the responses before their utilization to fulfill internal or customer demands.

This is a critical success factor to eliminate adoption barriers and very valuable feedback for the IT Team to be used for data source contents refinement and enrichment.

Up to this moment, we have our AI Assistant already operating for internal use. We need now to take the next step to DEPLOY the AI Assistant to our organization so we can have multiple users leveraging its capabilities with the required security measures in place.

Let's meet again in our next post and please feel free to share your feedback and comments!

Accelerating the Implementation of DevOps Culture in Your Organization with Amazon CodeCatalyst

Luis Parraguez — Fri, 21 Jul 2023 17:40:19 +0000

Good morning everyone!

DevOps culture has become increasingly popular among software development organizations as it fosters a collaborative approach between development and operations teams, enabling the continuous delivery of high quality software. However, effectively implementing the DevOps culture can be a complex challenge.

This is where the Amazon CodeCatalyst service can play a relevant role. In this post, we’ll explore how Amazon CodeCatalyst can help organizations accelerate and improve the implementation of DevOps culture by providing a complete collaboration and automation platform.

Amazon CodeCatalyst Overview

Amazon CodeCatalyst is an Amazon Web Services (AWS) service that provides an integrated platform to help teams collaborate, automate processes, and adopt DevOps culture best practices. It offers capabilities for code versioning, integration and continuous delivery (CI/CD) pipeline management, issue tracking, configuration management, and more.

Enhanced collaboration: One of the keys to the success of the DevOps culture is effective collaboration between development and operations teams. Amazon CodeCatalyst offers advanced features to promote collaboration, such as centralized code repositories, integration with communication tools (such as Slack), and code review capabilities. These features allow teams to work together efficiently, share knowledge, and collaborate on projects with ease.
Process automation: Automating processes is key to accelerating the implementation of the DevOps culture. Amazon CodeCatalyst provides comprehensive capabilities for automation, enabling you to create CI/CD pipelines to automate software build, test, and deploy. This reduces reliance on time-consuming manual processes, increasing the efficiency and speed of software delivery.
Configuration management: Configuration management is an essential part of the DevOps culture. Amazon CodeCatalyst provides capabilities to efficiently manage the configuration of infrastructure and application environments. It supports the use of popular tools, such as AWS CloudFormation and Terraform, to provision and manage infrastructure resources as code. This ensures infrastructure consistency and traceability and simplifies the management of environments at different stages of the software lifecycle.
Issue tracking: Issue tracking and effective project management are crucial to the successful implementation of the DevOps culture. Amazon CodeCatalyst provides built-in capabilities for issue tracking, allowing teams to record, prioritize, and track issues and development tasks. This improves visibility and collaboration around issues, making them easier to resolve quickly and efficiently.
Security and compliance: Security and compliance are important considerations when implementing the DevOps culture. Amazon CodeCatalyst provides capabilities for access control, security monitoring, and integration with other security-focused AWS services, such as AWS Identity and Access Management (IAM) and AWS CloudTrail. This ensures that organizations can implement appropriate security practices and meet regulatory requirements.

Sharing Lessons Learned Using Amazon CodeCatalyst

I would like to share with you lessons learned from using Amazon CodeCatalyst applied in a project that combined Multi-Cloud and DevOps requirements. The final objective of this project was the implementation of a Static Serverless Website in Multi-Cloud architecture, having as main requirements:

Implement the static website through the use of Serverless storage resources in AWS (S3), Azure (Blob Storage) and OCI (Buckets).
Create centralized repositories of infrastructure as code (IaC) and static website code, integrated with integration and continuous delivery pipelines.
Centralize automated provisioning, configuration, and management of infrastructure across multiple Clouds using Amazon CloudCatalyst and Terraform.
Centralize the management and automation of static website integration and continuous delivery pipelines across multiple Clouds using Amazon CloudCatalyst.
Communicate with AWS Cloud through native Amazon CloudCatalyst resources.
Communicate with Azure and OCI using CLIs (Command Line Interfaces) and leveraging compute resources from Amazon CloudCatalyst.

We can see the solution architecture applied in this project in the following diagram:

Let’s now look at the key steps followed in this project:

Step 1 — Preparing Infrastructure as Code repository

As a first step to using the features of Amazon CloudCatalyst we must create a “Space” (in our project called “TerraformCodeCatalystLPG”). During the creation of the “Space” we need to specify the AWS Account that will be used for billing and for creating the resources in the AWS Cloud through authorization of IAM roles.

Created the “Space” we can create a “Project” (in our project called “TerraformCodeCatalyst”):

Within the “Project” we find the two groups of functionalities required to meet the requirements of the project: “Code” and “CI/CD”.

In the figure below we can see these options on the left, including the detail of the features related to “Code”:

Source Repositories: Functionality that allows the creation and versioning control of code repositories;
Pull Requests: Functionality that allows the management of code update requests in the repositories, including support for approval/disapproval and application of updates (Merge);
Dev Environments: Functionality that allows the creation of pre-configured development environments that we can use to work with the code of our infrastructure and / or applications.

In the project, we tested the creation of development environments and checked the availability of environments with support for AWS Cloud9 (running in web browser) and Visual Studio Code as well as other options with support for JetBrains IDEs. For the project we chose to use a local Visual Studio Code environment for the reuse of previously prepared Terraform code.

Using the feature Source Repository, we created our first repository (“bootstrapping-terraform-automation for-amazon-codecatalyst”) to store the Terraform code that was used for the provisioning of our infrastructure.

Within this repository we first created a folder (“_bootstrap”) to store the code of the base infrastructure required for the operation of Terraform using an S3 Backend in AWS. This base infrastructure requires the creation of (1) an S3 bucket to store the Terraform file to control provisioned resources (terraform.tfstate), (2) a DynamoDB table to control concurrent access to the terraform.tfstate file in the case of parallel executions, and (3) IAM roles and policies required to connect Amazon CloudCatalyst to the AWS account where the resources will be created — one IAM role for Branch Main with permission to create resources and another IAM role to Branch Pull Request with read-only permission.

Step 2 — Creation of CI/CD workflows to update the Infrastructure as Code

Once the base infrastructure resources have been created, we are ready to create the CI/CD workflows to manage the update of the infrastructure as code of our application. To do this first, we must associate the two previously created IAM roles with Amazon CloudCatalyst so that we can use them in workflows.

As you can see in the figure below, we now use the “CI/CD | Workflows” functionality, selecting the code repository, to create 03 workflows in Branch Main:

“TerraformPRBranch”: Workflow that will manage the evaluation of requested updates through Pull Requests from a Branch. This workflow will perform the installation of Terraform in a virtual machine EC2 and will execute the commands Terraform init, validate e plan aiming to validate the updates made to the infrastructure code;
“TerraformMainBranch”: Workflow that will manage the automatic application of the approved updates in the code of Branch Main of our repository. In a similar way this workflow will execute the commands Terraform init, validate, plan e apply aiming to apply the updates made to the infrastructure code;
“TerraformMainBranch_Destroy”: Workflow that will manage the removal of infrastructure created through Branch Main code. This workflow is configured to run manually and will execute the Terraform init and destroy commands to eliminate the provisioned resources.

As an example, we can see below the YAML code of the workflow “TerraformMainBranch”:

# Adaptation of the https://developer.hashicorp.com/terraform/tutorials/automation/github-actions workflow
Name: TerraformMainBranch
SchemaVersion: "1.0"

# Here we are including the trigger for this workflow: Push / Pull Request. If not included then this workflow will be executed only manually
Triggers:
  - Type: Push
    Branches:
      - main

# Here we are defining the actions that will be executed for this workflow
Actions:
  Terraform-Main-Branch-Apply:
    Identifier: aws/build@v1
    Inputs:
      Sources:
        - WorkflowSource
    Environment:
      Connections:
        - Role: Main-Branch-Infrastructure
          Name: "XXXXXXXXXXXX"
      Name: TerraformBootstrap
    Configuration: 
      Steps:
        - Run: export TF_VERSION=1.5.2 && wget -O terraform.zip "https://releases.hashicorp.com/terraform/${TF_VERSION}/terraform_${TF_VERSION}_linux_amd64.zip"
        - Run: unzip terraform.zip && rm terraform.zip && mv terraform /usr/bin/terraform && chmod +x /usr/bin/terraform
        - Run: terraform init -no-color
        - Run: terraform validate -no-color
        - Run: terraform plan -no-color -input=false
        - Run: terraform apply -auto-approve -no-color -input=false
    Compute:
      Type: EC2

Step 3 — Execution of CI/CD workflows to update the Infrastructure as Code

Next, we created a Branch (“test-pr-workflow”) that was used to validate the updates to the Terraform code of our infrastructure.

The Terraform files of the application were organized into groups: the first one focused on connecting to AWS, Azure and OCI (multicloud_provider.tf and multicloud_variables.tf) and another three for provisioning the storage resources in each Cloud (Example: aws_storage.tf and aws_variables.tf). For the provisioning of this infrastructure we also used the S3 Backend previously created but storing the file terraform.tfstate in a different key of the bucket.

Using Visual Studio Code Insiders, we synchronized the Terraform files of the infrastructure with our repository in Amazon CloudCatalyst using the “test-pr-workflow” Branch.

Updated the files in the Branch “test-pr-workflow” we created a Pull Request to start the workflow “TerraformPRBranch” in this Branch. In the figure below you can see the data for creation of a Pull Request including the source Branch “test-pr-workflow” and Target Branch “main” as well as the specification of Reviewers, mandatory and optional, of the requested changes with which we are applying collaboration within the team of DevOps.

The creation of the Pull Request triggered the workflow “TerraformPRBranch” in Branch “test-pr-workflow” and after its completion we were able to verify, through the Logs of the command Terraform Plan, that the application of the updates in the infrastructure could be carried out successfully and, being this the case, we authorized Merge of updates in the Branch “main”.

By authorizing the Merge, the workflow “TerraformMainBranch” was initiated and with it were carried out the infrastructure updates defined by the Terraform code:

This demonstrated a full CI/CD cycle of automating infrastructure upgrades using Amazon CodeCatalyst!!

Step 4 — Preparation of the application code repository (Serverless Static Website in Multi-Cloud architecture)

Similar to Step 1, using the Source Repository functionality we created the repository for the code of our application (“static-website-repo”), containing the files required for the website:

Step 5 — Creation of CI/CD workflows to update the application

Following the same procedure as before, we proceeded to the creation of workflows to store the updates of the files of the static website in the buckets of each Cloud. Each of the workflows was configured to sequentially update the three specified environments: Testing, Homologation and Production, with the condition of advancing to the next environment only in case of success in the application in the previous environment.

Let’s see below the highlights in the preparation of each workflow:

“Upload_to_AWS_S3” Workflow — AWS S3 Bucket Storage

As you can see below, in the visual representation of this workflow, we have configured as an automatic trigger the code update event in the “main” branch of the repository. This trigger will start the workflow that will consist of 03 actions of type “aws/s3-publish@v1.0.5”. This action is a native feature of Amazon CodeCatalyst that allows you to upload files to an S3 Bucket by running commands from an EC2 virtual machine:

“Upload_to_Azure_Blob” Workflow — Azure Blob Storage

As you can see below, in the visual representation of this workflow, we have also configured as an automatic trigger the code update event in the “main” branch of the repository. This trigger will start the workflow that will also be composed of 03 actions (Actions).

According to project requirements, communication with Azure was performed using Azure CLI (Command Line Interface). To enable this, the technical alternative was to apply Amazon CloudCatalyst’s ability to allow the creation of the processing environment using a custom container image. The custom image was the containerized version of Azure CLI (Image: mcr.microsoft.com/azure-cli). Authentication for CLI use was performed by leveraging Amazon CloudCatalyst Secrets functionality for the security of access information.

“Upload_to_OCI_Bucket” Workflow — OCI Buckets Storage

According to the requirements of the project, the communication with OCI was also carried out using the OCI CLI (Command Line Interface) and to enable this the technical alternative applied was to run the containerized version of the OCI CLI on the EC2 virtual machine (Imagen: ghcr.io/oracle/oci-cli:latest ) using Docker commands in the version already available in the EC2 processing environment. Authentication for CLI use was also performed by leveraging Amazon CloudCatalyst Secrets functionality for the security of access information.

Step 6 — Execution of CI/CD workflows to update the application code

The update of the application code was done applying the same methodology that we applied in the case of the infrastructure as code repository (branch creation for Pull Requests). After this process was carried out and authorizing the Merge, a Push event was generated in the “main” Branch and with that the three previous workflows were triggered initiating the update of the Buckets in AWS, Azure and OCI. The images in the sequence below show the result of the execution of the workflows:

And as a result of the processing we had our Static Serverless Website in Multi-Cloud architecture up and running on AWS, Azure and OCI powered by an end-to-end DevOps Process supported by Amazon CodeCatalyst!!

Conclusion

Efficient implementation of DevOps culture is critical to the success of software development organizations. Amazon CodeCatalyst provides a comprehensive platform to accelerate and enhance this implementation.

With advanced collaboration, process automation, configuration management, and issue tracking capabilities, Amazon CodeCatalyst enables teams to collaborate more efficiently, improve speed of delivery, and ensure software quality. By adopting Amazon CodeCatalyst, organizations can drive their DevOps journey quickly and efficiently, leveraging the benefits of an agile, collaborative approach to software development.

And, as we saw in the project presented above, Amazon CodeCatalyst also has the necessary resources to work with solutions in Multi-Cloud architecture in partnership with Terraform and Docker. I suggest you experiment the solution too!!

We are available to support you in this process and in the continuity of your Cloud Journey!

Let’s meet again in our next post!