GenAI Developer Pro exam is coming with big, loud steps and instead of reading through AI slop on SkillBuilder that even ChatGPT can't comprehend, I decided to learn through experience by building. My goal is to create an AI Agent that will act as IT support for a fake company Acme. As the first step I would like to create a Bedrock Knowledge Base that will be connected to Atlassian Confluence. That way we will prepare our support agent to act as a simple chatbot to find relevant troubleshooting guides and company policies stored on Confluence. This way when a colleague comes to the chat interface and asks "How can I connect to company VPN", the chatbot will search through all the documents stored in Confluence and give an answer that for example "You need to use WireGuard, generate key pair, provide public key to IT and they will send back you a configuration, here's how to do it step by step...".
You can find the code on GitHub: ppabis/it-agent-bedrock.
Creating Confluence
Atlassian gives you some small free tier of their cloud services (which includes JIRA, Trello, Confluence, Bitbucket). Just sign up for an account if you don't have one (if you ever used any of the above services, you already have an Atlassian account, so you just need to create Confluence instance). Visit atlassian.com and after you have an account proceed to Confluence. Click "Get it free" at the top and choose your <myname>.atlassian.net domain. Once you are onboarded, you can visit that new address any time and switch to the Confluence by clicking on the top left (4 squares) and selecting "Confluence".
Customize the wiki however you want. I won't guide you here how to use Confluence as it's a large software on it's own but explore it and create some Pages that you can reference later with a chatbot. I asked ChatGPT to generate me some IT documents, some more general, some more technical and created some pages. In one Page I also "hid" where the IT room is so I wonder if our Knowledge Base will be able to retrieve it from an unrelated document.
Creating credentials
After you have already played around with Confluence we will generate an API key. As an administrator of the Atlassian organization go to your personal account settings in the top-right corner, then Security and Manage API tokens. It will likely ask you for 2FA.
Next select "Create API token". I tried using the one with scopes but with the setup I have but I believe it works only with OAuth and not Basic HTTP credentials I plan to use. For production use cases, you should definitely use OAuth but for this demo, we will just rely on the key that has full permissions on your account. You can read more about OAuth setup here.
Base infrastructure
We are going to build our infrastructure in OpenTofu. I will first import two providers: aws and null and create some variables and a new secret in Secrets Manager.
terraform {
required_providers {
aws = {
source = "hashicorp/aws"
version = "~> 6.0"
}
null = {
source = "hashicorp/null"
version = "~> 3.0"
}
}
}
provider "aws" { region = "eu-central-1" }
variable "confluence_username" { type = string }
variable "confluence_password" { type = string }
resource "aws_secretsmanager_secret" "confluence" {
name = "confluence"
description = "Confluence credentials"
recovery_window_in_days = 7
}
resource "aws_secretsmanager_secret_version" "confluence" {
secret_id = aws_secretsmanager_secret.confluence.id
secret_string = jsonencode({
username = var.confluence_username
password = var.confluence_password
})
lifecycle { ignore_changes = [secret_string] }
}
You can set it up in the following way: the e-mail can be stored in terraform.tfvars safely: this is the e-mail you are registered in Atlassian as. For the API token I recommend running the following Bash commands, pasting the token and only then applying the infrastructure. Because I added ignore_changes to the secret version, Terraform will not ask you again for the token and it won't be stored anywhere.
read -s TF_VAR_confluence_password # after running immediately paste the token and press enter
export TF_VAR_confluence_password
tofu apply
export TF_VAR_confluence_password="undefined" # So it doesn't ask again
That way we have stored credentials in AWS Secrets Manager so that later on, Bedrock will be able to use them for crawling.
Creating OpenSearch backend
As our underlying storage for processed text chunks from the wiki we need to use OpenSearch Serverless. Unfortunately, it is a costly service that you pay even for not using (and pay a lot for a serverless offering) so be cautious. I don't know if AWS will ever support S3 Vector Buckets that are way more reasonably priced. Let's start by defining an OpenSearch policies because without them, you will have hard time creating anything in this service 🙄. Because encryption policy has to be defined before creating the collection, it cannot depend on it so we will define the new collection name in advance in locals (unless you already have something in place like encryption policy with *). Apply this so we can continue.
locals { collection_name = "ticketagent-collection" }
resource "aws_opensearchserverless_security_policy" "encryption" {
name = "ticketagent-encryption"
type = "encryption"
policy = jsonencode({
Rules = [
{
Resource = ["collection/${local.collection_name}"]
ResourceType = "collection"
}
],
AWSOwnedKey = true
})
}
Now after we have the encryption policy we can define the collection and other two policies: network and data access. Network defines if the collection is public and which private IPs can access it if you lock it down into VPC as well as AWS services permitted to connect and data access is something like resource policy on S3 bucket defining actions that can be taken by IAM principals.
Because we will need access just in a minute, I will open the OpenSearch collection to the public (as there's no public IP restriction list 😵💫). Also I will give my current user that executes OpenTofu permissions to manage all data within the collection. And the collection itself will be of type VECTORSEARCH and no replicas 🫠.
data "aws_caller_identity" "current" {}
resource "aws_opensearchserverless_collection" "knowledge_base" {
name = "ticketagent-collection"
description = "OpenSearch Serverless collection powering the Bedrock KB"
depends_on = [aws_opensearchserverless_security_policy.encryption]
type = "VECTORSEARCH"
standby_replicas = "DISABLED"
}
resource "aws_opensearchserverless_security_policy" "network" {
name = "ticketagent-network"
type = "network"
policy = jsonencode([
{
# SourceServices = ["bedrock.amazonaws.com"],
Rules = [{
ResourceType = "collection",
Resource = ["collection/${aws_opensearchserverless_collection.knowledge_base.name}"]
}],
AllowFromPublic = true
}
])
}
resource "aws_opensearchserverless_access_policy" "data_policy" {
name = "ticketagent-data-policy"
type = "data"
policy = jsonencode([
{
Description = "Full access"
Rules = [
{
ResourceType = "collection"
Resource = ["collection/${aws_opensearchserverless_collection.knowledge_base.name}"]
Permission = ["aoss:*"]
},
{
ResourceType = "index"
Resource = ["index/${aws_opensearchserverless_collection.knowledge_base.name}/*"]
Permission = ["aoss:*"]
},
{
ResourceType = "model"
Resource = ["model/${aws_opensearchserverless_collection.knowledge_base.name}/*"]
Permission = ["aoss:*"]
}
]
Principal = [
data.aws_caller_identity.current.arn
]
}
])
}
Creating index in OpenSearch
When you use AWS Console, there's some magic happening in the background that the index is being created in OpenSearch collection. Somehow Bedrock is not smart enough to create the collection by itself and you have to do it for it. You might think about another Terraform resource, don't you? WRONG ❌. Even though there's AWS CLI command to create and delete an index, there's no IaC construct doing it anywhere. No CloudFormation, no Terraform, no CDK, nothing. (Still think new Elastic license was so bad?) So we will use null resource to create the index using AWS CLI. For the index we need a good schema, so I will define some locals as well for better control.
locals {
index_name = "ticketagent_index"
vector_dimension = 256
vector_text_field = "chunk"
vector_metadata_field = "metadata"
vector_field = "vector"
schema = jsonencode({
settings = {
index = {
knn = true
"knn.algo_param.ef_search" : 512
}
},
mappings = {
properties = {
"${local.vector_field}" = {
type = "knn_vector"
dimension = local.vector_dimension
method = {
name = "hnsw"
engine = "faiss"
parameters = {}
space_type = "l2"
}
}
"${local.vector_text_field}" = {
type = "text"
index = "true"
}
"${local.vector_metadata_field}" = {
type = "text"
index = "false"
}
}
}
})
}
Let me explain to you what is happening above. First we configure the vector dimensions - for each text chunk (200 or 300 tokens or words), we generate a vector of size 256 using an embedding model. This vector will hold semantic meaning of this text fragment. If you didn't understand, let me send you to huggingface 🤗. Continuing, I chose my own names for each field in OpenSearch: vector for the vector (we can assume it's the primary key), chunk for the text piece from Confluence and metadata for some Bedrock info like URL of the page scanned, date and time and so on. I will turn of indexing on metadata as we will primarily do queries by vector or maybe chunk. You might want to ask what are the other things in the JSON. hnsw is vector indexing method that puts the index into memory and is fast. As we don't have that much data in Confluence we can easily use that. If we had a lot of text, we might consider ivf method because this one is memory efficient but slow. As an engine we use faiss which stands for... Facebook AI Similarity Search (very scientific name, rite?). And for comparison we use l2 which is Euclidean distance (aka literal distance between two points in a straight line). Some say cosinesimil (which is angle between two points) is more reliable but I will just stay with that 🤷. If you want to learn more about all the configurations for vector search see here.
Ok, we have the JSON for the schema, we have collection, we can now create the index with AWS CLI. It might happen that this fails if you apply it all together with the creation of the policies, and if so, try applying again. The index will be recreated each time you destroy/create the collection.
data "aws_region" "current" {}
resource "null_resource" "index" {
provisioner "local-exec" {
command = "aws opensearchserverless create-index --region ${data.aws_region.current.name} --id ${aws_opensearchserverless_collection.knowledge_base.id} --index-name ${local.index_name} --index-schema '${local.schema}'"
}
depends_on = [aws_opensearchserverless_collection.knowledge_base, aws_opensearchserverless_access_policy.data_policy, aws_opensearchserverless_security_policy.network]
lifecycle {
replace_triggered_by = [aws_opensearchserverless_collection.knowledge_base]
}
}
Bedrock Knowledge Base (and IAM Role)
Finally, we can create the actual Knowledge Base that will connect to and pull data from Confluence. It will need an IAM Role to use OpenSearch and get the credentials and we will define it first.
data "aws_iam_policy_document" "bedrock_kb_assume_role" {
statement {
effect = "Allow"
principals {
type = "Service"
identifiers = ["bedrock.amazonaws.com"]
}
actions = ["sts:AssumeRole"]
}
}
resource "aws_iam_role" "bedrock_kb_role" {
name = "ticketagent-bedrock-kb-role"
assume_role_policy = data.aws_iam_policy_document.bedrock_kb_assume_role.json
}
data "aws_iam_policy_document" "bedrock_kb_policy" {
statement {
effect = "Allow"
actions = ["secretsmanager:GetSecretValue"]
resources = [aws_secretsmanager_secret.confluence.arn]
}
statement {
effect = "Allow"
actions = ["kms:Decrypt"]
resources = ["*"]
condition {
test = "StringLike"
variable = "kms:ViaService"
values = ["secretsmanager.${data.aws_region.current.name}.amazonaws.com"]
}
}
statement {
effect = "Allow"
actions = [
"bedrock:InvokeModel",
"bedrock:StartIngestionJob",
"bedrock:CreateDataSource",
"bedrock:GetDataSource",
"bedrock:DescribeKnowledgeBase",
"bedrock:Retrieve"
]
resources = ["*"]
}
statement {
effect = "Allow"
actions = ["aoss:APIAccessAll"]
resources = [aws_opensearchserverless_collection.knowledge_base.arn]
}
}
resource "aws_iam_role_policy" "bedrock_kb_policy" {
name = "ticketagent-bedrock-kb-policy"
role = aws_iam_role.bedrock_kb_role.id
policy = data.aws_iam_policy_document.bedrock_kb_policy.json
}
Next we need to allow this role in OpenSearch'es Data Policy. Let's go back to it and add another principal.
resource "aws_opensearchserverless_access_policy" "data_policy" {
name = "ticketagent-data-policy"
type = "data"
policy = jsonencode([
{
...
Principal = [
data.aws_caller_identity.current.arn,
aws_iam_role.bedrock_kb_role.arn
]
}
])
}
Now we can create the Knowledge Base and data source that will use Confluence. We need to also choose an embedding model. I will simply use Titan v2. I will refer to all the locals we defined during index creation so that every value matches.
locals {
embedding_model_arn = "arn:aws:bedrock:${data.aws_region.current.name}::foundation-model/amazon.titan-embed-text-v2:0"
}
resource "aws_bedrockagent_knowledge_base" "confluence" {
depends_on = [null_resource.index]
name = "ticketagent-confluence-kb"
role_arn = aws_iam_role.bedrock_kb_role.arn
description = "Confluence knowledge base backed by OpenSearch Serverless."
knowledge_base_configuration {
type = "VECTOR"
vector_knowledge_base_configuration {
embedding_model_arn = local.embedding_model_arn
embedding_model_configuration {
bedrock_embedding_model_configuration {
dimensions = local.vector_dimension
embedding_data_type = "FLOAT32"
}
}
}
}
storage_configuration {
type = "OPENSEARCH_SERVERLESS"
opensearch_serverless_configuration {
collection_arn = aws_opensearchserverless_collection.knowledge_base.arn
vector_index_name = local.index_name
field_mapping {
vector_field = local.vector_field
text_field = local.vector_text_field
metadata_field = local.vector_metadata_field
}
}
}
}
In data source we need to provide the Secrets Manager's secret and URL of the Atlassian organization you have created at the beginning. That should be something like <my_org>.atlassian.net.
variable "confluence_instance_url" {
type = string
description = "Base URL for the Confluence Cloud site (e.g. https://org.atlassian.net)."
}
resource "aws_bedrockagent_data_source" "confluence" {
name = "ticketagent-confluence-source"
knowledge_base_id = aws_bedrockagent_knowledge_base.confluence.id
data_source_configuration {
type = "CONFLUENCE"
confluence_configuration {
source_configuration {
host_url = var.confluence_instance_url
host_type = "SAAS"
auth_type = "BASIC"
credentials_secret_arn = aws_secretsmanager_secret.confluence.arn
}
}
}
}
Syncing and testing the knowledge base
You can now sync the knowledge base and test retrieval, either using AWS Console or AWS CLI. You can also put some filters when querying. For example I want to just retrieve chunks of type Page and max results of 3.
aws bedrock-agent start-ingestion-job \
--knowledge-base-id $(tofu output -raw knowledge_base_id) \
--data-source-id $(tofu output -raw data_source_id | cut -d, -f1)
aws bedrock-agent list-ingestion-jobs \
--knowledge-base-id $(tofu output -raw knowledge_base_id) \
--data-source-id $(tofu output -raw data_source_id | cut -d, -f1) \
--query 'ingestionJobSummaries[].[ingestionJobId,status]' \
--output table
# Wait until all jobs are COMPLETE
aws bedrock-agent-runtime retrieve \
--knowledge-base-id $(tofu output -raw knowledge_base_id) \
--retrieval-query "text=How to connect to VPN"
# Responds with some JSONs
# Add filtering and max results
FILTER_JSON='{
"vectorSearchConfiguration": {
"numberOfResults": 3,
"filter": {
"equals": {
"key": "x-amz-bedrock-kb-category",
"value": "Page"
}
}
}
}'
aws bedrock-agent-runtime retrieve \
--knowledge-base-id $(tofu output -raw knowledge_base_id) \
--retrieval-query "text=How to connect to VPN" \
--retrieval-configuration "$FILTER_JSON"
Going further
In the next part we will implement a helpful agent that will answer our queries and maybe create a first tool for this agent to take some actions. I believe your today's adventure with OpenSearch Serverless was more than enough.
Top comments (0)