In my last article https://dev.to/aws-builders/are-you-compliant-building-an-ai-compliance-agent-with-scf-and-bedrock-agentcore-1jf1, I wrote about building a compliance agent using Bedrock Agentcore. I added more capabilities to the agent with some additional tools this time around. (https://github.com/mgbec/Secure-Controls-Framework-Agent)
Frontend
I’ve added a local frontend, with the same capabilities as the agent. Eventually I would want to have something available to other users with authentication, scaling, and more. There is a plan for hosted deployment at frontend-deployment.md. It covers:
- Cognito User Pool (MFA, strong passwords, email recovery)
- ECS Fargate hosting the Streamlit container
- ALB with Cognito authentication on the listener
- CloudFront for HTTPS/CDN
- Full Terraform snippets for each component
- Security checklist
The current local frontend looks like this:
Approved Answers and Auditable Versioning
We could have our agent get approved answers for filling out compliance questionnaires. There is a script to upload documents and then the agent can use the questions and answers as a source of knowledge. For example: “What is our standard incident response procedure”?
Then the agent will respond with what the approved response would be:
In the interactive mode I am in, we can ask further related questions, like, can you find the relevant incident response SOC 2 mappings?
How do you upload new documents, approve them, and how is this auditable?
Upload
-From a CSV file (columns: question, answer, category)
python scripts/ingest_answers.py — file responses.csv — framework SIG — approved-by “Your Name”
-From an Excel file (auto-detects question/answer columns)
python scripts/ingest_answers.py — file soc2_responses.xlsx — framework SOC2
-From a directory of files
python scripts/ingest_answers.py — dir ./questionnaires/ — framework vendor_questionnaire
-Upload a PDF for OCR extraction (auto-triggers Textract pipeline)
aws s3 cp questionnaire.pdf s3://scf-agent-questionnaire-uploads--us-east-1/ — region us-east-1
PDFs and images uploaded to S3 are automatically processed by the Textract OCR pipeline. Extracted Q&A pairs are stored as DRAFT status and can be approved before use.
Approve
I decided to make this a frontend function, for usability issues. We could also restrict the approval process to certain users.
The approval page lets you:
- Filter by status (DRAFT, APPROVED, REJECTED, ALL)
- Review each answer’s question, response, source document, and extraction method
- Edit answer text inline before approving
- Approve- sets status to APPROVED, records your name and date
- Reject- marks as REJECTED (stays in DB for audit trail)
- Delete- permanently removes (captured in audit log)
Summary metrics at the bottom (total, approved, draft, rejected counts)
Audit
Running DynamoDB Streams on the approved answers table captures every insert, update, and delete with before/after values and timestamps.
Historical Answers
If your data has been uploaded to the agent knowledgebase, you can ask what answers you have provided in the past. For example, “what did we tell XYZ in 2025 about our subprocessors?”
Or an example, in the frontend, “what did we tell auditors about our backup procedures in 2025?”
Lessons Learned — Knowledge Base & Retrieval Edition
S3 Vectors has a hard 2KB metadata limit.
Every record auto-extracts metadata from the document text and if the extracted metadata exceeds 2048 bytes, the record silently fails to index.
My SCF controls (with maturity criteria + mappings) were 4–5KB each — 98% failed. It would have been better to test ingestion with a few documents BEFORE uploading 1,500.
“Vector search” and “full data retrieval” are different problems that need different tools.
Vector search is good at: “find me things related to encryption”. Vector search is bad at: “give me the complete Level 3 maturity criteria for IAC-15”.
In this instance, I used vector search for discovery, DynamoDB for detail. This solved the immediate problem but created another one later- more on that.
DynamoDB Scan with keyword scoring beats DynamoDB “contains” filter.
DynamoDB contains is case-sensitive and requires exact substring match.
Searching “risk assessment” won’t match “Risk Assessment”. In this case (less than 10K items), I could scan all items into Python and score with lowercased keyword matching.
The agent won’t necessarily use tools you built unless the system prompt is very explicit.
I had “search_approved_answers” available but the agent called “list_answer_categories” instead. The original version of the agent preferred to browse categories rather than keyword search. I needed to fix the system prompt to say: “ALWAYS use search_approved_answers with the QUERY parameter first. Do NOT call list_answer_categories first.”
Category-based indexing isn’t enough for natural language retrieval.
I categorized answers as “governance”, “encryption”, etc. The user could ask about “risk assessment”, which was under “governance”, not an obvious choice. I needed to always search across ALL content by keywords. Categories are filters, not primary lookup.
Bedrock KB ingestion gives no per-document error details.
The “ numberOfDocumentsFailed” was 1528, but which ones failed and why?
I should have started with a smaller batch and scaled up.
KB data freshness needs active pipeline management.
Uploading documents to S3 doesn’t update the KB, it still needs an ingestion job. Ingestion can take minutes and might fail silently
I needed to verify with a test query after ingestion and build an auto-updater mechanism immediately.
For questionnaire-style data, DynamoDB/keyword search was better than vector search.
Vector search (semantic similarity) is not necessary when questions are short and specific. Keyword matching on lowercased text with scoring works for less than 10K Q&A pairs.
Lesson Learned — Evals are Needed
With all the data we are ingesting, we will want to make sure our retrieval is accurate. Typically, with knowledge base/vector retrieval in AWS, we could use a Bedrock evaluator job — https://docs.aws.amazon.com/bedrock/latest/userguide/evaluation-kb.html. In this case, however, most of our data is coming from DynamoDB, which wouldn’t work with the Bedrock evaluation.
For the time being, there is a script (eval_retrieval.py) that can perform the following:
The output looks like this:
In the future, if we switched to using a vector database for all of the data, we could run the built in Bedrock evaluation.
Future Costs
If we wanted to make this more robust, we would add a hosted frontend with authentication and a truly functional vector database for all of the data. These are the current vs projected costs for making these changes:
So, in summary, we now have a Bedrock AgentCore Agent that helps with:
- Gap analysis against many frameworks
- Maturity assessment (SCR-CMM Levels 0–5)
- Framework mapping (HIPAA, NIST, ISO, PCI, …)
- Evidence checklists for audits
- Compensating controls for gaps
- Live web research for current regulatory info
- Historical answer research
- Approved answer compilation and tracking
Thanks again to the SCF Council! Please take a look at all of the resources they have available here: https://securecontrolsframework.com/.
Top comments (0)