DEV Community: Artur Schneider

Why Sovereignty fails when it isn’t measurable and what AWS tries to fix with ESC-SRF

Artur Schneider — Fri, 16 Jan 2026 17:44:54 +0000

The underestimated innovation of the AWS European Sovereign Cloud: Sovereignty as evidence (Not Slides)

As of 15 January 2026, the AWS European Sovereign Cloud (ESC) is officially live.

In Europe, “digital sovereignty” shows up everywhere: procurement requirements, regulatory conversations, risk committees, architecture boards. And yet the debate still often stalls at vague statements like “EU-only” or “fully sovereign,” without a consistent way to prove what those claims mean in practice.

That’s why I think the most interesting part of the AWS European Sovereign Cloud (ESC) isn’t the headline of “a new EU-based cloud,” but something much more operational:

AWS is trying to turn sovereignty into an auditable control model and not a marketing label.

AWS calls this approach the European Sovereign Cloud – Sovereign Reference Framework (ESC-SRF), published via AWS Artifact, and positioned as the basis for a dedicated SOC 2 attestation for the European Sovereign Cloud. [S1]

This post goes deeper than the usual headlines and focuses on the part that’s still not widely discussed:

Sovereignty as a control model (ESC-SRF), not a label
Customer-created metadata residency (a surprisingly big deal)
Operational autonomy: EU-only operations boundary + dedicated EU SOC
How to turn all of this into your own “Sovereignty Control Map” and evidence pipeline

1) First, define the problem correctly: sovereignty ≠ residency

In practice, teams mix three different concepts:

a) Data residency

Where your customer content is stored and processed.

b) Operational control

Who can operate, support, and access systems (especially during incidents).

c) Governance + jurisdiction exposure

How the provider is structured, and what legal and organizational boundaries exist.

ESC tries to address all three explicitly through technical measures, operational constraints, and governance structure*and then ties them together with an auditable reference framework (ESC-SRF).* [S1][S2]

That last part is the underappreciated innovation: a provider saying, “Here are the sovereignty criteria, here’s the control mapping, and here’s how it will be validated.”

2) What AWS publicly says is different about ESC (the “what”)

AWS describes ESC as an independent cloud for Europe that is separate and independent from existing Regions, with infrastructure located wholly within the EU. [S3]

The main differentiators include:

2.1 EU-restricted operations (“Qualified Staff” boundary)

The ESC whitepaper introduces Qualified AWS European Sovereign Cloud Staff, located within the EU, controlling day-to-day operations, including support and customer service. [S2][S3]

It also describes controlled consultation mechanisms with global specialists*but with policies and monitoring explicitly designed to protect the boundary.* [S2]

2.2 Dedicated European Security Operations Center (SOC)

AWS describes a dedicated European SOC for the ESC, supported by a dedicated security leader who is an EU citizen residing in the EU. [S2]

2.3 Isolation beyond “normal” Regions

AWS Regions are designed to be isolated, but the ESC whitepaper explicitly positions ESC as a distinct cloud partition and calls out that it is designed to go further (e.g., independent core systems such as billing/account/identity). [S2]

2.4 “Customer-created metadata” residency (this is the sleeper topic)

AWS explicitly includes customer-created metadata in its sovereignty scope (e.g., roles, permissions, tags/labels, configurations) and states it will remain in the EU. [S2][S6]

2.5 Dedicated European trust & certificate services

AWS describes establishing a dedicated European trust service provider to autonomously operate CA key material and perform certificate issuance functions within ESC. [S4]

The ESC whitepaper also mentions a dedicated root European Certificate Authority. [S2]

2.6 Dedicated Route 53 routing and autonomous Direct Connect

AWS describes sovereign DNS/routing properties (dedicated Route 53 routing, European TLD-based nameserver naming) and “sovereign” Direct Connect PoPs. [S2]

3) The 4 sovereignty pillars AWS is emphasizing — and what they actually mean in practice

Most “sovereign cloud” articles stop at geography: “data stays in the EU.”

AWS’ European Sovereign Cloud (ESC) narrative goes further and breaks sovereignty into four pillars that are much closer to an assurance model than a marketing statement:

Data sovereignty (including customer content and customer-created metadata)
Corporate governance under EU law
Operational autonomy
A defined approach to law enforcement requests

Below is the detailed context behind each pillar — plus the engineering implications and the “questions auditors will ask” version.

3.1 Data sovereignty: it’s not just where your data sits — it’s also your control-plane artifacts

AWS draws a clear line between different data categories:

A) Customer content

This is the “obvious” part: objects, database rows, files, messages—your business data.

AWS states ESC is designed so customer content is stored and processed within the ESC boundary, unless you explicitly choose otherwise (for example, by configuring cross-boundary access or integrations). [S2]

B) Customer-created metadata (the sleeper topic)

This is the part many posts ignore. AWS explicitly says ESC also keeps customer-created metadata in the EU and defines it as metadata customers create to manage/configure resources—examples include:

IAM roles and permissions
Resource labels/tags
Resource configurations [S2][S6]

AWS even provides a concrete mental model (e.g., for S3):

Images stored in S3 → customer content
Bucket name, object names, permissions, tags → customer-created metadata [S2]

Why metadata residency matters more than people think:
In regulated environments, metadata often reveals as much as the data itself:

IAM roles can expose privileged access patterns
Tags may unintentionally include sensitive business context (and sometimes PII)
Resource names can leak internal project names, customer names, locations
Network configuration metadata can reveal segmentation and trust boundaries

So, by explicitly treating customer-created metadata as part of sovereignty scope, AWS is addressing a class of audit and threat-model concerns that typically show up late (right before go-live or during the first big audit).

C) The nuance you must not hide: AWS operational data

AWS also states that some information that is neither customer content nor customer-created metadata may leave the EU (examples include internal service metrics used for capacity management, performance monitoring, and certain security support functions). [S2]

This is important because it forces a grown-up sovereignty discussion:

Are those operational signals acceptable under your regulator or customer contracts?
If yes: under which protections and transparency expectations?
If no: what compensating controls are required?

Blog angle: sovereignty isn’t “nothing ever leaves the EU,” it’s “define scope precisely, control it, and prove it.”

Practical engineering takeaways (customer-side controls):

Define a hard policy: no sensitive data in tags/names
Classify “customer-created metadata” explicitly in your data classification scheme
Add automated checks (policy-as-code) to block risky metadata patterns (e.g., tag value regex rules)
Treat observability as a data surface: avoid logging payloads that create accidental “content in logs”

3.2 Corporate governance under EU law: sovereignty includes who is in charge and who can authorize exceptions

ESC is not only described as infrastructure in the EU; AWS describes a distinct governance model under EU/German corporate law for the first region. [S2][S3]

From the published structure, AWS outlines multiple entities (e.g., staffing, infrastructure, trust certificates, holding structure) and describes leadership and oversight constraints. [S2]

What this means in real life (not just legal diagrams):
In sovereignty programs, governance matters because it determines:

Who can approve operational exceptions (e.g., emergency access paths)
Who owns sovereignty-related policy decisions (and can be held accountable)
How oversight works (advisory/board mechanisms, escalation paths)
Whether sovereignty commitments have institutional guardrails beyond technical controls

AWS describes an advisory board mechanism with sovereignty-related oversight responsibilities and EU-resident constraints. [S2]

Audit version of this topic:
Auditors don’t only ask “Where is the data?”

They ask “Who can make an exception—and under what authority?”

Practical takeaways for your architecture + governance docs:

Document your “exception path” (break-glass, escalations, emergency changes)
Map who approves what (RACI) and align it with your sovereignty narrative
Ensure your contracting and support model is consistent with your sovereignty scope [S2]

3.3 Operational autonomy: sovereignty is tested during incidents, not during PowerPoint

AWS positions operational autonomy as a combination of:

A) EU-restricted operations (“Qualified Staff” boundary)

AWS describes restricting ESC operations (including support and data center operations) to Qualified AWS European Sovereign Cloud Staff located in the EU, with controlled consultation mechanisms to access global expertise—but under policies designed to preserve the boundary. [S2][S3]

Why this matters:

Your highest-risk sovereignty moment is incident response
Next is support escalation
Then forensics and post-incident evidence handling

If operational autonomy isn’t designed for these moments, it’s not autonomy.

B) Dedicated European Security Operations Center (SOC)

AWS also describes a dedicated European SOC and EU-resident security leadership for ESC. [S2]

Why this matters:

Incident command structure is part of sovereignty
Who can access telemetry and make containment decisions is part of sovereignty
Who interacts with external parties (including authorities) is part of sovereignty

C) Autonomy at the platform layer (partition-level separation)

AWS describes ESC as “more than another Region”—it’s positioned as a distinct cloud partition with independent billing, account, and identity systems, and “no critical dependencies” on non-EU infrastructure. [S2][S3]

This implies: it’s not only about “EU workloads” but also about reducing non-EU dependencies for core control functions.

Practical engineering takeaways:

Write a sovereignty-focused incident runbook:
- Who can execute break-glass actions?
- Where are approvals logged?
- Who can access logs and snapshots?
- How is evidence preserved?
Run “sovereignty game days”:
- Simulate an incident escalation
- Validate the boundary: who can access what, from where, with what approvals

3.4 Approach to law enforcement requests: the real sovereignty question is what happens when pressure arrives

AWS explicitly frames ESC’s approach to law enforcement requests as a combination of:

technical measures
operational measures
legal and contractual measures [S2]

This topic is often the most politically charged—so it’s worth being very precise.

A) Technical measures (constrain what’s possible)

AWS describes design elements intended to limit access to ESC-restricted data from outside the EU and emphasizes control mechanisms plus logging/visibility. [S2]

Your takeaway as a customer:

Strong encryption + key management strategy matters more than ever
Evidence (logs, access records) is not optional—it’s your safety net

B) Operational measures (define the decision chain)

AWS describes operational handling that involves the appropriate EU-qualified staff, training, and locally aligned processes for request handling. [S2]

Your takeaway:

If your own IR process doesn’t define “who is allowed to respond to external requests,” you’ll improvise under stress—which is exactly what auditors don’t want.

C) Legal + contractual measures (define the posture)

AWS states it reviews requests individually, aims to redirect requests to customers where possible, notifies customers when allowed, and challenges requests that conflict with applicable law, plus points to transparency reporting. [S2]

Your takeaway:

Your sovereignty narrative should include:
- notification expectations
- customer ownership of disclosures
- how requests are routed and handled
- what evidence you keep

Practical takeaways for a “sovereignty-by-design” program:

Decide what your stance is on third-party requests (and document it)
Ensure your encryption key strategy supports your sovereignty posture
Make transparency and auditability part of your operating model (not “as needed”)

3.5 Turning the four pillars into an assurance model: Requirements → Controls → Evidence

If you want sovereignty to survive procurement, audits, and real incidents, treat it exactly like security compliance:

1) Requirements

Example: “Customer-created metadata must remain within the EU boundary.” [S2]

2) Controls

Policy-as-code guardrails (tag conventions, naming constraints)
Access controls (least privilege, break-glass approvals)
Data egress controls (explicit allowlists for cross-boundary integration)
Operational runbooks (incident response, escalation)

3) Evidence

AWS Config / CloudTrail exports and conformance snapshots
Break-glass approval logs + incident reports
Architecture Decision Records (ADRs) documenting boundary choices
Provider artifacts (ESC-SRF via AWS Artifact; assurance reports when available) [S1]

This is why ESC-SRF is interesting: it pushes the conversation toward a reusable mapping between sovereignty criteria and verifiable controls. [S1]

3.6 Questions to include in your blog (because they force real answers)

If you want this section to hit harder, end with questions people can’t dodge:

What do we classify as customer-created metadata in our organization—and how do we prevent sensitive leakage into tags, names, or logs?
What data do we accept as “AWS operational data,” and how do we document that risk decision? [S2]
Who can authorize sovereignty exceptions—and how is that approval audited?
In an incident, who can access what, from where, and how is it evidenced?
What is our documented approach to third-party requests, and how does encryption/key custody support it?

These questions transform “sovereignty” from a slogan into a system.

4) The under-discussed part: ESC-SRF turns sovereignty into a control model (the “how”)

Here’s the key: AWS doesn’t just list features. It positions ESC sovereignty as criteria aligned across domains like:

governance independence
operational control
data residency
technical isolation [S1]

Then it publishes the reference framework in AWS Artifact and states it forms the basis for a dedicated ESC SOC 2 attestation. [S1]

What’s new about that?

In most sovereignty programs, teams build a messy “evidence story” from scratch:

vendor PDFs + scattered attestations
internal architecture decisions
custom risk narratives
painful audit cycles

ESC-SRF is trying to standardize the provider side into an auditable mapping you can reuse.

AWS also states ESC controls are undergoing independent third-party audit, and that ESC-SRF can be used as:

an assurance model (end-to-end traceability from criteria to implementation)
a design reference framework (how customers build sovereignty controls on top) [S1]

5) Deep dive: the most important ESC design nuances (and what they imply for architects)

5.1 Understand the boundary: content vs metadata vs “AWS operational data”

AWS draws a line between:

customer content
customer-created metadata
some AWS operational data (neither content nor customer-created metadata) that may leave the EU (e.g., internal metrics) [S2]

What this means for your risk model
If your sovereignty program requires strict “nothing leaves the EU,” you must explicitly address:

what counts as operational data
how it’s protected
whether it’s acceptable under your regulator / customer contracts

5.2 Governance isn’t optional

The ESC whitepaper describes EU-law governance constraints for the first region, including oversight mechanisms like an advisory board. [S2]

Even if you’re “just an engineer,” your architecture can fail the sovereignty conversation if governance and exception authority aren’t explainable.

5.3 Operational autonomy: plan your incident mechanics

If you’re adopting ESC (or designing with it in mind), define:

Break-glass policy
Support engagement model
Forensics flow
Evidence preservation workflow

ESC describes EU-only operational control and a dedicated EU SOC; your job is mapping that into your org’s runbooks. [S2][S3]

6) Service reality: “sovereign” doesn’t mean “all services on day one”

AWS states ESC will launch with a set of core services across categories (compute, storage, database, networking, security, and also AI/ML) and then expand based on demand. [S3][S7]

AWS also published roadmap updates (example items include IAM Identity Center expected in Q1 2026; CloudFront expected by end of 2026). [S7]

Practical implication: plan for service gaps and design substitution patterns (auth, DNS, observability, delivery edge).

7) Make sovereignty measurable: your “Sovereignty Control Map” template

Copy/paste and adapt this table. This is where your sovereignty program becomes operational.

Sovereignty Control Map (SCM) — starter template

Domain	Requirement (your wording)	ESC capability (provider)	Your control (customer)	Evidence you produce	Cadence / Owner
Data residency	Customer content stays in EU	Customer content stored/processed within boundary unless customer chooses otherwise [S2]	Data classification + deny policies for replication/export	Config + CloudTrail + ADRs	Monthly / Platform Sec
Metadata residency	Customer-created metadata stays in EU	Explicitly kept in EU [S2][S6]	Tagging/IAM naming rules + policy-as-code	Policy repo, Config rules, Access Analyzer reports	Monthly / IAM Owner
Operational control	Only EU-resident staff operate day-to-day	EU “Qualified Staff” model [S2][S3]	Break-glass workflow + approvals + ticket SOP	Ticket evidence + access logs + PIRs	Quarterly / SOC Lead
Incident response	EU-operated SOC & escalation	Dedicated EU SOC [S2]	IR runbooks + tabletop exercises	IR exercise reports	Quarterly / Security
Trust anchors	CA operations controlled in EU	EU trust service provider + EU root CA [S2][S4]	Certificate lifecycle policy + key custody policy	CA policy docs + issuance logs	Quarterly / PKI Owner
Isolation	Strong separation from other Regions	Separate/independent partition; independent core systems [S2][S3]	Network/org boundary + egress controls	Network configs + firewall policies + egress logs	Monthly / Network Sec
Assurance	Sovereignty controls auditable	ESC-SRF via Artifact; SOC2 basis [S1]	Integrate provider evidence into your GRC	Artifact exports + mapping doc	Quarterly / Compliance

8) Common misconceptions (and better questions to ask)

“We’re already in an EU Region, so we’re sovereign.”

EU Regions help with residency, but sovereignty often requires deeper answers:

Who can operate/support?
What about metadata?
What evidence exists?

“Sovereignty means nothing ever leaves the EU.”

Then explicitly define how you treat:

operational metrics / telemetry
managed service signals
support workflows AWS explicitly distinguishes these categories in its ESC materials. [S2]

“Sovereignty is a procurement problem.”

Sovereignty becomes a production problem the first time you have:

a major incident
a regulator question
a customer audit Treat it like engineering.

Conclusion: the real story is auditability

The AWS European Sovereign Cloud will be summarized by many as “an EU-based cloud with EU operations.” But the more interesting story—still not widely reflected—is the move toward sovereignty as an assurance model:

Criteria → Controls → Independent validation → Evidence (ESC-SRF in Artifact; SOC 2 basis). [S1]

If sovereignty matters to you, treat it like security:

Requirements → Controls → Evidence.

Everything else is just vocabulary.

Sources

[S1] Exploring the new AWS European Sovereign Cloud: Sovereign Reference Framework (ESC-SRF)

https://aws.amazon.com/blogs/security/exploring-the-new-aws-european-sovereign-cloud-sovereign-reference-framework/
[S2] Overview of the AWS European Sovereign Cloud (whitepaper PDF)

https://d1.awsstatic.com/onedam/marketing-channels/website/aws/en_US/whitepapers/compliance/Overview_of_the_AWS_European_Sovereign_Cloud.pdf
[S3] European Digital Sovereignty FAQ (AWS)

[S4] Establishing a European trust service provider for the AWS European Sovereign Cloud (AWS Security Blog)

https://aws.amazon.com/blogs/security/establishing-a-european-trust-service-provider-for-the-aws-european-sovereign-cloud/
[S5] AWS plans to invest €7.8 billion into the AWS European Sovereign Cloud (AboutAmazon EU)

https://www.aboutamazon.eu/news/aws/aws-plans-to-invest-7-8-billion-into-the-aws-european-sovereign-cloud
[S6] Design approach (AWS Docs – ESC whitepaper page)

https://docs.aws.amazon.com/whitepapers/latest/overview-aws-european-sovereign-cloud/design-approach.html
[S7] Announcing initial services available in the AWS European Sovereign Cloud (incl. roadmap updates)

https://aws.amazon.com/blogs/security/announcing-initial-services-available-in-the-aws-european-sovereign-cloud-backed-by-the-full-power-of-aws/

Architecting your GenAI data pipeline with AWS native services

Artur Schneider — Sun, 22 Jun 2025 18:40:11 +0000

When I started building GenAI solutions, I felt confident working with models, prompts, and architecture — but the data part always felt like a black box. I kept asking myself: Where do I even begin if I want to use my own data? Every time I looked into it, I found a pile of scattered advice, incomplete setups, or tools that didn’t quite fit together. It reminded me of moving into a new house and opening the garage — only to find it packed with boxes from ten different people. You know there’s valuable stuff in there, but it’s all mixed up, mislabeled, and overwhelming.

This post is what I wish someone had handed me back then — a clear, hands-on walkthrough of how to turn that chaotic garage into a well-organized workshop. If you’re comfortable with AWS and GenAI but still wondering how to structure, clean, and prepare your own data properly, this is for you.

1. Getting Started

First things first: what do I need before I even begin? Beyond an AWS account and some basic AWS skills, start by taking inventory of your data. Ask yourself: What types of data do I have? (CSVs, JSON logs, PDFs of documents, etc.) Where is it coming from? (On-prem systems, databases, S3 buckets, etc.) This will inform your pipeline design. Honestly, at first I felt paralyzed: “Should I use Glue? Athena? Lambda? Everything?” I was like a newbie chef staring at a pantry full of ingredients. The answer is: you don’t need magic – just a plan.

Start small:

Create or identify an S3 bucket (or a couple) for raw data.
If you have existing sources (RDS, on-prem, SaaS APIs, etc.), consider how to move that data in – AWS Glue has connectors, and AWS DataSync or Transfer Family can help with files.
Enable AWS CloudTrail for your S3 bucket so events (like new file uploads) can trigger processing.

Honestly, the first thing I realized was: you don’t have to nail everything at once. Get one data source flowing into S3, see how it goes, then expand. It’s like testing a new recipe by cooking a small batch first.

2. Structuring Your S3 Data Lake

Once you’re ready to store data, structure is key. AWS best practices recommend multiple S3 “zones” or layers, typically separate buckets for raw, stage/processing, and analytics/curated data. Think of it as organizing your garage: one rack for “just moved in – untouched stuff” (raw), one workbench for “mid-cleanup – in progress” (stage), and one shelf for “ready-to-use” (analytics). For example:

Raw layer – store files exactly as you received them (CSV, JSON, PDF, etc.). Enable versioning here so you never lose the original. This is your time-capsule.
Stage layer – place intermediate, cleaned-up data here. Convert formats (e.g. CSV→Parquet), perform initial transformations, and catalog the schema with AWS Glue.
Analytics layer – put your fully processed, query-ready tables (often Parquet or Iceberg) here. This is what your models or analysts will ultimately consume.

You will likely use separate S3 buckets named by layer (e.g. myorg-data-raw, myorg-data-stage, myorg-data-analytics), possibly including environment or account info for clarity. Good naming makes governance and cost-tracking easier.

A handy checklist:

Use at least 3 layers (raw, stage, analytics), each in its own S3 bucket.
Keep originals intact in raw (no manual edits!).
Plan a folder structure/prefixes inside each bucket (date-based partitions can help).
Enable encryption (S3 or via AWS KMS) and versioning on raw and stage buckets.

By thinking of your S3 lake as a well-labeled garage, you will save countless headaches later.

3. Ingesting Mixed Data Types

Your data recipe likely has all sorts of ingredients: relational tables, CSVs, JSON logs, and even PDFs or images. AWS provides tools for each:

For batch files (CSV, JSON, images, PDFs in S3): you can simply upload them to S3 (via CLI, SDK, or GUI) or use AWS DataSync/Transfer for large/migrated datasets. Once in S3, set up AWS Glue Crawlers to auto-detect schema on CSV/JSON and populate the Glue Data Catalog. Crawlers are lazy scientists that automatically say, “Hey, new data here!” and create tables you can query with Athena.
For streaming or real-time data (e.g. logs, IoT, social feeds): use Amazon Kinesis Data Firehose to ingest streams directly into S3. Firehose can even convert JSON to Parquet on the fly for you, and it supports invoking Lambda for custom transforms (e.g. turning CSV logs into JSON via a blueprint). It’s like having a smart conveyor belt that batches, compresses, and drops your data into S3.
For databases (on-prem or RDS): consider AWS Database Migration Service (DMS) or Glue’s JDBC connections to pull data into S3 or directly into Redshift/Athena as needed.
For documents and images (PDFs, scanned docs): Amazon Textract is your friend. It’s an AI OCR service that can extract text and structured data from scanned files. For example, drop a PDF of a report into S3 and trigger a Lambda that calls Textract to get the text. Save that output (say as plain text or JSON) back into S3 so it can join the rest of the lake. Bedrock Knowledge Bases (KBs) can even ingest common document formats like PDF, Word, HTML, and CSV directly. (Just keep each source file ≤50 MB.)

Key points: Your pipeline will likely be a mix of the above. You might set an EventBridge rule (via S3 events/CloudTrail) so that any new S3 upload triggers a Glue or Lambda job to process it. For example, new CSVs trigger a Glue ETL that cleans and moves data into the stage bucket; new PDFs trigger a Textract process that spits out text to S3. Remember, AWS is great at handling heterogenous data – just string the right services together.

4. Cataloging and Schema Harmonization (Glue & DataBrew)

Once data is landing in S3, you need a catalog(index) and some cleaning. AWS Glue handles the catalog and ETL magic. Glue Crawlers will scan your raw S3 folders and register tables in the Glue Data Catalog. Think of the Data Catalog as a library card catalog for all your datasets. You can then use AWS Glue jobs (PySpark) or AWS Glue DataBrew (no-code GUI) to clean and harmonize.

DataBrew is a visual data-prep tool – no coding needed – that can profile and transform your data. It has 250+ built-in functions (filtering, renaming columns, converting formats, handling missing values, etc.). It is like having a friendly spreadsheet on steroids: you load a dataset, click to apply fixes (e.g. fix date formats, standardize field names, remove duplicates), and output a clean file. For example, if one source has “FirstName” and another has “first_name”, you can use DataBrew to unify them.

Meanwhile, Glue ETL jobs can join, enrich, or further transform data. The advantage: everything can be automated in Glue Workflows. Glue (with DataBrew) lets “clean and normalize without writing code” – perfect for teams where not everyone is a developer. After transformation, write outputs to the “stage” or “analytics” bucket, and update the Data Catalog with the new schema.

In practice, you can do:

Crawlers auto-detect schema on raw data and log errors for missing or invalid values.
Run a DataBrew recipe to fix common issues across datasets (format dates, fill nulls, etc.).
Use a Glue job (Spark) to join datasets or convert everything to an analytics-friendly format (like Parquet/Apache Iceberg tables).

This process is like cleaning that messy garage: Dust off the data, sort it onto shelves, and create a manifest (the Glue catalog) so you can find what you need. All those crawlers and DataBrew steps mean your files go from “hot mess” to “whew, this is actually usable”.

5. Normalizing Documents for GenAI (Bedrock KBs, Textract)

Text documents are a special case. For Generative AI (especially Retrieval-Augmented Generation), you often feed documents into a knowledge base or query engine. AWS’s Amazon Bedrock Knowledge Bases let you ingest text-based files (see supported formats above) and then query them with LLMs. But first you may need to normalize and chunk the text.

For example, if you have PDFs of manuals or reports, use Textract to extract the text and tables. Once you have raw text, you might run a Glue job or Lambda to split it into logical sections or “chunks” of a few hundred words each. This is because Bedrock KBs often work best when content is broken into pieces (like paragraphs). Think of it as chopping a long document into bite-size pieces.

Advanced note: Bedrock now supports things like semantic and hierarchical chunking and even using foundation models to parse tricky PDFs. But at a beginner level, start simple: extract text, remove any scanned gibberish, and put everything in plain .txt or .md files in S3. Then connect that S3 as the data source for your Bedrock KB. The service will parse and index it for RAG queries.

One more tip – normalizing terminology: Different documents might use “DOB”, “Date of Birth”, and “Birth Date”. You can use Glue or even LLM prompts to standardize these keys (so your GenAI sees them as the same concept). In intelligent document processing terms, this is “template and normalization” (defining aliases for different terms). It’s like telling the AI, “hey, whenever you see DOB or Birthdate, they all mean the same thing.” This ensures your knowledge base is clean and consistent.

6. Securing and Governing the Data Lake (IAM, Lake Formation, LF-Tags)

Now that you have valuable data, lock it up properly. Start with IAM: enforce the principle of least privilege, use IAM roles for your Glue jobs and Lambda functions (they get just the S3/Glue permissions they need), and consider AWS KMS for key management. But IAM alone can get tedious for per-table or per-column access. That’s where AWS Lake Formation comes in.

Lake Formation lets you set fine-grained access controls on your data catalogs and S3 tables. A powerful feature is LF-Tags: attribute-based tags you attach to tables/columns (like department=finance or sensitivity=PII), and then grant user roles those same tag values. It’s literally like parking passes and parking lots: imagine a corporate garage with zones. Each dataset has a parking sticker (LF-Tag) and each data analyst has a matching pass. Only matching stickers let you “drive through” to access the data. LF-Tags scale better than hand-granting each table to each user one by one.

(Important note: LF-Tags are not the same as IAM tags. LF-Tags gate Lake Formation table access; IAM tags control IAM policies. Don’t confuse the two – one is for data access, the other for service permissions.)

Also use Lake Formation to enforce column- or row-level filtering if needed (e.g. mask emails or only show Europe-region rows). And don’t forget S3 bucket policies or Access Points for cross-account sharing. In short, treat your data lake like Fort Knox: multi-layer security, logging everything.

I like to joke that LF-Tags are like parking passes: you stick a tag on the dataset and give matching passes to users; if the tags match, voila, access granted. It’s way easier than juggling dozens of user policies as your lake grows.

7. Automating the Pipeline (EventBridge, Glue Workflows, Step Functions)

Manual clicks are doomed to error. Automate your pipeline so new data flows on its own. AWS EventBridge is your event router: for example, set it to catch all S3 PutObject events (via CloudTrail) and trigger AWS Glue Workflows. Glue Workflows let you chain multiple Glue jobs, crawlers, and triggers as a single pipeline.

A common pattern: S3 → CloudTrail → EventBridge rule → Glue workflow. In practice I do this: when ten files land or after a 5-minute window, EventBridge starts the Glue workflow. Glue then runs a crawler (to detect new schema), runs an ETL job (to clean/convert data), and maybe another job to move data to analytics zone. This is event-driven ETL. The nice part is you don’t have to poll or schedule useless jobs – it reacts to real events.

You could also use AWS Step Functions for orchestration (especially if you need branching logic or parallel tasks). Step Functions can call Glue, Lambda, SageMaker, etc., with built-in retries. I’ve used it to coordinate complex flows: e.g. after Glue finishes, step through custom quality checks (via Lambda) before loading data.

In short, trigger-on-upload or scheduled rules in EventBridge start your pipeline; Glue workflows or Step Functions manage the steps; and the pipeline runs itself. It’s like setting up a line of dominoes – once you tip the first piece (new data arrived), everything else happens automatically.

8. Making Your Data Discoverable (Amazon DataZone & SageMaker)

Lastly, once your data lake is humming, make it easy for your teams to find and use the data. Amazon DataZone is a data catalog and governance service – think of it as a Google for your company’s data assets. You can register your S3 data (via Glue tables) in DataZone, tag them (e.g. sales, raw-data), and write descriptions. Users across the org can then search or browse for the datasets they need. This is golden for collaboration.

The best part: SageMaker now integrates directly with DataZone. Data scientists and ML engineers can search the DataZone catalog inside SageMaker Studio or Canvas, and even pull data into notebooks or Canvas by clicking “Subscribe” to a dataset. It’s like having a built-in data shopping mall – shop for datasets (or features/models) right in your IDE.

Amazon SageMaker now integrates with Amazon DataZone to streamline machine learning governance | Artificial Intelligence

aws.amazon.com

For example, you might publish a cleaned Parquet table to DataZone as a “SalesStats” asset. Anyone on the ML team can now discover SalesStats from SageMaker and add it to their Jupyter environment. After training a model, they could even publish the model back to DataZone for others to reuse.

In short, leverage DataZone (linked to your Lake Formation / Glue Catalog) to catalog everything, and enable SageMaker’s data search/subscribe. That way, your data becomes a first-class citizen in the ML workflow. Your messy garage is now a public library where everyone can check out books (i.e. datasets) they need.

Putting It All Together

Phew! That was a lot of detail, but remember: start simple and iterate. The steps are roughly: get your AWS setup, organize S3, ingest all your data sources, clean and catalog with Glue/DataBrew, extract text from documents (Textract) for Bedrock, secure everything with IAM/LF-Tags, automate with EventBridge/Glue/Step Functions, and finally plug it into DataZone/SageMaker for others to find.

At first it might feel like there are a million AWS services to learn – trust me, I’ve been there. But take it one step at a time. Each piece you set up (a crawler, a Tag, a DataBrew recipe) is like cleaning one corner of that garage. Eventually you’ll stand back and say, “Wow, my GenAI application now actually sees my data!”

Good luck with your GenAI data adventure. I am still learning every day, and I will admit sometimes I screw up a bucket policy or mix up LF-Tag names. But as long as we keep our data organized, secure, and discoverable, our AI projects have the fuel they need.

MCP explained: How the Model Context Protocol transforms AI in the Cloud

Artur Schneider — Mon, 07 Apr 2025 11:28:52 +0000

When I first read about MCP, I wasn't exactly sure what it was. But once I understood the Model Context Protocol (MCP), it was like one of those rare "aha!" moments you sometimes have in tech. Suddenly, it clicked why developers and cloud experts were so excited about this technology. Imagine giving your AI assistant superpowers without needing to fully retrain it — that's precisely what MCP makes possible!

In today's fast-paced tech world, we're constantly looking for ways to accelerate development processes while improving quality. Especially in the AWS environment, where complexity increases with each new service, we need smarter tools that make our work easier. This is where MCP comes into play – a protocol that fundamentally changes the way we interact with AI models.

But what exactly is MCP? Imagine having a brilliant colleague who is incredibly intelligent but has no access to your company data or systems. Without this information, their abilities are limited. MCP is like a bridge that suddenly gives this colleague access to all your data sources, tools, and systems – in a secure, controlled way. The result? An AI assistant that not only has general knowledge but also deep, specialized expertise in exactly the areas that are relevant to you.

In this article, I want to show you why MCP is a real game-changer – especially for beginners who are just diving into the world of AI and cloud development. Together, we'll discover:

How MCP works and why it's revolutionizing the AI landscape
What concrete benefits MCP offers for developers and companies
How you can immediately recognize the value of MCP through practical examples
How you can take your first steps with MCP without being an AI expert

After this article, you'll not only understand what MCP is, but also why it's so important for modern cloud applications. You'll see how MCP can help you save time, improve quality, and ensure the security of your data.

So buckle up – we're embarking on an exciting journey into the world of Model Context Protocols that will show you what the future of AI integration in the cloud looks like!

What is the Model Context Protocol (MCP)?

If you've ever worked with AI assistants like ChatGPT or Claude, you know their impressive capabilities – but also their limitations. They can help you with many tasks, but as soon as it comes to specialized knowledge or accessing your own data, they quickly reach their limits. This is exactly where the Model Context Protocol (MCP) comes in.

MCP Simply Explained

At its core, MCP is a standardized, open protocol that enables seamless interaction between large language models (LLMs), data sources, and tools. Think of MCP as a universal translator that mediates between your AI model and the outside world.

Without losing technical expertise, here's a simple analogy: Think of your LLM as a brilliant advisor sitting in a soundproof room. This advisor has learned a lot during their training, but cannot access current information or interact with systems outside their room. MCP is like a communication system that suddenly allows this advisor to speak with the outside world, access your company data, and even perform actions in your systems – all while sensitive data remains securely in place.

Why Was MCP Developed?

The development of MCP was a response to a fundamental problem: How can we expand the capabilities of AI models without constantly having to retrain them?

Before MCP, there were essentially two approaches:

Complete retraining of the model with specialized data – expensive, time-consuming, and inefficient
Prompt Engineering – embedding context in requests, which is limited by token limits and lack of up-to-date information

MCP offers an elegant third way: It extends the capabilities of the model by giving it access to external knowledge sources and tools without changing the model itself.

How Does MCP Work Technically?

Without overwhelming you with too many technical details, here's a simplified look under the hood:

Request: You ask a question or give a task to your LLM
Recognition: The LLM recognizes that it needs external information or tools
Communication: Via the MCP protocol, the LLM communicates with specialized servers
Data Retrieval: The MCP servers access relevant data sources or tools
Integration: The obtained information is integrated into the context of the LLM
Response: The LLM generates a response that is now enriched with specialized knowledge

The special thing about this: The sensitive data remains local and is not integrated into the model itself. This is an enormous advantage for data security.

MCP in the AWS World

AWS recognized the potential of MCP early on and developed a suite of specialized MCP servers with the "AWS MCP Servers for code assistants." These servers bring AWS best practices directly into your development workflow.

Imagine you're working on a complex AWS project. Without MCP, you would have to:

Spend hours reading documentation
Research best practices
Understand and implement security policies
Manually incorporate cost optimizations

With AWS MCP Servers, you can simply ask your AI assistant: "How do I implement a secure, cost-optimized Amazon Bedrock Knowledge Base?" and immediately receive code that follows AWS best practices, with built-in security controls and optimized resource configurations.

MCP Visualized

To make the concept more tangible, here's a simplified representation of the MCP architecture:

+---------------+       +----------------+       +-------------------+
|               |       |                |       |                   |
|  User         |------>|  LLM with MCP  |<----->|  MCP Server       |
|  (You)        |       |  Integration   |       |  (Specialized)    |
|               |       |                |       |                   |
+---------------+       +----------------+       +------+------------+
                                                        |
                                                        v
                                               +------------------+
                                               |                  |
                                               |  Data Sources    |
                                               |  & Tools         |
                                               |                  |
                                               +------------------+

In this model, the LLM remains unchanged but gains access to specialized knowledge and capabilities through the MCP connection.

Why is MCP a Breakthrough?

MCP fundamentally changes how we can work with AI models:

Extensibility: Models can acquire new capabilities without being retrained
Currency: Access to the latest information, not just training data
Specialization: General models can become domain experts
Data Protection: Sensitive data remains local and secure
Agentic AI: Enables AI assistants that can actually perform actions in your systems

In the next section, we'll look at the concrete benefits of MCP for developers and companies – with practical examples that show you how MCP can revolutionize your daily work.

The Benefits of MCP for Developers and Companies

Now that we understand what MCP is and how it works, let's look at what concrete benefits it brings for you as a developer or for your company. And don't worry – I'll explain everything with practical examples that are easy to understand even for beginners.

1. Access to Specialized Knowledge Without Retraining Models

What does this mean? Imagine being able to transform a general practitioner into a heart specialist within seconds – without them having to study for years. That's exactly what MCP enables for AI models.

Practical example:
Let's say you're working on an AWS project and need help implementing a secure Amazon Bedrock Knowledge Base. Without MCP, you would either:

Spend hours in the AWS documentation
Hire an expensive AWS specialist
Experiment with trial and error and hope not to overlook anything

With AWS MCP Servers, you can simply ask: "How do I implement a secure Amazon Bedrock Knowledge Base for my company data?" and immediately receive:

# AWS CDK Code with Best Practices for Amazon Bedrock Knowledge Base
const knowledgeBase = new BedrockKnowledgeBase(this, "CompanyKB", {
  embeddingModel: BedrockFoundationModel.TITAN_EMBED_TEXT_V1,
  vectorStore: new OpenSearchServerlessVectorStore(this, "VectorStore", {
    encryption: OpenSearchEncryption.KMS,
    ebs: OpenSearchEbsOptions.provisioned(100, OpenSearchVolumeType.GP3)
  })
});

// Automatically generated security controls
cdk-nag.NagSuppressions.addResourceSuppressions(
  knowledgeBase, 
  [{ id: 'AwsSolutions-IAM4', reason: 'Managed policy used only for Bedrock service role' }]
);

What's special: This code already contains all AWS best practices, security controls, and optimizations – without you having to be an AWS expert yourself.

2. Data Security Through Local Data Storage

What does this mean? Your sensitive company data doesn't need to be uploaded to the AI model or used for training. It remains secure in your environment.

Practical example:
Imagine you have confidential customer data that must not leave your company under any circumstances. With MCP, you can:

Set up a local MCP server that accesses your internal database
Connect your AI assistant to this server via MCP
Make requests like: "Summarize the sales figures for the last quarter"

The process then looks like this:

The request goes to the LLM
The LLM recognizes that it needs sales data
Via MCP, the request is forwarded to your local server
The server retrieves the data from your database (the data never leaves your network)
The summary is generated and returned

The sensitive sales data remains in your secure environment the entire time – an enormous advantage over conventional approaches.

3. Reduction of Development Time

What does this mean? Tasks that would normally take days or weeks can be completed in minutes with MCP.

Practical example:
Let's say you need to create an AWS Lambda function that reads data from a DynamoDB table, processes it, and writes it to an S3 bucket. Traditionally, you would:

Read the documentation for Lambda, DynamoDB, and S3 (2-3 hours)
Understand and configure IAM roles and policies (1-2 hours)
Write and test the code (3-4 hours)
Fix bugs and optimize (2-3 hours)

Total time: 8-12 hours

With AWS MCP Servers:

You describe your project: "Create a Lambda function that reads data from DynamoDB and writes to S3"
The MCP server generates:
- The complete Lambda code
- The IAM roles with least-privilege principle
- CloudFormation/CDK for the infrastructure
- Logging and monitoring configuration
- Error handling and retry logic

Total time: 10-15 minutes

This is not an exaggeration – I've experienced myself how MCP can drastically reduce development time, especially for AWS implementations.

4. Automatic Application of Best Practices

What does this mean? You no longer need to know and manually implement all best practices – MCP does this automatically for you.

Practical example:
Imagine you're developing a new web application on AWS and need to ensure it meets security standards. Without MCP, you would have to:

Study the AWS Well-Architected Framework
Understand Security Hub recommendations
Research compliance requirements
Manually implement and verify everything

With AWS MCP Servers, you automatically receive:

# Automatically generated CloudFormation template with integrated best practices
Resources:
  WebAppBucket:
    Type: AWS::S3::Bucket
    Properties:
      BucketEncryption:
        ServerSideEncryptionConfiguration:
          - ServerSideEncryptionByDefault:
              SSEAlgorithm: AES256
      PublicAccessBlockConfiguration:
        BlockPublicAcls: true
        BlockPublicPolicy: true
        IgnorePublicAcls: true
        RestrictPublicBuckets: true
      LoggingConfiguration:
        DestinationBucketName: !Ref LoggingBucket
        LogFilePrefix: webapp-access-logs/

The MCP server has automatically:

Activated encryption
Blocked public access
Configured logging
Implemented other best practices

You no longer need to research and implement these things individually – an enormous time saver and security boost.

5. Scalability and Flexibility

What does this mean? MCP grows with your requirements and can be adapted to different use cases.

Practical example:
Imagine you start with a simple application that only uses basic AWS services. Over time, your project grows and you need:

Integration with Amazon Bedrock for AI functions
Complex data processing with AWS Glue
Serverless architecture with AWS Lambda and API Gateway

With MCP, you don't have to become an expert for each new service. You can simply add new MCP servers that specialize in these services:

Core MCP Server: Basic AWS functions
AWS CDK MCP Server: Infrastructure as code
Bedrock Knowledge Bases MCP Server: AI integration
Cost MCP Server: Cost optimization

Your AI assistant can then seamlessly switch between these servers depending on which expertise is needed – like a team of specialists working perfectly together.

Real Example: From Days to Minutes

Let me share a real example from my own experience:

Recently, I was tasked with developing a solution that integrates company documents into an Amazon Bedrock Knowledge Base and makes them accessible via a chatbot. Traditionally, this project would have taken about two weeks:

Research on Amazon Bedrock Knowledge Bases (2-3 days)
Development of the document ingestor (3-4 days)
Implementation of the chatbot with Bedrock integration (3-4 days)
Testing and optimization (2-3 days)

With AWS MCP Servers, I could:

Describe my project
Adapt the generated code
Deploy the solution

Total time: Under 4 hours instead of 2 weeks!

The generated code already contained:

Optimal vector database configuration
Secure IAM roles and policies
Efficient document processing
Cost-optimized resource configuration

How MCP Differs from Current Approaches with LLMs and Knowledge Bases

To truly understand the revolutionary nature of MCP, it's important to compare it with current approaches for enhancing LLMs with specialized knowledge. Let's explore how MCP differs from and improves upon existing methods:

1. Traditional RAG (Retrieval-Augmented Generation) vs. MCP

Traditional RAG:

Creates a vector database of documents
When a query arrives, it searches for relevant documents
Inserts these documents into the prompt
Limited by context window size (typically 8K-128K tokens)
Data must be pre-processed and indexed
Static knowledge that requires manual updates

MCP Approach:

Provides a standardized protocol for LLM-to-tool communication
Dynamically accesses data sources as needed, not just pre-indexed documents
Can perform complex queries and transformations on data
Not limited by context window size since data is processed externally
Can access real-time information and live systems
Automatically stays up-to-date with source systems

Key Difference: RAG is like giving the LLM a relevant book to read before answering your question. MCP is like giving the LLM the ability to use a computer, search databases, and run specialized tools to find and process information.

2. Fine-tuning vs. MCP

Fine-tuning:

Requires collecting specialized training data
Modifies the model's weights through additional training
Knowledge is "baked in" and static
Expensive and time-consuming process
Requires specialized ML expertise
Knowledge becomes outdated as the world changes

MCP Approach:

Leaves the base model unchanged
Extends capabilities through external connections
Knowledge remains in original systems, always current
Quick to implement with minimal setup
Requires minimal ML expertise
Automatically incorporates the latest information

Key Difference: Fine-tuning is like teaching a student everything they might need to know in advance. MCP is like teaching them how to use a library, internet, and specialized tools to find information when needed.

3. Function Calling vs. MCP

Function Calling:

Allows LLMs to call predefined functions
Functions must be registered in advance
Limited to the specific API endpoints defined
Often requires custom implementation for each use case
Typically lacks standardization across different systems

MCP Approach:

Provides a standardized protocol for tool use
Enables discovery and use of tools not defined at design time
Creates an ecosystem of compatible tools and services
Implements a consistent interface across different systems
Allows for complex workflows across multiple tools

Key Difference: Function calling is like giving an LLM a specific set of tools with instruction manuals. MCP is like giving it the ability to discover, learn about, and use any tool in an entire workshop, even ones that didn't exist when it was created.

4. Knowledge Bases vs. MCP

Traditional Knowledge Bases:

Static repositories of information
Require manual updates and maintenance
Often siloed from other systems
Limited to the information explicitly added
Typically text-based question and answer

MCP Approach:

Dynamic connections to live data sources
Automatically updated as source systems change
Integrated with multiple systems and tools
Can access any information available in connected systems
Enables not just answers but actions and complex workflows

Key Difference: A knowledge base is like a comprehensive encyclopedia. MCP is like having access to the entire internet, live databases, and specialized tools that can perform actions based on the information.

5. Agent Frameworks vs. MCP

Agent Frameworks:

Often proprietary implementations
Typically designed for specific use cases
May lack standardization
Can be complex to set up and maintain
Often require significant customization

MCP Approach:

Open, standardized protocol
Designed for general-purpose use
Consistent implementation across platforms
Simplified setup with standardized interfaces
Works out of the box with compatible systems

Key Difference: Agent frameworks are like custom-built robots designed for specific tasks. MCP is like a universal standard that allows any AI to connect with any compatible tool or data source.

The Fundamental Shift: From Static to Dynamic Intelligence

The most profound difference between MCP and traditional approaches is the shift from static to dynamic intelligence:

Traditional Approaches:

Knowledge is fixed at training or fine-tuning time
Updates require retraining or reindexing
Limited by what was anticipated during design
Intelligence is contained within the model

MCP Approach:

Knowledge is accessed dynamically when needed
Updates happen automatically as source systems change
Can adapt to unanticipated needs through tool discovery
Intelligence is distributed across the model and connected systems

This shift represents a fundamental evolution in how we think about AI systems - moving from isolated, static models to connected, dynamic systems that can leverage specialized tools and real-time information.

Conclusion: MCP as the Key to the AI Revolution in the Cloud

We've taken an exciting journey through the world of Model Context Protocols, and I hope you can now see why I'm so enthusiastic about this technology. MCP is not just another acronym in the already crowded tech world – it's a fundamental paradigm shift in the way we work with AI models.

Let's briefly summarize the key insights:

MCP bridges the gap between general AI models and specialized use cases by enabling seamless access to external data sources and tools.
The benefits are impressive:
- Access to specialized knowledge without retraining
- Improved data security through local data storage
- Drastic reduction in development time
- Automatic application of best practices
- High scalability and flexibility
Practical applications range from AWS infrastructure development to knowledge base integration to complex automation scenarios.
The entry barrier is low – you don't need to be an AI expert to benefit from MCP.
MCP represents a fundamental evolution from traditional approaches:
- Dynamic vs. static knowledge access
- Distributed vs. centralized intelligence
- Standardized vs. custom implementations
- Real-time vs. pre-processed information

I'm convinced that MCP will significantly shape the future of AI integration in the cloud. We're just at the beginning of this development, and the possibilities that arise from it are nearly limitless. Imagine how your daily work could change if you had an AI assistant that not only has general knowledge but also deep understanding of your specific domain, your company data, and your technical environment.

My Personal Assessment

As someone who works with AWS and cloud technologies daily, I can say from personal experience: MCP has revolutionized my productivity. Tasks that used to take days, I now complete in hours or even minutes. And the best part: The quality of my work has improved because I can access specialized knowledge and best practices without having to be an expert in every area myself.

I see MCP as a decisive step toward a future where AI is not just a tool, but a real partner in development – a partner that understands you, knows your requirements, and helps you develop better solutions faster.

Your Next Steps

If you've become curious about MCP after this article (and I hope you have!), here are some recommendations for your next steps:

Experiment with AWS MCP Servers: The MCP servers provided by AWS are an excellent entry point. They are well-documented and easy to use.
Connect your preferred AI assistant with MCP: Whether you use Claude, Amazon Q, or other tools – many modern AI assistants already support MCP integration.
Identify use cases in your environment: Consider which recurring tasks in your daily work could benefit from MCP.
Share your experiences: The MCP community is growing rapidly, and your insights could help others.

Further Resources

If you want to dive deeper into the subject, here are some resources I can recommend:

AWS MCP Servers on GitHub – Open-source code and documentation
Amazon MCP Blog – Introducing AWS MCP Servers for code assistants (Part 1)

The AI revolution is in full swing, and with MCP, you have a powerful tool in hand to actively shape this revolution. I'm excited to see what innovative solutions you'll develop with it!

Have you already had experiences with MCP or do you have questions about it? I look forward to the exchange – let me know in the comments or contact me directly via LinkedIn.

Until next time – happy coding and good luck with MCP!

How I built my AWS AI assistant: Integrating Amazon Bedrock Agents with Slack

Artur Schneider — Tue, 01 Apr 2025 14:47:43 +0000

Ever felt like Dr. Frankenstein, creating a digital assistant that follows your every command? That's exactly what happened when I integrated Amazon Bedrock Agent into Slack! "IT'S ALIVE!" was literally my reaction when my AWS AI assistant sent its first response.

Unlike Mary Shelley's monster, this creation won't turn against you (as long as you set those IAM permissions right! 😅). It's the perfect colleague - never complains about overtime, doesn't steal your lunch from the fridge, and actually reads your documentation!

In this article, I'll show you how to build your own agentic AI assistant for AWS that you have FULL control over. The power is intoxicating! "Restart that EC2 instance, my faithful servant!" evil laugh

But before we dive into the technical details, let's talk about something crucial: security. When building AI assistants that can interact with your AWS environment, security isn't just a feature—it's a fundamental requirement. Throughout this guide, we'll emphasize secure implementation practices, from properly scoped IAM permissions to secure API endpoints and authentication mechanisms. Remember, with great power comes great responsibility, and we want our AI assistant to be helpful without creating security vulnerabilities.

I'll demonstrate the capabilities of this integration using a simple example of describing EC2 instances, but the potential use cases extend far beyond this. At the end of this article, I'll share additional use cases that leverage this integration while maintaining a strong security posture. Whether you're looking to streamline operations, enhance monitoring, or improve incident response, this integration can be adapted to meet various needs—all with security at its core.

So, let's kick off this journey to create our own AWS AI assistant that's both powerful and secure. By the end, you'll have a fully functional integration between Amazon Bedrock Agents and Slack that you can customize to your specific requirements.

Prerequisites

Before we dive into creating our monster... I mean, assistant, let's make sure you have everything you need:

AWS Prerequisites

An AWS account with appropriate permissions
Access to Amazon Bedrock service (check regional availability)
IAM permissions to create and manage:
- Lambda functions
- API Gateway resources
- Amazon Bedrock Agents
- IAM roles and policies

Slack Prerequisites

A Slack workspace where you have permissions to create apps
Admin or appropriate permissions to add apps to your workspace
Ability to create channels where your bot will post messages

Development Tools

Basic understanding of AWS services
Familiarity with JSON and REST APIs
Access to a code editor for Lambda function development
Basic understanding of Python (for Lambda functions)

Optional Prerequisites (depending on approach)

For Slack Bot approach: Slack Bot token with appropriate scopes
For Webhook approach: No additional requirements beyond Slack app creation

Understanding Amazon Bedrock Agent

Amazon Bedrock Agent is like the brain of our Frankenstein monster. It's a fully managed service that allows you to create agentic AI assistants powered by foundation models. These agents can be connected to your enterprise systems and data sources, allowing them to take actions on your behalf.

The key components of Amazon Bedrock Agent include:

Foundation Models: The underlying AI models that power your agent (Claude, Llama, etc.)
Knowledge Bases: Connect your data sources for the agent to reference
Action Groups: Define specific actions your agent can perform
API Schema: Define the structure of your API calls
Security Controls: IAM permissions to control what your agent can do

For our AWS AI assistant, we'll create an agent that can perform common AWS operations like checking EC2 instance status. This is just for simplicity - you can use this example code to adjust based on your requirements.

Architecture Overview

Let's take a look at the overall architecture of our integration:

The architecture consists of four main components:

Slack App: This is the interface through which users interact with our AI assistant. Users send messages to the bot, and the bot responds with information or performs actions based on those messages.
Amazon API Gateway: Acts as the secure entry point for requests from Slack. It receives webhook events from Slack and forwards them to our Lambda function. The API Gateway provides a layer of security by validating requests and controlling access.
AWS Lambda: The SlackHandlerLambda function processes incoming messages from Slack, invokes the Bedrock Agent, and sends responses back to Slack. This Lambda function is the bridge between Slack and Amazon Bedrock.
Amazon Bedrock: The brain of our operation. The Bedrock Agent processes user requests, understands their intent, and executes the appropriate actions through the EC2 Operations Lambda function.

From a security perspective, this architecture implements several important safeguards:

API Gateway Authentication: Validates incoming requests from Slack using signature verification
IAM Role-Based Access: Each Lambda function has a specific IAM role with only the permissions it needs
Secure Communication: All communication between components uses HTTPS
Request Validation: The SlackHandlerLambda validates incoming requests before processing them
Least Privilege Principle: The EC2 Operations Lambda has only the specific permissions needed to describe EC2 instances

Now that we understand the architecture, let's dive into creating each component, starting with our Amazon Bedrock Agent.

Creating Your Amazon Bedrock Agent

Let's start by creating our agent in Amazon Bedrock:

Navigate to the Amazon Bedrock console
Select "Agents" from the left navigation
Click "Create agent"
Provide a name for your agent (e.g., "AWS-AI-Assistant")
Select a foundation model (I recommend Claude for this use case)
Configure basic settings and click "Next"
Add instructions for the Agent - Your agent needs clear instructions to function properly. Without instructions, you'll get the dreaded "Agent Instruction cannot be null" error when trying to prepare your agent.

Here's an example of good instructions for our AWS AI Assistant:

You are an AWS AI Assistant designed to help users manage their AWS environment. 
Your primary functions include:
1. Providing information about AWS resources like EC2 instances, S3 buckets, and RDS databases
2. Performing basic operations like restarting EC2 instances
3. Monitoring AWS resources and providing status updates

You should be helpful, concise, and security-conscious. Always confirm before taking actions that modify resources. If you're unsure about a request, ask for clarification rather than guessing.

When users ask about AWS services you don't have direct access to, explain that you can only interact with services configured in your action groups.

Maintain a professional but friendly tone. You can use technical AWS terminology but explain concepts when needed for clarity.

These instructions serve as the personality and purpose guide for your agent. Without them, your agent won't know what it's supposed to do or how to behave, resulting in the preparation error.

Creating Action Groups

Now, let's define what our monster... I mean, assistant can do:

In the Agent Builder, select "Action groups"
Click "Add action group"
Name your action group (e.g., "EC2Operations")
Define your API schema. Here's the example for EC2 operations:

openapi: "3.0.1"
info:
  title: "EC2 Management API"
  version: "1.0.0"
paths:
  /instances:
    summary: "Operations to list EC2 instances"
    get:
      operationId: listInstances
      summary: "List running EC2 instances in a region"
      description: "Describe EC2 instances"
      parameters:
        - in: query
          name: region
          required: true
          schema:
            type: string
          description: "AWS region to check (e.g., us-east-1)"
      responses:
        '200':
          description: "Success"
          content:
            application/json:
              schema:
                type: object
                properties:
                  count:
                    type: integer
                    description: "Number of running EC2 instances in the region"

This schema defines a simple API endpoint that allows our agent to list EC2 instances in a specified region. The schema is intentionally kept simple for this example, but you can expand it to include more operations like starting, stopping, or describing specific instances.

Lambda Functions Implementation

Now, let's create the Lambda functions that will power our integration. We'll need two Lambda functions:

EC2 Operations Lambda: Handles AWS operations requested by the Bedrock Agent
Slack Handler Lambda: Processes incoming Slack messages and communicates with the Bedrock Agent

Creating the EC2 Operations Lambda

Navigate to the AWS Lambda console
Click "Create function"
Select "Author from scratch"
Name your function (e.g., "EC2OperationsLambda")
Select Python 3.9 as the runtime
Create a new execution role with basic Lambda permissions
Click "Create function"

Now, let's add the code for our Lambda function:

import json
import boto3
import os
import traceback
import datetime

def lambda_handler(event, context):
    print(f"Received event: {json.dumps(event)}")

    # Extract required fields from the event
    agent = event['agent']
    actionGroup = event['actionGroup']
    apiPath = event['apiPath']
    httpMethod = event['httpMethod']
    parameters = event.get('parameters', [])
    requestBody = event.get('requestBody', {})

    # Convert parameters list to dictionary if needed
    if isinstance(parameters, list):
        print("Parameters is a list, converting to dictionary")
        parameters_dict = {}
        for param in parameters:
            if isinstance(param, dict) and 'name' in param and 'value' in param:
                parameters_dict[param['name']] = param['value']
        parameters = parameters_dict

    print(f"API Path: {apiPath}")
    print(f"Parameters (processed): {json.dumps(parameters)}")

    try:
        # Handle different API paths based on the OpenAPI schema
        if apiPath == '/instances':
            print("Calling describe_instances function")
            result = describe_instances(parameters)
            print(f"describe_instances response: {json.dumps(result)}")

            # Format response according to Bedrock Agents expected structure
            # Convert result to JSON string to match the example format
            result_json = json.dumps(result)
            responseBody = {
                "application/json": {
                    "body": result_json
                }
            }

        else:
            print(f"Unsupported API path: {apiPath}")
            error_message = f'Unsupported API path: {apiPath}'

            # Format error response according to Bedrock Agents expected structure
            error_json = json.dumps({
                'error': error_message,
                '_source': 'lambda_error',
                '_warning': 'DO NOT FABRICATE OR MODIFY THIS DATA'
            })

            responseBody = {
                "application/json": {
                    "body": error_json
                }
            }

            # Create the action response in the expected format
            action_response = {
                'actionGroup': actionGroup,
                'apiPath': apiPath,
                'httpMethod': httpMethod,
                'httpStatusCode': 400,
                'responseBody': responseBody
            }

            # Return the final response with messageVersion
            api_response = {
                'response': action_response,
                'messageVersion': event['messageVersion']
            }

            print(f"Response: {json.dumps(api_response)}")
            return api_response

        # Create the action response in the expected format
        action_response = {
            'actionGroup': actionGroup,
            'apiPath': apiPath,
            'httpMethod': httpMethod,
            'httpStatusCode': 200,
            'responseBody': responseBody
        }

        # Return the final response with messageVersion
        api_response = {
            'response': action_response,
            'messageVersion': event['messageVersion']
        }

        print(f"Response: {json.dumps(api_response)}")
        return api_response

    except Exception as e:
        print(f"Error: {str(e)}")
        print(traceback.format_exc())

        # Format error response according to Bedrock Agents expected structure
        error_json = json.dumps({
            'error': str(e),
            '_source': 'lambda_error',
            '_warning': 'DO NOT FABRICATE OR MODIFY THIS DATA'
        })

        responseBody = {
            "application/json": {
                "body": error_json
            }
        }

        # Create the action response in the expected format
        action_response = {
            'actionGroup': actionGroup,
            'apiPath': apiPath,
            'httpMethod': httpMethod,
            'httpStatusCode': 500,
            'responseBody': responseBody
        }

        # Return the final response with messageVersion
        api_response = {
            'response': action_response,
            'messageVersion': event['messageVersion']
        }

        print(f"Response: {json.dumps(api_response)}")
        return api_response

def describe_instances(parameters):
    # Extract parameters - with additional error handling
    region = parameters.get('region', 'us-east-1') if isinstance(parameters, dict) else 'us-east-1'
    instance_id = parameters.get('instanceId', None) if isinstance(parameters, dict) else None

    print(f"Describing instances in region {region}, instance_id filter: {instance_id}")

    # Initialize EC2 client
    ec2 = boto3.client('ec2', region_name=region)

    # Prepare filters
    filters = []
    if instance_id:
        filters.append({
            'Name': 'instance-id',
            'Values': [instance_id]
        })

    # Describe instances with pagination to avoid timeouts
    instances = []
    try:
        # Use paginator for better handling of large results
        paginator = ec2.get_paginator('describe_instances')

        # Set a small page size to process results faster
        page_iterator = paginator.paginate(
            Filters=filters if filters else [],
            PaginationConfig={'MaxItems': 100, 'PageSize': 20}
        )

        # Process each page of results
        instance_count = 0
        for page in page_iterator:
            print(f"Processing page of EC2 results with {len(page['Reservations'])} reservations")

            for reservation in page['Reservations']:
                for instance in reservation['Instances']:
                    instance_count += 1
                    instance_info = {
                        'InstanceId': instance['InstanceId'],
                        'InstanceType': instance['InstanceType'],
                        'State': instance['State']['Name'],
                        'LaunchTime': instance['LaunchTime'].isoformat(),
                        'PublicIpAddress': instance.get('PublicIpAddress', 'N/A'),
                        'PrivateIpAddress': instance.get('PrivateIpAddress', 'N/A')
                    }
                    print(f"Found instance {instance_count}: {instance_info['InstanceId']}")
                    instances.append(instance_info)

                    # Limit the number of instances to return to avoid response size issues
                    if instance_count >= 50:
                        print(f"Reached maximum instance count (50), stopping pagination")
                        break

                if instance_count >= 50:
                    break

            if instance_count >= 50:
                break

    except Exception as e:
        print(f"Error calling EC2 API: {str(e)}")
        raise

    # Format the response according to the OpenAPI schema
    # Use current timestamp instead of credentials expiry time
    current_time = datetime.datetime.now().isoformat()

    result = {
        'Instances': instances,
        'Count': len(instances),
        'Region': region,
        'DataTimestamp': current_time,
        '_source': 'lambda_actual_data',
        '_warning': 'DO NOT FABRICATE OR MODIFY THIS DATA'
    }

    print(f"Returning {len(instances)} instances")
    return result

This Lambda function handles requests from the Bedrock Agent to describe EC2 instances. Let's break down what it does:

The lambda_handler function processes incoming events from the Bedrock Agent, extracting the API path and parameters.
It handles the /instances API path by calling the describe_instances function.
The describe_instances function uses the AWS SDK to query EC2 instances in the specified region.
It formats the response according to the expected structure for Bedrock Agents.
Error handling is implemented throughout to ensure robust operation.

Important: Make sure to extend the Lambda timeout beyond the default 3 seconds. For the EC2 Operations Lambda, I recommend setting it to at least 30 seconds, as querying EC2 instances across regions can take time, especially with pagination.

Creating the Slack Handler Lambda

Now, let's create the Lambda function that will handle Slack interactions:

Navigate to the AWS Lambda console
Click "Create function"
Select "Author from scratch"
Name your function (e.g., "SlackHandlerLambda")
Select Python 3.9 as the runtime
Create a new execution role with basic Lambda permissions
Click "Create function"

Now, let's add the code for our Slack Handler Lambda:

import os
import json
import hmac, hashlib
import boto3
import base64
import time
from datetime import datetime, timedelta
from botocore.exceptions import ClientError

# Read important constants from environment or configuration
BEDROCK_AGENT_ID = os.environ.get("BEDROCK_AGENT_ID")       # e.g., "agent-1234567890abcdef"
BEDROCK_AGENT_ALIAS = os.environ.get("BEDROCK_AGENT_ALIAS") # e.g., the alias ID (not the name)
SLACK_BOT_TOKEN = os.environ.get("SLACK_BOT_TOKEN")         # xoxb- token
SLACK_SIGNING_SECRET = os.environ.get("SLACK_SIGNING_SECRET")  # Slack signing secret
BEDROCK_REGION = os.environ.get("BEDROCK_REGION", "eu-central-1")  # Default to eu-central-1 if not specified

# Initialize the Bedrock client - Using bedrock-agent-runtime for agent invocation with region from environment variable
bedrock_client = boto3.client("bedrock-agent-runtime", region_name=BEDROCK_REGION)

# In-memory cache for event deduplication
processed_events = {}

def lambda_handler(event, context):
    # If coming via API Gateway HTTP API, the actual Slack payload will be in 'body'
    body = event.get('body')
    if body:
        # The body might be URL-encoded from Slack, but Slack can send JSON if configured.
        # Assume JSON for simplicity:
        try:
            slack_event = json.loads(body)
        except json.JSONDecodeError:
            # If the body was URL encoded (application/x-www-form-urlencoded), parse it differently
            import urllib.parse
            slack_event = json.loads(urllib.parse.parse_qs(body)['payload'][0]) if 'payload' in body else urllib.parse.parse_qs(body)
    else:
        # If running in a Flask app, you might directly get JSON
        slack_event = event

    # 1. Verification challenge handshake
    if slack_event.get("type") == "url_verification":
        challenge = slack_event.get("challenge", "")

        # Respond with the challenge token to verify endpoint
        return {
            "statusCode": 200,
            "headers": {"Content-Type": "text/plain"},
            "body": challenge
        }

    # 2. Verify Slack request signature (to ensure request is from Slack)
    headers = event.get('headers', {})
    if SLACK_SIGNING_SECRET:
        timestamp = headers.get('x-slack-request-timestamp') or headers.get('X-Slack-Request-Timestamp')
        signature = headers.get('x-slack-signature') or headers.get('X-Slack-Signature')
        if not verify_slack_signature(SLACK_SIGNING_SECRET, body, timestamp, signature):
            return {"statusCode": 401, "body": "Invalid signature"}  # reject if signature check fails

    # 3. Process event callbacks
    # Slack might batch events; we handle one event at a time for simplicity.
    if "event" in slack_event:
        event_info = slack_event["event"]
        user_id = event_info.get("user")
        text = event_info.get("text", "")
        channel_id = event_info.get("channel")

        # Ignore events that are from the bot itself to prevent loops
        if event_info.get("bot_id") or user_id is None:
            return {"statusCode": 200}  # nothing to do

        # Event deduplication - Check if we've seen this event recently
        event_id = slack_event.get("event_id") or slack_event.get("event_time")
        if not event_id:
            # Create a synthetic ID if none exists
            event_id = f"{channel_id}:{user_id}:{text[:10]}:{int(time.time())}"

        # Check if we've seen this event recently (within last 5 minutes)
        now = datetime.now()
        if event_id in processed_events:
            last_processed = processed_events[event_id]
            if now - last_processed < timedelta(minutes=5):
                return {"statusCode": 200}  # Acknowledge but don't process

        # Mark this event as processed
        processed_events[event_id] = now

        # Clean up old entries to prevent memory growth
        for old_id in list(processed_events.keys()):
            if now - processed_events[old_id] > timedelta(minutes=30):
                del processed_events[old_id]

        # (Optional auth) Check if user_id is allowed to use the bot
        allowed_users = os.environ.get("ALLOWED_USER_IDS")
        if allowed_users:
            allowed_list = [u.strip() for u in allowed_users.split(",")]
            if user_id not in allowed_list:
                # Notify user they are not authorized
                send_slack_message(channel_id, ":x: You are not authorized to use this bot.")
                return {"statusCode": 200}

        # Validate Bedrock agent configuration
        if not BEDROCK_AGENT_ID or not BEDROCK_AGENT_ALIAS:
            error_msg = "Bedrock agent configuration is incomplete. Please check BEDROCK_AGENT_ID and BEDROCK_AGENT_ALIAS environment variables."
            send_slack_message(channel_id, f"⚠️ Configuration Error: {error_msg}")
            return {"statusCode": 200}

        # 4. Invoke the Bedrock agent with the user's text
        session_id = channel_id  # use channel (or user) as session identifier for continuity
        try:
            # Using the correct method with the bedrock-agent-runtime client
            response = bedrock_client.invoke_agent(
                agentId=BEDROCK_AGENT_ID,
                agentAliasId=BEDROCK_AGENT_ALIAS,
                sessionId=session_id,
                inputText=text
            )

            # The response is streamed in chunks. Combine them to get full answer.
            answer = ""
            for chunk in response.get("completion", []):
                chunk_text = chunk["chunk"]["bytes"].decode('utf-8', errors='ignore')
                answer += chunk_text

            # 5. Send the agent's answer back to Slack
            if answer:
                send_slack_message(channel_id, answer)
            else:
                # If no answer (empty completion), send a default reply
                send_slack_message(channel_id, "🤖 (No response)")

        except ClientError as e:
            error_code = e.response.get('Error', {}).get('Code', '')
            error_message = e.response.get('Error', {}).get('Message', str(e))

            if error_code == 'ResourceNotFoundException':
                send_slack_message(channel_id, f"Sorry, I couldn't process that request. Error: Resource not found. Please check your Bedrock agent configuration.")
            else:
                send_slack_message(channel_id, f"Sorry, I couldn't process that request. Error: {error_message}")

        except Exception as e:
            # Log error and reply with a failure message
            send_slack_message(channel_id, f"Sorry, I couldn't process that request. Error: {str(e)}")

    # Return 200 to acknowledge the event was received
    return {"statusCode": 200}

# Utility to verify Slack signatures (optional but recommended)
def verify_slack_signature(signing_secret, request_body, timestamp, signature):
    if not signature or not timestamp:
        return False
    # Slack sends 'v0=' prefix in signature
    req = str.encode('v0:' + str(timestamp) + ':' + request_body) 
    secret = str.encode(signing_secret)
    hash_hex = 'v0=' + hmac.new(secret, req, hashlib.sha256).hexdigest()
    return hmac.compare_digest(hash_hex, signature)

# Function to send messages to Slack without using requests library
def send_slack_message(channel_id, text):
    """
    Send a message to a Slack channel using AWS Lambda's URL fetch capabilities
    instead of the requests library.
    """
    try:
        # Prepare the request payload
        payload = {
            "channel": channel_id,
            "text": text
        }

        # Convert payload to JSON string
        payload_json = json.dumps(payload)

        # Create a presigned URL for the Slack API
        url = "https://slack.com/api/chat.postMessage"

        # Use AWS SDK to make the HTTP request
        import urllib.request

        # Create a request object
        req = urllib.request.Request(url)

        # Add headers
        req.add_header("Content-Type", "application/json")
        req.add_header("Authorization", f"Bearer {SLACK_BOT_TOKEN}")

        # Send the request
        with urllib.request.urlopen(req, data=payload_json.encode('utf-8')) as response:
            response_body = response.read().decode('utf-8')

            # Parse the response
            response_json = json.loads(response_body)
            if not response_json.get("ok", False):
                print(f"Error sending message to Slack: {response_json.get('error', 'Unknown error')}")

    except Exception as e:
        print(f"Error sending message to Slack: {str(e)}")

This Lambda function handles the communication between Slack and the Bedrock Agent. Let's break down what it does:

It processes incoming webhook events from Slack.
It verifies the Slack signature to ensure the request is legitimate.
It extracts the user's message and invokes the Bedrock Agent with that message.
It receives the response from the Bedrock Agent and sends it back to the user in Slack.
It includes error handling and event deduplication to prevent processing the same message multiple times.

Important: The SlackHandlerLambda requires several environment variables to function properly:

BEDROCK_AGENT_ID: The ID of your Bedrock Agent (e.g., "agent-1234567890abcdef")

BEDROCK_AGENT_ALIAS: The alias ID of your Bedrock Agent (not the name)

SLACK_BOT_TOKEN: Your Slack Bot token (starts with "xoxb-")

SLACK_SIGNING_SECRET: Your Slack signing secret for request verification

BEDROCK_REGION: The AWS region where your Bedrock Agent is deployed (e.g., "eu-central-1")

ALLOWED_USER_IDS (optional): Comma-separated list of Slack user IDs allowed to use the bot

Also, make sure to extend the Lambda timeout beyond the default 3 seconds. For the SlackHandlerLambda, I recommend setting it to at least 30 seconds, as the Bedrock Agent invocation can take time, especially for complex queries.

Connecting the Lambda Function to the Action Group

This is a crucial step that connects your Lambda function to your action group:

In the Amazon Bedrock console, navigate back to your agent's action group
In the "Action group invocation" section, you have three options:
- Quick create a new Lambda function - Amazon Bedrock creates a basic Lambda function for you
- Select an existing Lambda function - Choose the Lambda function you created earlier
- Return control - The agent returns control to your application without invoking a Lambda
Select "Select an existing Lambda function"
Choose your Lambda function from the dropdown (e.g., "EC2OperationsLambda")
Select the appropriate function version (usually $LATEST)
Click "Save" to connect your Lambda function to the action group

Additionally, you need to grant Amazon Bedrock permission to invoke your Lambda function:

Navigate to the Lambda console and select your function
Go to the "Permissions" tab
Under "Resource-based policy statements", click "Add permissions"
Choose "AWS service" as the principal
For "Service", select "bedrock.amazonaws.com"
For "Action", select "lambda:InvokeFunction"
Click "Save" to add the permission

Sidenote: If you create the Lambda through the Bedrock Console Action Group view, the resource-based policy statements will be set automatically otherwise you need to add that into your Lambda manually.

Setting Up IAM Permissions

Remember Dr. Frankenstein's mistake? He gave his creation too much freedom. Learn from his error and implement proper IAM permissions! Your monster should only have exactly the access it needs - no more wandering into the village terrifying the EC2 instances.

Let's create a proper IAM policy for our EC2 Operations Lambda function:

Navigate to the IAM console
Select "Policies" and click "Create policy"
Use the JSON editor to create a policy:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances"
      ],
      "Resource": "*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}

Name your policy (e.g., "BedrockAgentEC2Operations")
Create the policy
Attach this policy to your Lambda function's execution role

For the SlackHandlerLambda, you'll need a different policy that allows it to invoke the Bedrock Agent:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "bedrock:InvokeAgent"
      ],
      "Resource": "arn:aws:bedrock:*:*:agent/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "logs:CreateLogGroup",
        "logs:CreateLogStream",
        "logs:PutLogEvents"
      ],
      "Resource": "arn:aws:logs:*:*:*"
    }
  ]
}

Attach this policy to your SlackHandlerLambda's execution role.

API Gateway Setup

Now that we have our Lambda functions ready, we need to create an API Gateway to receive webhook events from Slack. This will serve as the secure entry point for our integration.

Creating the API Gateway

Navigate to the Amazon API Gateway console
Click "Create API"
Select "HTTP API" (not REST API) for a simpler, more cost-effective solution
Click "Build"
In the "Integrations" section, select "Lambda"
Choose your SlackHandlerLambda function
For "API name", enter a descriptive name (e.g., "slack-bedrock-integration")
Click "Next"

Configuring Routes

In the "Configure routes" section, click "Add route"
Set the method to "POST"
Set the path to "/" (or any path you prefer)
Select your SlackHandlerLambda as the integration target
Click "Next"

Configuring Stages

In the "Configure stages" section, keep the default stage name "$default"
Enable automatic deployments
Click "Next"

Reviewing and Creating

Review your API configuration
Click "Create"
Once created, note the API endpoint URL (e.g., "https://abcdef123.execute-api.us-east-1.amazonaws.com")

Enhancing API Gateway Security

For a production environment, you should implement additional security measures for your API Gateway:

Add Request Validation:
- Navigate to your API in the API Gateway console
- Select "Routes" and then your POST route
- Click "Edit"
- Under "Request validator", select "Validate body"
- Click "Save"
Add Throttling:
- Navigate to your API in the API Gateway console
- Select "Stages" and then your stage
- Click "Edit"
- Set appropriate rate and burst limits (e.g., 10 requests per second)
- Click "Save"
Add CORS Configuration (if needed):
- Navigate to your API in the API Gateway console
- Select "CORS"
- Configure appropriate CORS settings
- Click "Save"
Add WAF Integration (optional but recommended for production):
- Navigate to the AWS WAF console
- Create a web ACL with appropriate rules
- Associate the web ACL with your API Gateway
Add CloudWatch Logging:
- Navigate to your API in the API Gateway console
- Select "Logging"
- Enable CloudWatch logging
- Set an appropriate log level (e.g., ERROR for production)
- Click "Save"

These security enhancements will help protect your API Gateway from common threats and ensure it operates reliably in a production environment.

Testing the API Gateway

Before proceeding to the Slack integration, let's test our API Gateway to ensure it's working correctly:

Use a tool like curl or Postman to send a POST request to your API endpoint
Include a simple JSON payload in the request body
Check the CloudWatch logs for your SlackHandlerLambda to ensure it received and processed the request

If everything is working correctly, you should see logs indicating that your Lambda function was invoked and processed the request. Now we're ready to integrate with Slack!

Slack Integration

Now that we have our Bedrock Agent and API Gateway set up, it's time to integrate with Slack. This is where our creation truly comes to life!

Creating a Slack App

Go to the Slack API website
Click "Create New App"
Choose "From scratch"
Enter a name for your app (e.g., "AWS AI Assistant")
Select the workspace where you want to install the app
Click "Create App"

Configuring Bot Token Scopes

For our Slack Bot to function properly, we need to configure the appropriate permissions:

In the left sidebar, click on "OAuth & Permissions"
Scroll down to the "Scopes" section
Under "Bot Token Scopes", click "Add an OAuth Scope"
Add the following scopes:
- app_mentions:read - View messages that directly mention @AWS AI Assistant in conversations that the app is in
- chat:write - Send messages as @AWS AI Assistant
- im:history - View messages and other content in direct messages that AWS AI Assistant has been added to
- im:read - View basic information about direct messages that AWS AI Assistant has been added to
- im:write - Start direct messages with people
- mpim:write - Start group direct messages with people

These scopes allow your bot to read messages, send responses, and interact with users in direct messages and group conversations.

Activating Messages Tab in App Home

An important step that's easy to miss is activating the Messages Tab in your App Home:

In the left sidebar, click on "App Home"
Scroll down to the "Show Tabs" section
Enable the "Messages Tab"
Save your changes

This step is crucial for allowing users to send direct messages to your bot. Without it, users won't be able to initiate conversations with your assistant.

Setting Up Event Subscriptions

Now, we need to configure Slack to send events to our API Gateway:

In the left sidebar, click on "Event Subscriptions"
Toggle "Enable Events" to On
In the "Request URL" field, enter your API Gateway URL
Slack will send a verification challenge to your endpoint - if your Lambda function is set up correctly, it should automatically respond and verify the URL
Under "Subscribe to bot events", click "Add Bot User Event"
Add the following events:
- message.im - Subscribe to direct messages sent to your bot
- app_mention - Subscribe to mentions of your bot in channels
Click "Save Changes"

Installing the App to Your Workspace

In the left sidebar, click on "Install App"
Click "Install to Workspace"
Review the permissions and click "Allow"
Copy the "Bot User OAuth Token" (starts with xoxb-) - you'll need this for your SlackHandlerLambda environment variable

Updating the SlackHandlerLambda Environment Variables

Now that we have our Slack app set up, we need to update the environment variables in our SlackHandlerLambda:

Navigate to the AWS Lambda console
Select your SlackHandlerLambda function
Scroll down to the "Environment variables" section
Add the following environment variables:
- BEDROCK_AGENT_ID: Your Bedrock Agent ID
- BEDROCK_AGENT_ALIAS: Your Bedrock Agent Alias ID
- SLACK_BOT_TOKEN: The Bot User OAuth Token you copied earlier
- SLACK_SIGNING_SECRET: Found in the "Basic Information" section of your Slack app under "App Credentials"
- BEDROCK_REGION: The AWS region where your Bedrock Agent is deployed
Click "Save"

Testing the Integration

Now it's time to test our integration! There are two ways to test:

Testing in the Bedrock Agent Console:

Navigate to the Amazon Bedrock console
Select your agent
Click "Test"
Try asking about your EC2 instances: "Show me the ec2 instances in eu-central-1"

Testing in Slack:

Open your Slack workspace
Find your bot in the Apps section or direct messages
Send a message like "Show me the ec2 instances in eu-central-1"
Your bot should respond with a list of instances

If everything is set up correctly, you should see your bot responding with information about your EC2 instances. Congratulations! Your AWS AI Assistant is alive and working!

Troubleshooting Common Issues

If you encounter issues with your integration, here are some common problems and solutions:

Bot not responding in Slack:
- Check that your SlackHandlerLambda environment variables are set correctly
- Verify that your API Gateway endpoint is correctly configured in Slack's Event Subscriptions
- Check the CloudWatch logs for your SlackHandlerLambda for error messages
- Ensure the Messages Tab is activated in App Home
Lambda timeout errors:
- Increase the timeout for both Lambda functions beyond the default 3 seconds
- For SlackHandlerLambda, set it to at least 60 seconds
- For EC2OperationsLambda, set it to at least 30 seconds
Permission errors:
- Verify that your Lambda functions have the correct IAM permissions
- Check that Bedrock has permission to invoke your EC2OperationsLambda
- Ensure your Slack app has all the required scopes
Bedrock Agent errors:
- Make sure your agent is properly prepared and working in the test console
- Verify that your action group schema matches the implementation in your Lambda function
- Check that your agent has clear instructions

By following these troubleshooting steps, you should be able to resolve most common issues with your integration.

Security Considerations

Throughout this guide, we've touched on various security aspects, but let's take a moment to dive deeper into the security considerations for this integration. When building an AI assistant that can interact with your AWS environment, security should be your top priority.

IAM Permissions and Least Privilege

The principle of least privilege is crucial when setting up IAM permissions for your Lambda functions. Your EC2OperationsLambda should only have the specific permissions it needs to perform its tasks - in our example, that's just ec2:DescribeInstances. If you expand the functionality to include other operations, add only the specific permissions needed for those operations.

For example, if you want to allow your assistant to start and stop instances, you would add only these specific permissions:

{
  "Effect": "Allow",
  "Action": [
    "ec2:StartInstances",
    "ec2:StopInstances"
  ],
  "Resource": "arn:aws:ec2:*:*:instance/*"
}

Never give your Lambda functions broad permissions like ec2:* as this would allow the function to perform any EC2 operation, including deleting instances or creating new ones.

Slack App Security

Your Slack app is the interface through which users interact with your AWS environment, so it's important to secure it properly:

Request Verification: Always verify that requests are coming from Slack using the signing secret and signature verification process.
User Authorization: Consider implementing user-based authorization to control who can use your bot. The SlackHandlerLambda includes an optional ALLOWED_USER_IDS environment variable for this purpose.
Token Security: Store your Slack tokens securely. In our example, we use environment variables, but for production, consider using AWS Secrets Manager.
Scoped Permissions: Only request the specific Slack scopes your bot needs. Avoid requesting broad scopes like channels:read if you only need to interact with direct messages.

API Gateway Security

Your API Gateway is the entry point to your AWS environment, so it needs to be properly secured:

Request Validation: Validate incoming requests to ensure they match the expected format.
Throttling: Implement throttling to prevent abuse and denial-of-service attacks.
WAF Integration: Consider integrating with AWS WAF to protect against common web exploits.
Logging and Monitoring: Enable detailed logging and set up monitoring to detect unusual activity.

Data Handling and Privacy

When handling user data and AWS resource information, consider these privacy aspects:

Data Minimization: Only collect and process the data you need.
Data Retention: Don't store sensitive information longer than necessary.
Secure Transmission: Ensure all data is transmitted securely using HTTPS.
Response Sanitization: Be careful about what information you return to users. For example, you might want to redact or mask certain sensitive information like IP addresses.

Additional Use Cases

While our example focused on describing EC2 instances, this integration can be extended to support a wide range of use cases. Here are some ideas, all with security in mind:

Security-Focused Use Cases

Security Posture Checking: Ask your assistant to check for security issues like open security groups, public S3 buckets, or IAM users without MFA.

Example: "Check for S3 buckets with public access in all regions"

Compliance Verification: Verify that resources comply with your organization's security policies.

Example: "Show me EC2 instances that don't have encryption enabled"

Security Incident Response: Use your assistant to help with security incident response.

Example: "Show me all CloudTrail events for user john.doe in the last 24 hours"

Security Notifications: Set up your assistant to notify you about security events.

Example: "Alert me when there are failed login attempts to the AWS console"

Operational Efficiency Use Cases

Resource Monitoring: Monitor the status and health of your AWS resources.

Example: "Show me all RDS instances with high CPU usage"

Cost Optimization: Identify opportunities to reduce costs.

Example: "Find unused EBS volumes across all regions"

Automated Remediation: Automate common remediation tasks.

Example: "Stop all development environment instances outside of business hours"

Documentation Access: Access and search through your documentation.

Example: "What's our process for deploying to production?"

DevOps Use Cases

Deployment Status: Check the status of your deployments.

Example: "What's the status of the latest deployment to production?"

Infrastructure Management: Manage your infrastructure through natural language.

Example: "Scale the web server auto-scaling group to 5 instances"

Log Analysis: Search and analyze logs.

Example: "Show me error logs from the payment service in the last hour"

Environment Management: Manage different environments.

Example: "Compare the configuration between staging and production"

Conclusion

In this guide, we've built a powerful AWS AI assistant by integrating Amazon Bedrock Agents with Slack. We've created a secure, scalable architecture that allows users to interact with AWS resources using natural language.

The integration we've built demonstrates the power of agentic AI assistants in simplifying AWS operations. By leveraging Amazon Bedrock's capabilities and connecting them to Slack, we've created a tool that can significantly improve productivity and accessibility of AWS resources.

Remember that security should always be at the forefront when building tools that interact with your AWS environment. By following the principle of least privilege, implementing proper authentication and authorization, and being mindful of data privacy, you can create a powerful assistant without compromising security.

As you expand your assistant's capabilities beyond the EC2 instance example we've covered, continue to apply these security principles to each new feature. With the right balance of functionality and security, your AWS AI assistant can become an invaluable tool for your team.

So go ahead, bring your AWS AI assistant to life! Just remember to give it the right permissions, unlike Dr. Frankenstein's creation. Your monster should be helpful, not terrifying! 😉

Securing and enhancing LLM prompts & outputs: A guide using Amazon Bedrock Guardrails and open-source solutions

Artur Schneider — Tue, 01 Oct 2024 13:53:18 +0000

In my conversations with customers, I frequently encounter the same critical questions: "How do you secure your LLMs?" and "How do you ensure the quality of the answers?" These concerns highlight a common hesitation among businesses to provide solutions with LLM intelligence to their own customers due to the significant risks. These risks include not only the possibility of incorrect outputs, which can damage a business's reputation and adversely affect their customers, but also security threats such as prompt injection attacks that could lead to the loss of proprietary data or other sensitive information.

Given these concerns, this blog post aims to address the twin challenges of securing LLMs and ensuring their output quality. Leveraging my AWS background, we will explore the capabilities of Amazon Bedrock native capabilities as well as various open-source tools. We will explore strategies to mitigate these risks and enhance the reliability of your LLM-powered applications.

Understanding the risks

As technology evolves, so do the security threats associated with it. The rapid advancement of Large Language Models (LLMs) has brought significant benefits, but it has also introduced new challenges. Ensuring the security and quality of LLM outputs is crucial to prevent potential harm and misuse. Before we dive into the solutions, let’s first understand the key security concerns and quality issues that businesses need to address when offering LLM solutions to their customers.

For example, in 2023, OpenAI faced scrutiny when a bug allowed users to see snippets of other users' chat histories, raising serious data privacy concerns ChatGPT confirms data breach, raising security concerns. Similarly, Google's AI chat tool Bard sparked controversy due to biased and factually incorrect responses during its debut, demonstrating the importance of maintaining output quality Google’s Bard: AI chatbot makes $100bn mistake. In this section, we will explore the various security concerns and quality issues that businesses must address when making LLM solutions available to their customers.

Security Concerns

Data Privacy: Sensitive data used for training can be inadvertently exposed.LLMs, if not properly managed, can leak information contained in the training data.
Prompt Injection Attacks: Malicious inputs can be crafted to manipulate the model's behavior, potentially causing it to reveal sensitive information or perform unintended actions.
Model Theft: Unauthorized access to the model can lead to intellectual property theft, compromising the proprietary techniques and data used in its development.
Adversarial Attacks: Adversarial inputs designed to deceive the model can cause it to generate harmful or incorrect outputs, impacting the reliability of the LLM.
Unauthorized Access:Insufficient access controls can result in unauthorized personnel gaining access to the model, data, or infrastructure, leading to potential misuse.
Inference Attacks: Attackers can infer sensitive attributes about the training data by analyzing the model's outputs, posing a privacy risk.
Data Poisoning: Attackers can corrupt the training data with malicious content to compromise the integrity of the model's outputs.
Model Evasion: Attackers may develop techniques to bypass security measures and exploit vulnerabilities in the model.

Quality Issues

Bias and Fairness: LLMs can perpetuate and amplify biases present in training data, leading to outputs that are unfair or discriminatory.
Accuracy: Generated content can be factually incorrect or misleading, which is critical to address, especially in domains where incorrect information can have serious consequences.
Coherence and Relevance: Outputs need to be contextually appropriate and coherent. Inconsistent or irrelevant responses can undermine the usefulness and trustworthiness of the LLM.
Ethical Considerations: Ensuring that the model's outputs adhere to ethical standards and do not produce harmful, offensive, or inappropriate content.
Robustness: Ensuring that the model can handle a wide range of inputs without producing errors or undesirable outputs.
Transparency and Explainability: Making sure that the model’s decision-making process is transparent and its outputs are explainable, which helps in building trust with users.

How to secure and ensure high-quality inputs and outputs for LLMs with Amazon Bedrock Guardrails

Amazon Bedrock provides built-in capabilities to secure and validate Large Language Model (LLM) inputs and outputs through its native guardrails feature. These guardrails are essential for ensuring that LLMs behave according to predefined security, ethical, and operational standards. With Guardrails, you can set up filtering mechanisms to protect against harmful outputs, inappropriate inputs, sensitive information leakage, and more. In this section, we will dive into the details of how you can implement these features using Amazon Bedrock, walking through the setup process with visual aids.

Step 1: Provide Guardrail Details

In the first step, you can define the basic information for your guardrail setup:

Name and Description: Here, you define a name (e.g., "DemoGuardrail") and a description of what the guardrail is designed to do (e.g., "Guardrail for demos").
Messaging for Blocked Prompts: If a prompt is blocked by the guardrail, you can customize the message shown to users, such as “Sorry, the model cannot answer this question.”
KMS Key Selection (Optional): Optionally, you can select a KMS (Key Management Service) key to encrypt sensitive information within this guardrail.

This provides a foundation for guardrail implementation, allowing you to define how the model responds to blocked content.

Step 2: Configure Content Filters

Content filters allow you to detect and block harmful user inputs and model responses across predefined categories like Hate, Insults, Sexual Content, Violence, and Misconduct.

Harmful Categories: For each category, you can adjust the sensitivity to "None", "Low", "Medium", or "High". This flexibility allows you to fine-tune how strictly the model filters content.
Prompt Attacks: Enabling prompt attack filters helps detect and block user inputs attempting to override the system's instructions. You can adjust the sensitivity for prompt attacks to ensure robust protection against injection attacks.

These filters are crucial for preventing harmful or unwanted content from being generated by the model or entered by users.

Step 3: Add Denied Topics

You can define specific topics that should not be discussed by the model, ensuring the LLM doesn’t respond to sensitive or restricted queries.

Denied Topics: You can add up to 30 denied topics (e.g., "Investment"), and provide sample phrases that the model should block related to that topic.
Customization: For each topic, you define a clear explanation and add up to five phrases (e.g., “Where should I invest my money?”) to ensure the model avoids restricted discussions.

This helps prevent the LLM from engaging in specific conversations, such as offering financial or medical advice.

Step 4: Add Word Filters

With word filters, you can further refine the model's behavior by blocking certain words or phrases from being used in inputs and outputs.

Profanity Filter: This built-in filter allows you to block profane words globally across inputs and outputs.
Custom Word List: You can manually add specific words or phrases, upload them from a local file, or use an S3 object. This lets you block specific terminology that may be inappropriate for your use case.

These word filters ensure that sensitive or inappropriate terms do not appear in the LLM's responses or user inputs.

Step 5: Add Sensitive Information Filters

This step focuses on safeguarding personally identifiable information (PII). You can specify which types of PII should be masked or blocked in LLM responses.

PII Types: The system lets you add specific PII types such as Phone Numbers, Email Addresses, and Credit/Debit Card Numbers.
Guardrail Behavior: For each PII type, you can choose to either mask or block it completely, ensuring that sensitive information is not inadvertently exposed.

This ensures robust data protection and compliance with privacy regulations like GDPR.

Step 6: Add Contextual Grounding Check

One of the key features for ensuring output quality is the contextual grounding check. This validates whether model responses are grounded in factual information and are relevant to the user’s query.

Grounding Check: The grounding check ensures that model responses are factually correct and based on provided reference sources.
Relevance Check: This feature validates whether model responses are relevant to the query and blocks outputs that fall below a defined threshold for relevance and accuracy.

These checks are particularly useful in preventing hallucinations, where the model generates incorrect or irrelevant responses.

Step 7: Review and Create Guardrails

After configuring all necessary filters and checks, you can review your setup and create the guardrail. Once activated, the system allows you to immediately test the guardrail directly within the console by entering prompts to see how it blocks or modifies responses according to your settings. Additionally, you can attach the guardrail to your specific use case to ensure it functions correctly in real-world scenarios, providing a seamless way to protect and enhance your LLM application.

Amazon Bedrock provides powerful native capabilties to secure and ensure high-quality outputs for your LLMs, allowing you to protect against harmful content, ensure data privacy, and prevent model hallucinations. By configuring content filters, denied topics, sensitive information filters, and grounding checks, you can fine-tune your model to meet security, ethical, and operational standards. Now, let's explore some open-source solutions that can be implemented to secure your LLM independently of using Amazon Bedrock.

How to secure and ensure high-quality inputs and outputs for LLMs with open-source solutions

When it comes to securing and ensuring high-quality outputs from LLMs, there are numerous open-source solutions available. The choice of solution depends on your specific demands and use case, such as the level of security required, the sensitivity of the data being processed, and the quality standards your business must meet.

In this section, we will focus on two of the most popular and widely adopted open-source tools for securing LLMs and maintaining output quality:

LLM Guard by Protect AI - A robust solution designed to provide security for LLMs, protecting against various input and output vulnerabilities.
DeepEval by Confident AI - A leading tool for evaluating and maintaining the quality of LLM outputs, ensuring accuracy, coherence, and relevance in responses.

Both solutions offer extensive features to enhance the security and quality of your LLM applications. Let's take a closer look at how they work and the benefits they offer.

How to secure your LLM input and output with LLM Guard

LLM Guard, developed by Protect AI, is an open-source solution that acts as a proxy between your application and the LLM, filtering inputs and outputs in real-time. By sitting between your application and the LLM, LLM Guard ensures that sensitive data, inappropriate content, or malicious prompt injections are intercepted and handled before they reach or leave the model. This makes LLM Guard highly adaptable to any environment where LLMs are integrated, allowing seamless deployment without directly modifying your LLM's architecture. The tool can be easily included into existing LLM workflows, serving as a middle layer to secure the entire interaction cycle between users and the model.

> Image source:https://llm-guard.com/

Key features of LLM Guard include:

Input and Output Filtering: LLM Guard applies a wide range of filters, including anonymization, topic banning, regex matching, and more, to ensure that inputs and outputs comply with security protocols.
Prompt Injection Protection: The tool is designed to detect and block prompt injection attempts that could lead to unwanted or harmful behaviors from the LLM.
PII Detection and Redaction: LLM Guard automatically identifies and redacts sensitive information, such as names, phone numbers, and email addresses, ensuring that private data is not exposed in outputs.
Customizable Scanners: LLM Guard allows users to define specific "scanners" that monitor for different types of sensitive or inappropriate content, giving flexibility in controlling the behavior of the LLM.

LLM Guard can be easily integrated into your infrastructure as it functions as a proxy, ensuring that all inputs and outputs go through a comprehensive security check before and after interacting with the LLM. You can find more details on the project’s code and features on the LLM Guard GitHub repository. To test the tool interactively, Protect AI has provided a playground hosted on Hugging Face, where you can try different filters and configurations.

Let’s now walk through how LLM Guard functions using the Hugging Face playground and real-world examples of processing inputs and outputs.

Step 1: Setting Up Input Filters

When configuring LLM Guard, you have the flexibility to apply filters to either prompts (inputs) or outputs, ensuring that both the data being sent to the model and the data generated by the model are secure and compliant. The range of scanners allows for thorough customization, and each scanner can be individually adjusted to meet specific security and compliance needs.

You can activate multiple scanners based on your requirements. Additionally, each scanner offers fine-tuned control, allowing you to modify thresholds, sensitivity, or specific filter behaviors. For example, you can set the strictness of the BanCode scanner or configure the Anonymize scanner to target specific entities such as credit card numbers or email addresses.

Prompt (Input) scanners:
Anonymize, BanCode, BanCompetitors, BanSubstrings, BanTopics, Code, Gibberish, Language, PromptInjection, Regex, Secrets, Sentiment, TokenLimit, Toxicity.

Output scanners:
BanCode, BanCompetitors, BanSubstrings, BanTopics, Bias, Code, Deanonymize, JSON, Language, LanguageSame, MaliciousURLs, NoRefusal, NoRefusalLightResponse, FactualConsistency, Gibberish, Regex, Relevance, Sensitive, Sentiment, Toxicity, URLReachability.

Step 2: Processing and Sanitizing the Prompt

Once the input filters are configured, LLM Guard processes the prompt. In this example, a detailed resume containing PII is passed through the system. The tool identifies and sanitizes the sensitive information, including names, addresses, phone numbers, and employment details, ensuring that the LLM only receives sanitized input.

Step 3: Viewing the Results

After LLM Guard processes the prompt, you can view the sanitized results. In this case, all personal information such as full names, phone numbers, and email addresses have been redacted. The output is clean and complies with privacy standards. Additionally, a detailed breakdown of each filter's performance is provided, indicating whether the input passed or was flagged by each active scanner.

Wrap-UP

LLM Guard offers robust security and content quality control for large language models by acting as a proxy between your application and the LLM. With its extensive range of customizable scanners for both input and output, it provides granular control over what passes through the model, ensuring compliance with privacy, security, and ethical standards.

In addition to its powerful filtering capabilities, LLM Guard integrates seamlessly into existing workflows. It can be deployed as a middleware layer in your AI infrastructure without needing to modify the LLM itself. This proxy-style deployment allows you to enforce security rules and quality checks transparently across various applications using LLMs. Whether you are working with APIs, cloud-native architectures, or on-premise models, LLM Guard can be integrated with minimal friction. It also supports real-time scanning and protection, ensuring your LLMs are secured and monitored continuously.

How to ensure high-quality input and output of your LLM with DeepEval

DeepEval, developed by Confident AI, is an open-source framework that automates the evaluation of LLM responses based on customizable metrics, ensuring high-quality inputs and outputs. It offers various features to measure LLM performance, helping users improve and maintain model reliability across different applications.

Key Features of DeepEval include:

Customizable Metrics: Define specific evaluation metrics, such as relevance, consistency, and correctness, based on your use case.
Automated Test Runs: Automate the evaluation of test cases, providing detailed insights into LLM performance.
Experiments and Hyperparameters: Compare test runs across various hyperparameter settings, allowing for optimal fine-tuning.
Monitoring & Observability: Track LLM performance in real-time, identifying areas for improvement.
Human Feedback Integration: Incorporate human feedback into the evaluation cycle for deeper insights into model behavior.

You can get started with DeepEval by simply downloading it from the GitHub repository. Once installed, you can create custom evaluation metrics, run test cases, and analyze results directly on your system. DeepEval provides flexibility, making it suitable for anyone looking to test LLMs without additional overhead or setup requirements.

While the tool can be used independently, creating an account on Confident AI’s platform offers additional benefits. By registering, you gain access to centralized storage for your test results and the ability to manage multiple experiments in one place. This feature can be particularly useful for teams working on larger projects, where tracking and overseeing various iterations and performance evaluations is critical. Additionally, the platform offers enhanced features like integrated monitoring and real-time evaluations, which can streamline the testing process.

Now, let's dive into how to set up DeepEval, configure a test case, run the test and analyze the output.

Step 1: Install DeepEval

First, make sure DeepEval is installed:

pip install -U deepeval

Step 2: Write a New Health Recommendation Test Case

Let’s create a new test file that evaluates whether the LLM can provide relevant and accurate recommendations for maintaining heart health.

Create a new test file:

touch test_health_recommendations.py

Now, open test_health_recommendations.py and write the following test case:

import pytest
from deepeval import assert_test
from deepeval.metrics import AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase

def test_case():
    # Define the relevancy metric
    answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.5)

    # Define the test case with input, actual output, and retrieval context
    test_case = LLMTestCase(
        input="What are the best practices for maintaining heart health?",
        # Replace this with the actual output generated by your LLM
        actual_output="To maintain heart health, it's important to eat a balanced diet, exercise regularly, and avoid smoking.",
        # Relevant information from a knowledge source
        retrieval_context=[
            "Maintaining heart health includes regular physical activity, a healthy diet, quitting smoking, and managing stress.",
            "A diet rich in fruits, vegetables, whole grains, and lean proteins is recommended for heart health.",
            "Limiting alcohol consumption and regular medical checkups can help monitor heart health."
        ]
    )
    # Run the test and evaluate against the relevancy metric
    assert_test(test_case, [answer_relevancy_metric])

Step 3: Run the Test

Run the test case using the following command:

deepeval test run test_health_recommendations.py

Breakdown of This Test Case:

Input:

"What are the best practices for maintaining heart health?"

This is a common health-related question.

Actual Output:

"To maintain heart health, it's important to eat a balanced diet, exercise regularly, and avoid smoking."

This is the LLM’s response to the query.

Retrieval Context: This is the relevant information that the LLM can refer to when answering the question:

"Maintaining heart health includes regular physical activity, a healthy diet, quitting smoking, and managing stress."

"A diet rich in fruits, vegetables, whole grains, and lean proteins is recommended for heart health."

"Limiting alcohol consumption and regular medical checkups can help monitor heart health."

Answer Relevancy Metric: The AnswerRelevancyMetric is used to evaluate how closely the LLM’s output matches the relevant context from a knowledge source. The threshold of 0.5 means the test passes if the relevancy score is 0.5 or higher.

Wrap-Up

DeepEval is an essential tool for maintaining the quality and reliability of LLMs, particularly in high-stakes industries such as healthcare, where the recommendations and outputs provided by AI systems directly impact people's well-being. By leveraging DeepEval, you can rigorously test and evaluate the performance of your LLMs, ensuring that the model's outputs are accurate, relevant, and free from harmful errors or hallucinations.

One of the key advantages of using DeepEval is its comprehensive set of metrics that allow you to assess various aspects of your model’s performance. Whether you’re monitoring the relevancy of answers to the provided context, the fluency of language used, or detecting potential hallucinations (incorrect or unsupported statements), DeepEval provides out-of-the-box solutions to streamline the testing process. In industries like healthcare, financial services, or legal advice, where strict compliance with factual accuracy and safe recommendations is vital, DeepEvals suite of tools helps minimize risks. For instance, the HallucinationMetric identifies cases where the model introduces information not supported by the retrieval context, which is critical when a model is deployed in sensitive environments like hospitals or clinics. The AnswerRelevancyMetric ensures the response aligns well with the relevant information, eliminating misleading or irrelevant answers that could confuse or harm users.

Using DeepEval is straightforward. After installing the package and configuring your environment, you can write test cases using familiar Python frameworks like pytest. With DeepEval, you define inputs, expected outputs, and relevant knowledge sources (retrieval contexts). These components allow you to track the model's ability to retrieve accurate information, detect hallucinations, or evaluate the overall fluency of its responses. The set of available metrics makes it uniquely suited to such critical use cases. By utilizing these metrics, organizations can confidently deploy LLM-based solutions in environments that require the highest level of reliability, ensuring that the outputs provided do not harm users or provide incorrect recommendations.

Final thoughts

When it comes to ensuring high-quality LLM prompts and outputs, accuracy and safety are crucial. Amazon Bedrock Guardrails provides a robust set of native features for securing, managing, and monitoring LLM outputs within AWS, offering governance and real-time protection to prevent harmful or incorrect outputs. However, when further customization is needed, or if there is no dependency on AWS services, open-source solutions like LLM Guard and DeepEval offer a powerful alternative. These tools enable comprehensive testing, evaluation, and real-time monitoring, ensuring accuracy, relevance, and reliability.

To put this into practice, focus on developing clear strategies for continuously monitoring performance, refining models to meet the demands of specific use cases, and implementing thorough testing processes to catch inaccuracies before deployment. Real-time oversight is key, especially when models are in production, to ensure only reliable outputs are delivered. And don't forget the importance of collaboration—bringing in domain experts alongside AI teams to validate outputs can help keep things on track, especially in areas where precision is critical.

Choosing the right AWS solution: Comparing AWS European Sovereign Cloud against other offerings

Artur Schneider — Fri, 22 Dec 2023 12:04:56 +0000

The recent announcement of the AWS European Sovereign Cloud has been extremely well welcomed by the AWS community, indicating a growing demand for specialized cloud solutions that meet specific regional and regulatory requirements.

While the full details of the newly announced AWS European Sovereign Cloud, are not yet known, this blog post attempts to extract and analyze the best possible assumptions from the information currently available. My aim is to provide a preliminary but insightful comparison of these services, focusing on aspects such as purpose and target audience, compliance and regulatory standards, data sovereignty and location, among others.

For organizations grappling with the complexities of cloud service adoption and integration, it is important to understand the nuances and potential of these AWS offerings. This blog aims to analyze and compare these services along several key dimensions to help businesses, IT professionals and decision makers choose the AWS service that best aligns with their specific needs. Whether it's compliance with strict data regulations, the search for high-performance computing solutions or the need for on-premises cloud infrastructure, this comparison is designed to point you in the direction of the optimal AWS solution for your individual circumstances.

In this section, we will look at the distinct purposes and target audiences of the AWS European Sovereign Cloud, AWS GovCloud, AWS Dedicated Local Zones, and AWS Outposts. Understanding these elements is key to discerning which service aligns best with specific organizational needs and regulatory environments.

AWS European Sovereign Cloud

Purpose: Tailored for the unique demands of data sovereignty and privacy within the European Union, this service ensures that data handling and storage comply with EU regulations.
Target Audience: Primarily suited for EU-based companies, governmental organizations, and any entity that needs to adhere to the strict data protection laws of the EU, such as GDPR.

AWS GovCloud

Purpose: Specifically designed for U.S. government agencies and contractors, AWS GovCloud offers a secure and compliant cloud environment meeting U.S. government regulatory standards.
Target Audience: U.S. government entities, contractors, and businesses handling sensitive data, needing to comply with U.S. specific regulations like FedRAMP and ITAR.

AWS Dedicated Local Zones

Purpose: These zones extend AWS's infrastructure to particular geographic locations, delivering low-latency and high-performance connectivity.
Target Audience: Best for businesses requiring immediate, low-latency access to their applications, including media, entertainment, enterprises with critical real-time operations, and those needing localized data processing.

AWS Outposts

Purpose: Offering AWS services and infrastructure on-premises, AWS Outposts is designed for a consistent hybrid cloud experience.
Target Audience: Ideal for organizations that require a blend of on-premises and cloud environments, especially where low latency or localized data processing is a priority.

Compliance with regulatory standards is a cornerstone in cloud computing, especially for organizations operating under strict data protection and privacy laws. This section provides an in-depth look into the specific compliance frameworks and certifications that are integral to the selected AWS solutions.

AWS European Sovereign Cloud

Compliance Standards: This service is designed to align with the European Union's data protection laws, including the GDPR. It ensures adherence to data residency and sovereignty regulations, which is central to the EU's regulatory framework. A notable aspect of this service is AWS's collaboration with the BSI in Germany. This partnership is a testament to AWS's commitment to meeting the highest standards of data security and regulatory compliance, specifically addressing concerns around data sovereignty and privacy within the EU

AWS GovCloud

Compliance Standards: AWS GovCloud is built to comply with U.S. government standards such as FedRAMP for cloud security and ITAR for defense-related data. It also adheres to the Department of Defense (DoD) Cloud Computing Security Requirements Guide (SRG) for various impact levels, ensuring robust data protection and operational security.

AWS Dedicated Local Zones

Compliance Standards: These zones adhere to AWS's core security and compliance policies, including ISO/IEC certifications and SOC reports. They also comply with regional data protection laws, with specific capabilities varying based on the local zone's geographic location.

AWS Outposts

Compliance Standards: Outposts extend AWS's compliance and security controls into on-premises environments. This includes meeting standards such as HIPAA, PCI DSS, and ISO/IEC certifications, enabling businesses to fulfill local compliance and data residency requirements while using AWS services.

Performance and latency are critical factors in cloud computing, impacting user experience and operational efficiency. This section examines how the AWS European Sovereign Cloud, AWS GovCloud, AWS Dedicated Local Zones, and AWS Outposts are engineered to address these aspects.

AWS European Sovereign Cloud

Performance Characteristics: While the primary focus is on data sovereignty, this service is also designed to deliver high-performance computing capabilities within the EU. It is optimized to minimize latency for EU-based users and applications.
Latency Aspects: Ideal for scenarios where data must remain within the EU without compromising on operational speed and efficiency, ensuring responsive access to cloud resources.

AWS GovCloud

Performance Characteristics: AWS GovCloud offers robust performance for U.S. government agencies and contractors. It is equipped to handle high-demand workloads, ensuring efficient processing of sensitive data.
Latency Aspects: Tailored to provide low-latency interactions for users within U.S. territories, it's suitable for time-sensitive government operations and applications.

AWS Dedicated Local Zones

Performance Characteristics: These zones stand out for their ability to offer ultra-low latency by placing AWS infrastructure closer to end-users. They are specifically designed for applications requiring real-time or near-real-time response rates.
Latency Aspects: Perfect for use cases such as interactive gaming, live video streaming, and other latency-sensitive applications, providing a seamless user experience by reducing delay in data processing.

AWS Outposts

Performance Characteristics: AWS Outposts bring AWS services to on-premises locations, ensuring low latency for applications that require close proximity to data sources or end-users. They are particularly beneficial in environments where internet connectivity is limited or unreliable.
Latency Aspects: Ideal for applications that demand on-site data processing, such as industrial automation and healthcare systems, where every millisecond counts.

The ease of deployment and integration into existing IT infrastructures is a crucial consideration for many organizations. This section discusses the deployment models and integration capabilities of the AWS European Sovereign Cloud, AWS GovCloud, AWS Dedicated Local Zones, and AWS Outposts.

AWS European Sovereign Cloud

Deployment Model: Mirroring the approach of AWS GovCloud, the AWS European Sovereign Cloud will be established as a separate region. This design is intended to cater specifically to the data sovereignty and privacy requirements within the European Union.
Integration Capabilities: While functioning as an independent region, it will maintain compatibility with the broader AWS ecosystem, enabling users to leverage AWS services while adhering to EU-specific regulations. This setup allows for a hybrid environment where EU data residency is strictly maintained, yet the service benefits from the scalability and robustness of AWS's global infrastructure.

AWS GovCloud

Deployment Model: While AWS GovCloud leverages the core technology of AWS, it operates in isolated environments to ensure that data does not mix with non-government data on the public AWS cloud. This separation is crucial for meeting stringent U.S. government compliance and regulatory standards.
Integration Capabilities: Provides specialized services that comply with U.S. federal regulations, enabling secure and seamless integration with sensitive workloads and government-specific applications.

AWS Dedicated Local Zones

Deployment Model: These local zones are designed as extensions of AWS Regions, providing the capability to deploy AWS services locally. They bridge the gap between local data processing needs and cloud scalability.
Integration Capabilities: Enables seamless integration with existing AWS services, offering a hybrid solution that combines the benefits of local processing with the vast array of AWS cloud services.

AWS Outposts

Deployment Model: AWS Outposts uniquely brings AWS infrastructure to on-premises locations, offering a hybrid cloud solution that integrates with existing on-site systems.
Integration Capabilities: It supports a broad range of AWS services and tools, allowing for deep integration with on-premises systems and applications. This is particularly beneficial for environments where local data processing and cloud services need to work in tandem.

Wrapping up: My key insights

After exploring the AWS European Sovereign Cloud, AWS GovCloud, AWS Dedicated Local Zones and AWS Outposts, I have discovered a raft of information that is key for anyone navigating the AWS ecosystem. Below are my insights and key takeaways from this research to help you to make a decision about the AWS solution that best fit your needs:

AWS European Sovereign Cloud:

Tailor-made for compliance with EU data laws and regulations.
A perfect fit for EU-based entities needing adherence to GDPR and similar requirements.
Prioritizes data sovereignty and protection within the EU legal framework.

AWS GovCloud:

Specifically designed for U.S. government agencies and contractors.
Aligns with U.S. federal standards, including FedRAMP and ITAR, for handling sensitive data.
Ensures a secure and compliant environment for U.S. government-specific data.

AWS Dedicated Local Zones:

Brings AWS closer to specific locations for ultra-low latency.
Ideal for real-time or near-real-time applications, from gaming to live streaming.
Merges the benefits of local data processing with cloud scalability.

AWS Outposts:

Delivers AWS infrastructure and services directly to on-premises facilities.
Best suited for scenarios demanding on-site data processing in a hybrid cloud setup.
Consistently upholds AWS’s security and compliance standards on-premises.

Amazon Bedrock vs Amazon SageMaker: Understanding the difference between AWS's AI/ML ecosystem

Artur Schneider — Sun, 06 Aug 2023 13:59:42 +0000

As businesses and organizations continue to leverage the power of artificial intelligence and machine learning, cloud service providers like Amazon Web Services (AWS) are relentlessly innovating and introducing new services to support these modern needs. In this post we'll be pulling back the curtain on two of AWS´s centerpieces AI/ML services: the new kid on the block Amazon Bedrock and the well-established Amazon SageMaker.

We're going to pit these two contenders against each other, in a friendly way of course. We’ll weigh them up, focusing on the heavy hitters: general differences, data protection and security, setup efforts, customizability, and potential use cases.

It's only fair to tell you that Amazon Bedrock is not yet available to the masses. It's currently only presenting itself to a select audience of customers and partners. But don't worry, we've gathered enough information to send it into the ring with SageMaker. As of the 28th of September 2023, Amazon Bedrock has become generally available in the AWS regions of US East (N. Virginia) and US West (Oregon)

General Differences

Amazon Bedrock

Amazon Bedrock is a fully managed service that provides access to pre-trained foundation models from Amazon/AWS and well-known AI startups. Not only does it come with an impressive set of features, it also eliminates the need to manage the underlying infrastructure and fits seamlessly into the AWS service landscape.

Amazon Bedrock features summarized:

Diverse Foundation Models: Amazon Bedrock provides access to a range of pre-trained foundation models, offering a significant multifunction advantage compared to other AI services that usually come with only text foundation models. Some of these models include Amazon Titan for text summarization and generation, Jurassic-2 for multilingual text generation, Claude 2 for thoughtful dialogue and content creation, Command and Embed for text generation for business applications, and Stable Diffusion for image generation.

Agents for Amazon Bedrock: Agents allow developers to build generative AI systems capable of incorporating data and integrating with internal APIs via Foundation Models (FMs), privately and securely, without the need to train the models. The FMs can complete complex tasks, in addition to generating text or chatting.

Serverless Experience: Bedrock is serverless, meaning there are no servers or infrastructure to manage. Users only need to interact with a simple API: provide an input and specify the model to use, and Bedrock will provide an output. It's as simple as that.

Easy Model Customization: Bedrock allows easy model customization through fine-tuning. Customers only need to point Bedrock to a few labeled examples in an S3 bucket, and the service can fine-tune the model for a specific task. You just need to provide a couple of dozen prompt/response examples, and you're good to go.

Data Privacy: None of the user data is used to train the underlying foundational models. All data is encrypted and does not leave the user's Virtual Private Cloud (VPC). This feature is a significant milestone for the real-world, production use of foundational models.

Service Integration: Bedrock integrates seamlessly with other AWS services, including SageMaker Experiments (for testing different models) and Pipelines (for managing Foundation Models at scale).

Amazon SageMaker

On the other hand, Amazon SageMaker is a comprehensive service that allows data scientists and developers to build, train, and deploy machine learning models for extended use cases. SageMaker supports the complete machine learning lifecycle, providing tools to label and prepare your data, choose an algorithm, train the model, tune and optimize it for deployment, make predictions, and take action.

Amazon SageMaker comes with a capability called "JumpStart" that essentially does exactly what it says. It's a jump start that speeds up machine learning projects by providing pre-built solutions and trained models, making it easier for users to get their projects off the ground. This feature is a blessing for developers who want to get started quickly without having to reinvent the wheel.

Amazon SageMaker features summarized:

Comprehensive Machine Learning Lifecycle Support: Amazon SageMaker supports every step of the machine learning process, from data labeling and preparation to model deployment and monitoring.

Built-in Algorithms and Frameworks: Amazon SageMaker provides various built-in algorithms and frameworks, allowing users to select the most suitable one for their needs without requiring them to build everything from scratch.

Automatic Model Tuning: Amazon SageMaker automatically tunes models by adjusting thousands of different combinations of algorithm parameters, leading to the most optimal model.

Training and Inference with SageMaker Studio: The SageMaker Studio provides a single, web-based visual interface where you can perform all ML development steps, making it easier to build, train, and tune machine learning models.

Managed Spot Training: Amazon SageMaker Managed Spot Training allows users to use Amazon EC2 Spot instances for training ML models, resulting in cost savings of up to 90% compared to on-demand instances.

Data Protection and Security Requirements

Both Bedrock and SageMaker offer robust security features inherent in the AWS ecosystem. However, the services differ in terms of data management as their underlying architectures bring significant differences that could potentially impact your use case.

With Amazon SageMaker, customers have complete control over their data and the underlying infrastructure. This means that users have full authority over where the data is processed. They have the ability to encrypt data both at rest and in transit, manage data access through Identity and Access Management (IAM) roles, and comply with regulations through AWS’s robust compliance offerings. Users can also use SageMaker in their Virtual Private Cloud (VPC) to have network level control. For customers with stringent data security requirements, this level of control is paramount.

On the other hand, Amazon Bedrock, being a managed service, processes data within the confines of the AWS environment. While this means users won't have to worry about infrastructure management, it also means that they have less direct control over where the data is processed. Nonetheless, Bedrock ensures that none of the user data is used to train the underlying foundational models and all data is encrypted and does not leave the user's VPC. However, organizations with extremely high-security requirements might need to evaluate these considerations carefully.

While both services provide robust security features, the decision will likely come down to how much direct control over data and infrastructure your organization requires.

Effort to Setup

Bedrock's setup, being fully managed, is expected to require less effort compared to SageMaker. Users simply select an appropriate pre-trained foundation model, customize it with their data, and start using it.

Conversely, despite its powered experience, SageMaker requires more setup effort due to its large feature set. Users have to prepare data, select or create an algorithm, train the model, and then deploy it. In addition, using SageMaker effectively requires more technical experience and additional infrastructure management compared to Bedrock.

Customizability

SageMaker excels in customizability as it allows you to use your own algorithms, built-in ones, or select from many available on the AWS Marketplace or third party. You can fine-tune these models to meet specific requirements. This means that requirements delving deeper into detail are better suited with SageMaker, as you can use any available open-source Large Language Model (LLM) and train it to your preference.

In contrast, Bedrock's customizability appears less flexible than SageMaker. While it allows users to customize foundation models with their data, Bedrock only comes with default Foundation Language Models, although with fine-tuning capabilities.

Use Cases

Bedrock is ideal for scenarios where organizations need advanced AI capabilities quickly without having to deal with building models or managing infrastructure. Use cases include creative content generation, dialog system creation, text summarization, multilingual text creation, and advanced image generation tasks.

SageMaker is suitable for a broader range of machine learning tasks that require detailed control over the process of model creation, training and deployment. It is ideal for predictive analytics, recommendation systems, anomaly detection, or any other task that requires a customized machine learning model.

Costs

When it comes to cost, Amazon Bedrock could potentially have an advantage over Amazon SageMaker. Since Bedrock is a fully managed service, it abstracts away much of the infrastructure management, potentially reducing operational costs. Additionally, given that it's serverless, you only pay for the resources you actually use. This contrasts with SageMaker, where you may need to maintain dedicated resources even when not in use.

Specifically, Amazon Bedrock's pricing model is based on usage, with charges for the number of tokens processed by the foundational models and the compute time used for fine-tuning. This means that the cost directly scales with your needs.

Conclusion

Both Amazon Bedrock and Amazon SageMaker are powerful tools in the AWS AI/ML service landscape. Choosing between the two depends on your specific needs. If you need to quickly integrate advanced AI capabilities without much customization, Bedrock is the way to go. But if your use case demands deep customizability and you're willing to invest more effort into setting up and managing the model training process, SageMaker would be a better fit.

Spicing Up AWS Architecture Diagrams: A Step-by-Step Guide To Creating Animated AWS Architecture GIFs

Artur Schneider — Fri, 30 Jun 2023 14:53:16 +0000

Introduction

When it comes to visualizing the design and structure of cloud infrastructures, AWS architecture diagrams are an indispensable tool. They provide clear, concise visual representations that can communicate complex system designs with ease. But what if we could take these static diagrams a step further? What if we could make them more engaging and interactive?

When I stumbled upon a LinkedIn post by Ankit Jodhani, I was immediately captivated by the animated architecture diagrams he presented. Like something out of a sci-fi movie, these diagrams twinkled, flashed, and undulated, drawing the viewer into the world of the architecture in a way static diagrams simply couldn't. The enigma surrounding this animation magic sparked a question: what advanced, perhaps extraterrestrial technology was behind this?

To my surprise, the solution was not found in the outer reaches of the cosmos, but rather, nestled within a software we've known and used for years - PowerPoint. Yes, the same PowerPoint we've used to make countless school presentations and office reports was the secret behind these lively diagrams.

With clever manipulation of PowerPoint's animation features, Ankit breathed a dynamic life into the typically static AWS architecture diagrams, giving them a whole new level of interactivity and engagement. His work truly illustrates that creativity is not about the tools we use, but how we use them.

The allure of this novel approach to presenting architecture diagrams was simply too much to resist. Therefore, I've decided to dissect and demystify this process, providing you with a comprehensive step-by-step guide to animate your own AWS architecture diagrams. By the end of this journey, not only will you have a deeper understanding of PowerPoint's capabilities, but you'll also have the skills to create eye-catching AWS architecture diagrams that are sure to impress your clients and colleagues.

Join me as we venture into the uncharted territory of animated AWS architecture diagrams, and let's bring your AWS architecture diagrams to life!

Requirements

Before we dive into the step-by-step process of creating lively and engaging AWS architecture diagrams, let's make sure we have all the necessary tools at our disposal. These prerequisites will not only equip you to follow along with this guide, but will also set the foundation for your future endeavors in creating dynamic presentations.

Here are the key requirements:

Microsoft PowerPoint: To make your AWS architecture diagrams come to life, the first thing you need is Microsoft PowerPoint. This guide is designed with the latest version of PowerPoint from Office 365 in mind, ensuring you're utilizing all the modern features and functionalities.
Latest AWS PowerPoint Toolkit: To build accurate and visually pleasing AWS diagrams, having a toolkit is invaluable. It offers you a comprehensive set of AWS-specific icons and resources. You can download it from the official AWS website here.
(Optional) AWS Icons Finder: While not mandatory, this tool can vastly improve your efficiency. This is a simple web app that allows you to quickly search and copy AWS icons for your diagrams. Visit James Kim's AWS Icons Finder to try it out.

Creating Your Animated AWS Architecture Diagrams

In this section, we will walk through the process of creating a dynamic, animated AWS architecture diagram. This process involves a sequence of detailed steps, from choosing your architectural layout to applying the finishing touches with animation and effects. Let's dive right in!

Selecting Your AWS Architecture: To start, ensure you have a suitable AWS Architecture on hand. For the purpose of this guide, I will be using the "Git to S3 Webhooks" example from the AWS Architecture Deck.

Choosing Your Shape: Now, select the shape that you want to animate around your architecture. For this example, I will use a simple circle.

Customizing Your Shape: This is where you can get creative! You can customize your chosen shape with various colors and shape effects like a glow for that extra bling.

Adding Animation: Move your shape to its starting position, then navigate to the animation tab. Here, select how your animation should appear. In this case, I prefer using the "Fade" effect.

Positioning Your Shapes: Copy and paste your shape to each position where you want it to move.

Fine-Tuning Your Animation: I recommend opening the Animation pane and adding an exit animation to your shapes. This will ensure they disappear after the movement is over. For consistency, I also choose "Fade" for the exit animation. Ensure you select the red "Fade" option as this represents the exit, whereas the green "Fade" is the entrance animation. Then, organize the order of your animations in the Animation pane.

Adding Movement to Your Animation: Click on each dot and select "Add animation". Then, under Motion Paths, choose the "Custom Path" option. This allows you to create a personalized animation path for your shapes. Alternatively, you can choose from a range of pre-determined paths such as straight lines or loops, based on your preference. Make sure these paths are in the correct order in the Animation pane.

Setting Your Animation Sequence: For animations that should start simultaneously, select them all and choose "Start With Previous". Also, organize the start of the animations in the following order: first the green (entrance animation), then blue (movement animation), and finally red (exit animation).

Reviewing and Exporting Your Animation: With all the animations set, play through it to ensure everything looks as expected. If you need to, adjust the order of the animations. Once you're satisfied with your animation, export the slide as a GIF. The time it takes to create your GIF will depend on the quality you choose.

With these steps, you're fully equipped to create your own captivating AWS architecture diagrams. If you encounter any challenges, don't worry - I can also provide an example PowerPoint as a reference to aid in your creative process, just send me an DM via LinkedIn. Now it's time to turn your static diagrams into dynamic, engaging presentations that are sure to wow your audience! Big Thanks again to Ankit Jodhani for the inspiration!

PrivateGPT and AWS EC2: A beginner's Guide to AI experimentation

Artur Schneider — Thu, 22 Jun 2023 15:10:00 +0000

Introduction

In this era of digital transformation, it's hard to miss the wave of artificial intelligence and machine learning that is sweeping across all sectors. As an enthusiast with a curious mind, but with a limited background in AI, I, like many others, was intrigued yet overwhelmed. This fascination led me down the path of exploring AI, specifically the world of large language models (LLMs).

The world of AI may seem daunting, filled with a myriad of complex terms, algorithms, and architectures. However, my goal was to simplify this journey for myself and, in doing so, create an opportunity for others who wish to venture into this domain.

In this quest for simplicity, I stumbled upon PrivateGPT, an easy-to-implement solution that allows individuals to host a large language models on their local machines. Its powerful functionalities and ease of use make it an ideal starting point for anyone looking to experiment with AI. What's even more interesting is that it provides the option to use your own datasets, opening up avenues for unique, personalized AI applications - all of this without the need for a constant internet connection.

PrivateGPT comes with a default language model named 'gpt4all-j-v1.3-groovy'. However, it does not limit the user to this single model. Users have the opportunity to experiment with various other open-source LLMs available on HuggingFace. One such model is Falcon 40B, the best performing open-source LLM currently available.

In this blog post, I'll guide you through the process of setting up PrivateGPT on an AWS EC2 instance and using your own documents as sources for conversations with the LLM. To make the interaction even more convenient, we will be using a solution that provides an intuitive user interface on top of PrivateGPT.

So, fasten your seatbelts and get ready for a journey into the exciting realm of AI, as we explore and experiment with large language models, all in the comfort of your own private environment.

Configuration

Launching the EC2 Instance

In this section, we will walk through the process of setting up an AWS EC2 instance tailored for running a PrivateGPT instance. We'll take it step by step. This will lay the groundwork for us to experiment with our language models and to use our own data sources.

Let's start by setting up the AWS EC2 instance:

Choosing an Operating System: In our case, we will be using Amazon Linux as our operating system. This is an excellent choice for hosting PrivateGPT due to its seamless integration with AWS services and robust security features. However, PrivateGPT is flexible and can also be hosted on other operating systems such as Windows or Mac.

Selecting Instance Type: For the needs of our task, we require an instance with a minimum of 16 GB memory. The specific instance type that you choose may depend on other factors like cost and performance. I recommend using one of the T3 instances, such as t3.large or t3.xlarge.

Storage Configuration: After choosing the instance type, we need to add additional storage for the language model and our data. The exact amount of storage you need will depend on the size of the model and your dataset. For instance, the Falcon 40B model requires higher amount of storage itself.

Instance Details: Proceed with the default instance details for the initial setup. These can be modified later based on specific requirements.

Security Group Configuration: To ensure we can access the instance from our client, it is essential to configure the security group appropriately. Add a new rule to the security group that allows inbound traffic for the ports 80 and 3000 from your client IP address. This will enable you to securely access the instance over the internet.

Remember that this setup is primarily for experimental purposes. Whitelisting IP addresses is one way to secure your instance from unwanted public access. However, for more complex or sensitive deployments, you may need a more robust security architecture.

At this point, you've successfully set up your AWS EC2 instance, creating a solid foundation for running PrivateGPT. Lets continue with the setup of PrivateGPT...

Setting up PrivateGPT

Now that we have our AWS EC2 instance up and running, it's time to move to the next step: installing and configuring PrivateGPT. The following sections will guide you through the process, from connecting to your instance to getting your PrivateGPT up and running.

Connecting to the EC2 Instance

Connection Setup: To start with, we need to connect to the EC2 instance. In this case, we will be using AWS Session Manager. Session Manager provides a secure and convenient way to interact with your instances. It allows you to connect to your instance without needing to open inbound ports, manage SSH keys, or use bastion hosts. If you're new to Session Manager, you can refer to the official AWS documentation to set up the required configuration. Alternatively, you could also connect to your instance via SSH if you prefer.

Installing Prerequisites

Once connected, we need to install a few prerequisites:

Git: Start by installing Git, which will allow us to clone the PrivateGPT repository. If you're using Amazon Linux, you can install Git by running the command:

sudo yum install git

Pip: Pip is a package installer for Python, which we will need to install the Python packages required by PrivateGPT. You can install it by running:

sudo yum install pip

NPM: NPM (Node Package Manager) is used to install Node.js packages. If it's not already installed, you can install it by first installing Node.js with the command:

sudo yum install npm

Configuration of PrivateGPT

With the prerequisites installed, we're now ready to set up PrivateGPT:

Cloning the Repository: Clone the PrivateGPT repository to your instance using Git. You can do this with the command:

git clone https://github.com/SamurAIGPT/privateGPT.git

First we start our environment, to do this we have to navigate to the client folder and execute the following commands:

npm install
npm run dev

Then we start a Flask application and install the necessary packages. For this we have to change to the server folder and execute the following commands:

pip install -r requirements.txt
python privateGPT.py

By following these steps, you should have a fully operational PrivateGPT instance running on your AWS EC2 instance. Now, you can start experimenting with large language models and using your own data sources for generating text!

Navigating the PrivateGPT User Interface

Now that we've successfully set up the PrivateGPT on our AWS EC2 instance, it's time to familiarize ourselves with its user-friendly interface. The UI is an intuitive tool, making it incredibly easy for you to interact with your language model, upload documents, manage your models, and generate text.

First and foremost, you need to access the PrivateGPT UI. You can accomplish this by typing the public IP address of your AWS EC2 instance into your web browser's address bar from your client, appending :3000 at the end. It should look something like this:

http://your-public-ip:3000

This will navigate you directly to the PrivateGPT interface hosted on your EC2 instance.

Once you've entered the UI, the next step is to download a Large Language Model (LLM). You'll find a button in the UI specifically for this purpose. Clicking this button will commence the download process for the default language model 'gpt4all-j-v1.3-groovy'. Remember, PrivateGPT comes with a default language model, but you also have the freedom to experiment with others, like Falcon 40B from HuggingFace.

With the language model ready, you're now prepared to upload your documents. Select the documents you'd like to use as a source for your LLM. After your documents have been successfully uploaded, the data needs to be ingested by the system. Look for an 'Ingest Data' button within the UI and click it. This action enables the model to consume and process the data from your documents.

Finally, with all the preparations complete, you're all set to start a conversation with your AI. Use the conversation input box to communicate with the model, and it will respond based on the knowledge it has gained from the ingested documents and its underlying model. (In my example I have generated PDF files from the official AWS documentations)

And voila! You've now set foot in the fascinating world of AI-powered text generation. You can continue to explore and experiment with different settings and models to refine your understanding and outcomes. Enjoy this exciting journey!

Summary

As we wind up this exploration, it's worth highlighting that while PrivateGPT may not offer the exact capabilities of something like ChatGPT, it provides a robust and secure environment to experiment with large language models, leveraging your own sources of data. From PDFs, HTML files, to Word documents and beyond, PrivateGPT offers flexibility in the types of documents you can use as data sources. (For the full list of supported document types, refer to the official PrivateGPT GitHub repository at https://github.com/imartinez/privateGPT.)

What makes PrivateGPT all the more compelling is the constant evolution of open-source large language models. As these models continue to improve, the gap between services like ChatGPT is rapidly closing. The added advantage is that you're in control of your own data and infrastructure, providing a level of trust and flexibility that is invaluable in the rapidly evolving AI landscape.

Undoubtedly, the journey into the realm of AI and large language models doesn't end here. With services like AWS SageMaker and open-source models from HuggingFace, the possibilities for experimentation and development are extensive. For those interested in more reliable solutions, I highly recommend checking out the insightful blog post by Phillip Schmid on using AWS SageMaker with large language models in a AWS environment. You can find his blog post here:

In conclusion, whether you're a seasoned AI practitioner or a curious enthusiast, tools like PrivateGPT offer an exciting playground to dive deeper into the world of AI. So go ahead, set up your PrivateGPT instance, play around with your data and models, and experience the incredible power of AI at your fingertips. Remember, "es lohnt sich" - it's worth it!

Shout out to the creators of PrivateGPT Ivan Martinez and group around SamurAIGPT which give us a great start into the AI world through this simplification.

Unlocking Cost Savings and Sustainability with AWS S3 and Synology Hyper Backup

Artur Schneider — Mon, 05 Jun 2023 14:31:10 +0000

The importance of safeguarding digital data cannot be overstated, and as a committed user of Synology NAS, I understood this imperative well. While my NAS offered centralized storage and easy accessibility, it also underscored the need for a reliable, cost-efficient, and sustainable disaster recovery (DR) solution. For a long time, Dropbox was my go-to for this. However, as my data needs evolved, Dropbox's fixed pricing structure began to feel increasingly ill-fitted and unsustainable economically.

That's when I decided to explore the potential of AWS S3. Renowned for its scalability, security, and performance, AWS S3 also offered a flexible "pay-as-you-use" pricing model that seemed like a more sustainable and financially sensible choice.

Setting Up AWS S3 with Synology Hyper Backup

Setting up AWS S3 with my Synology NAS was made incredibly straightforward through the Hyper Backup application. The transition involved:

Creating an AWS account and setting up an S3 bucket with proper access permissions.
Launching the Hyper Backup application on the Synology DSM.
Selecting AWS S3 as my backup destination.
Filling in the necessary AWS S3 bucket details, including Access Key ID, Secret Access Key, and the bucket name.
Configuring the backup settings, including selecting local folders to back up, setting automatic backups, and managing versions of my backups.
Launching the backup process to begin syncing my Synology NAS data with my AWS S3 bucket.

Economic and Environmental Impact

Dropbox's pricing model offered a static storage capacity, regardless of the actual usage. In my case, I didn't need the full size of the provided storage, so a sizable portion of the storage I was paying for went unused. This situation was not only financially inefficient but also unsustainable as it led to wasteful consumption of digital storage resources.

Conversely, AWS S3's pay-as-you-go model allows users to pay only for the exact amount of storage used. This flexible model eliminated unnecessary waste, leading to a more sustainable and financially efficient solution. I found myself making savings of nearly 50% compared to my Dropbox subscription, as I was now only paying for the storage I was actively using.

Moreover, AWS offers cost management tools to monitor spending and set up usage alerts. This ensured that I maintained control over my budget and further reduced waste, aligning with sustainable digital practices.

Final Thoughts

Switching to AWS S3 for my Synology Hyper Backup solution has proven to be a sustainable, economical, and custom-fit solution for my DR needs. The flexible pricing model of AWS S3 has allowed me to eliminate waste, both in terms of storage and cost, while retaining a secure and reliable backup of my data.

By moving from Dropbox to AWS S3, I not only found a cost-effective DR solution but also one that better aligns with principles of sustainable consumption and efficiency. If you're looking for an affordable, secure, adaptable, and sustainable DR solution for your Synology NAS, AWS S3 is a fantastic option to consider.

Migrating large amounts of data to Amazon FSx for Windows File Server with AWS DataSync

Artur Schneider — Mon, 29 May 2023 10:51:53 +0000

Introduction

In the evolving landscape of technology and business, companies often find themselves navigating through a myriad of choices. One such choice is the decision between maintaining on-premises infrastructure or shifting to cloud-based solutions. This dilemma was recently faced by one of our clients - a global container shipment company.

Their challenge was an aging fleet of local Windows file servers. As the hardware aged, issues of inefficiency, decreased performance, and increased maintenance costs became prevalent. Faced with the significant investment of time and resources required to refresh their local servers, they began to explore alternative options.

Their search led them to AWS, and specifically, Amazon FSx for Windows File Server. Amazon FSx for Windows File Server offers fully managed Microsoft Windows file servers, providing the compatibility and features that their organization needed, all with the flexibility, scalability, and cost benefits of a cloud-based solution. It was an obvious choice, not only from a technological perspective but also for a smoother transition in terms of operating environment.

The only obstacle that remained was the formidable task of migrating vast amounts of data from their local servers to the cloud. The complexity and potential risks of such a migration could not be underestimated. Yet, AWS once again provided the solution - AWS DataSync.

AWS DataSync is a data transfer service that simplifies, automates, and accelerates copying large amounts of data to and from AWS storage services over the internet or AWS Direct Connect. Leveraging DataSync, we aimed to streamline the file migration process from one single dashboard, ensuring a seamless and efficient transition.

In this blog post, we'll delve into our experience of this migration. I'll guide you through the process we followed, the challenges we faced, the solutions we devised, and most importantly, the success we achieved. Strap in for a deep dive into our journey from local Windows servers to Amazon FSx, orchestrated by AWS DataSync.

Requirements

Before we dive into the migration, it's essential to understand the necessary prerequisites for using AWS DataSync and Amazon FSx for Windows.

AWS DataSync requirements

Let's take a closer look at the DataSync requirements.

Network Connectivity: AWS DataSync requires network connectivity between your source servers and the AWS region where your destination is located. This can be achieved either via the internet or using AWS Direct Connect / VPN for a dedicated network connection like in our case through VPC Endpoints. It is worth it to have a closer look at the official AWS documentation AWS DataSync network requirements

DataSync Agent: The DataSync agent must be installed on a virtual machine or physical server with at least 4 vCPUs, 16 GB of RAM, and network connectivity to both your source storage and AWS. You can deploy your agent on a VMware ESXi, Linux Kernel-based Virtual Machine (KVM), or Microsoft Hyper-V hypervisor. For storage in a virtual private cloud (VPC) in AWS, you can deploy an agent even on an Amazon EC2 instance.

Outbound Network Access: The DataSync agent requires outbound network access over port 80 for HTTP or port 443 for HTTPS. These ports must be open on your firewall.

File System Compatibility: If you are transferring data from a Windows-based file system, it should support SMB (Server Message Block) protocol. For Unix or Linux source data, your file system must support NFS (Network File System).

AWS Permissions: In order to transfer data, the DataSync service needs the necessary permissions in AWS Identity and Access Management (IAM). This requires an IAM role that DataSync can assume to access resources.

Amazon for Windows File Server requirements

Now let's look at the requirements specific to using Amazon FSx for Windows File Server.

VPC and Security Group Configuration: Amazon FSx requires an Amazon VPC with the necessary security group rules. Your security groups must allow inbound traffic over the SMB port (usually port 445) from your clients, and the clients must be able to route traffic to the Amazon FSx file system.

Active Directory Integration: Amazon FSx must be joined to an AWS Managed Microsoft AD or self-managed AD. Your Windows users and groups must be part of this Active Directory.

Minimum Storage Capacity: The minimum storage capacity for Amazon FSx is 32 GiB.

Backup and Maintenance Preferences: You need to choose your preferred daily backup window, weekly maintenance window, and whether to enable automatic backup.

Network Connectivity: The FSx file server must have network connectivity to your workloads either within AWS (for workloads running on EC2 instances) or on-premises (for workloads running on your local servers).

Setting up Amazon FSx for Windows File Server

Before you can proceed with setting up AWS DataSync, your destination - in this case, Amazon FSx for Windows File Server - needs to be correctly set up and ready to receive data.

Step 1: Prepare for Amazon FSx

Before creating your Amazon FSx for Windows File Server, make sure you meet the following prerequisites:

Have a Active Directory ready to join your FSx and reachable from your AWS environment. Alternatively set up your AWS Managed Microsoft AD or self-managed Active Directory, which FSx will need to join.
Ensure you have a VPC and Security Group ready to be associated with FSx.

Step 2: Create Your Amazon FSx for Windows File System

Navigate to the Amazon FSx console and select "Create file system".
Choose the deployment type as "FSx for Windows File Server" and select "Next".

Step 3: Specify File System Details

Name your file system for easy identification and proceed to configure it.
Choose your desired storage capacity and performance level.
Configure the network & security settings by selecting the desired VPC, the preferred subnet, and the security groups.
In the Windows authentication section, select the Microsoft Active Directory that you've set up. The Amazon FSx for Windows File Server will be joined to this AD.

Step 4: Configure Optional Settings

You can choose to leave these settings at their default or customize them according to your needs:
Maintenance preferences: You can specify a weekly time window when automatic maintenance activities occur.
Data deduplication: This can reduce your storage costs if you have redundant data.
Encryption: FSx data is encrypted at rest using keys you manage through AWS Key Management Service (KMS). You can use the default key, or choose a key you created.
Throughput capacity: You can manually specify a throughput capacity, or you can allow Amazon FSx to automatically adjust it based on the workload.

Step 5: Review and Create

Review your settings to make sure everything is correct.
Finally, click "Create file system".

Your Amazon FSx for Windows File Server should now be set up and ready to serve as a destination for your data transfer task in AWS DataSync. Note the DNS name and Windows Remote Management (WinRM) port of your newly created FSx for Windows file system - you'll need these when configuring your data transfer task in AWS DataSync.

Now that your Amazon FSx is all set up, you can proceed with defining your target in AWS DataSync, and start transferring your data!

Setting up AWS DataSync

Step 1: Setting up VPC Endpoints

Before setting up AWS DataSync, ensure you've created a VPC endpoint to communicate with AWS DataSync securely within your Amazon VPC.

Navigate to the VPC Dashboard in your AWS Management Console.
Under the "Virtual Private Cloud" section, click on "Endpoints".
Click "Create Endpoint", and in the "Service category" choose "AWS services".
For the service name, select "com.amazonaws.[your-region].datasync" where [your-region] should be replaced with your AWS region (like us-east-1, us-west-2, etc.).
In the VPC section, choose the VPC where you want to create the endpoint.
Select the appropriate Route Table, Security Group, and other options as per your requirements and click "Create endpoint".

Step 2: Deploying the DataSync Agent

Visit the AWS DataSync console in your AWS account and select "Get started".
Choose "Deploy a new agent", and follow the instructions to download and deploy the agent in your local environment.
You can connect to the DataSync via console from you Hypervisor to e.g. test the connection the the VPC endpoint or to retrieve the activation key

Step 3: Configuring the DataSync Agent

After deploying the agent, you'll need to configure it to connect to your on-premises file system and your AWS account.

In the AWS DataSync console, select your newly deployed agent.
Click "Configure agent", and then enter the IP address or hostname of your on-premises file system.
Enter your AWS account ID, choose the AWS region where you want to transfer data, and input the VPC endpoint ID that you created in Step 1.
Choose a method to authenticate the agent to your AWS account (either by creating an IAM role or by entering your AWS access keys).

Step 4: Create a Location for your Data

In the DataSync console, choose "Create location".
Select the location type based on where your data is stored. For on-premises locations, choose "NFS" or "SMB".
Depending on the location type, provide the required information which includes the Agent, server hostname, and mount path for NFS or Agent, domain, user name, password, server hostname, and share name for SMB.
Similarly, create a location for your destination data in FSx.

Step 5: Create a Task to Transfer Data

After setting up locations, you can now create a data transfer task.
In the DataSync console, choose "Create task".
Select the source location and destination location that you created in Step 4.
Configure your data transfer settings like options for handling metadata, data verification, etc., and set a schedule if you want the task to run on a schedule.
Finally, start the data transfer task.

You should monitor your data transfer tasks on the AWS DataSync console. This is where you can view the progress of the task, check for any errors, and view performance metrics.Please remember that this is a simplified overview of the process and does not include every possible configuration option. Always refer to the official AWS documentation for the most accurate and up-to-date information.

Challenges

During our migration we experienced some challenges which I can summarize to the following:

Network Configuration Challenges: Establishing secure and efficient network connections between your local environment and AWS was one of the major challenges. The complexity of existing VPN connection and VPC endpoints, while ensuring that all network requirements are met and firewalls are appropriately configured, can be quite daunting.

Insufficient Local Compute Resources: Another issue was the need for a certain level of compute resources in your local environment to host the AWS DataSync Agent. If the minimum requirements were not met, although you were allowed to proceed with the migration, there could be performance issues. In such cases, using traditional methods like robocopy was an alternative.

IP Address Allocation: AWS DataSync creates an Elastic Network Interface (ENI) in your chosen VPC for each task, each with its own IP address. It was crucial to ensure sufficient IP addresses were available in your subnets to accommodate these ENIs, preventing possible IP address exhaustion.

Permissions in Active Directory Group: With Amazon FSx joining your existing domain, you needed to carefully select the AD group that would have administrative permissions. This decision is critical as it can't be modified after the FSx file system was created. Therefore, careful consideration was required when choosing the AD groups that would manage FSx in the future.

Conclusion

Reflecting on our migration journey, we've come to recognize several crucial takeaways and successes.

AWS DataSync proved to be an exceptional tool for orchestrating large-scale data migrations from multiple on-premises locations to AWS. The ability to consolidate all migration tasks in a single dashboard, regardless of the geographical location, offers an unmatched level of organization and visibility. This feature, combined with comprehensive logging and monitoring capabilities, allowed us to maintain a tight control on our migration process.

Choosing Amazon FSx for Windows File Server was indeed a strategic decision that aligned with our client's requirements and expectations. FSx provided a seamless transition from their familiar local Windows File Servers to the cloud. This ensured minimal disruption to their business operations, and allowed them to continue using file shares just as they were accustomed to, but with the added advantages of AWS' scalability, reliability, and robust security.

Our most significant achievement was successfully migrating vast amounts of data across the globe to AWS with near-zero downtime. This feat, executed effectively with a keen eye on minimizing business impact, marks a pivotal moment in our client's digital transformation journey.

Moving forward, we are confident in our ability to leverage AWS DataSync and Amazon FSx for Windows File Server to assist other businesses in their migration endeavors. As with any ambitious project, challenges are inevitable. Still, we have shown that through careful planning, robust technology, and a strong understanding of our infrastructure, even the most daunting hurdles can be overcome. Our experience stands testament to the power of cloud technology in transforming business landscapes, and we're excited to continue helping organizations navigate their unique paths to the cloud.