DEV Community: Amit Kayal

Google Open Knowledge Format: Why Enterprise Agents Need a Knowledge Layer, Not Just More Tools

Amit Kayal — Thu, 18 Jun 2026 06:41:06 +0000

Google Open Knowledge Format: Why Enterprise Agents Need a Knowledge Layer, Not Just More Tools

Most enterprise AI conversations still start in the wrong place.

They start with the model.

Which model should we use? Which framework should we adopt? Which vendor has the best agent platform? Which tools should we connect next?

These are fair questions. But in real enterprise architecture, they are not the hardest questions.

The harder question is this:

Can our AI systems actually understand how our business works?

That is why Google Cloud’s article on Open Knowledge Format caught my attention. The article talks about a simple but important idea: representing knowledge in a way that humans can read and machines can use. In OKF, that means markdown for the content and structured metadata for context.

At first glance, that may sound too simple.

But that simplicity is the point.

Enterprises do not need another place where knowledge goes to die. We already have enough portals, catalogs, wikis, dashboards, folders, and internal tools. What we need is a practical way to package knowledge so it can be reviewed, versioned, governed, searched, and reused by both people and AI agents.

That is where this idea becomes very relevant for agentic AI.

The Real Enterprise AI Problem

Most organizations already have the knowledge their AI agents need.

They have it in databases, dashboards, tickets, architecture notes, runbooks, Confluence pages, data catalogs, code comments, incident reports, old project documents, and the heads of experienced employees.

The issue is not that knowledge does not exist.

The issue is that it is fragmented.

Some of it is outdated. Some of it is duplicated. Some of it is tribal. Some of it is locked inside tools. Some of it is written for humans but not structured enough for AI systems to use reliably.

This becomes a serious problem when we move from AI assistants to AI agents.

An assistant can give a helpful answer. An agent does more. It plans, selects tools, queries systems, executes steps, generates outputs, and sometimes triggers workflows.

That means the cost of wrong context is much higher.

A data agent may know how to generate SQL. But does it know which table is the source of truth?

A finance agent may calculate revenue. But does it know whether the business means booked revenue, invoiced revenue, recognized revenue, or collected cash?

A support agent may summarize a customer case. But does it know what customer information must be masked before anything is shared externally?

A delivery agent may review project status. But does it understand governance rules, escalation paths, release gates, and dependency risks?

A cloud cost agent may recommend savings. But does it know which environments are production-critical and which ones are safe to shut down?

Without this context, agents do not become enterprise-ready. They become fast, confident, and risky.

More Tools Will Not Solve This

One common mistake in agentic AI is assuming that more tool access means better capability.

Connect the database.
Connect the CRM.
Connect the ticketing system.
Connect the cloud APIs.
Connect the document repository.
Connect the workflow engine.

This improves reach, but not necessarily judgment.

An agent with many tools and weak context can still choose the wrong source, apply the wrong rule, query the wrong table, expose the wrong field, or automate the wrong step.

That is why I believe every serious enterprise agentic AI framework needs three layers:

A reasoning layer
A tool/action layer
A governed knowledge layer

Most teams are investing heavily in the first two. They are testing models, orchestration frameworks, prompts, tools, APIs, and agent workflows.

That work is needed.

But the third layer is where enterprise differentiation will come from.

The model can be changed.

The tools can be integrated.

But the organization’s internal knowledge — its definitions, operating rules, business logic, exceptions, ownership, architecture, and lessons learned — is unique.

That is the real asset.

What Google Open Knowledge Format Gets Right

What I like about the Open Knowledge Format idea is that it does not overcomplicate the problem.

It treats knowledge as something that should be readable, portable, structured, and maintainable.

Markdown makes it easy for humans to read and contribute. Structured metadata makes it easier for systems and agents to classify, retrieve, and use the knowledge. Version control makes review and audit possible.

This matters because traditional documentation is passive.

Someone writes it. Someone may read it. Eventually, it becomes stale.

Agentic AI needs active knowledge.

The knowledge has to be available at runtime. It should help the agent decide what to do, what not to do, which source to trust, which rule to apply, and when to escalate.

A database schema may say that a table has a column called segment.

That is useful, but not enough.

The agent also needs to know:

What does segment mean?
Who owns the definition?
Which values are valid?
Is the field reliable?
Can it be used for reporting?
Can it be exposed externally?
Are there legacy exceptions?
Which workflows are allowed to use it?

This is the gap between data access and enterprise intelligence.

How This Fits into an Agentic AI Framework

In our agentic AI framework, I would treat an OKF-like structure as the Enterprise Knowledge Layer.

This layer should sit between enterprise systems and agent execution.

The agent should not jump directly from a user request to a tool call. That is where many mistakes happen.

A better flow is:

User request
   ↓
Agent identifies intent and domain
   ↓
Agent retrieves relevant knowledge
   ↓
Agent checks source of truth, ownership, caveats, access rules, and usage guidance
   ↓
Agent plans the action
   ↓
Agent calls the right tool
   ↓
Agent produces the answer or executes the workflow
   ↓
Agent proposes a knowledge update if reusable learning is found

This changes the quality of execution.

Take a simple question:

“Show me revenue by customer segment.”

A weak agent will search for tables with names like revenue, customer, and segment, then generate SQL.

A stronger enterprise agent will first check the knowledge layer:

Which revenue table is approved?
Which revenue definition applies?
Which customer segment field is trusted?
Which join is valid?
Which date logic should be used?
Are there caveats for legacy accounts?
Is the user allowed to see this output?

Only after that should it query the database.

That is the difference between automation and governed intelligence.

Generating Open Knowledge Format from AWS SQL Databases

For AWS SQL environments such as Amazon RDS, Aurora, and Redshift, the starting point is metadata extraction.

We can automatically extract:

Database names
Schema names
Table names
Column names
Data types
Primary keys
Foreign keys
Indexes
Nullability
Row counts
Table comments
Column comments
AWS tags
Freshness indicators
Query usage patterns

AWS Glue Crawlers and the Glue Data Catalog can help discover and centralize metadata. Database-native sources like information_schema can provide table and column-level structure.

But metadata is not knowledge.

A pipeline can discover that a table has a column called revenue_amount. It cannot automatically know whether that means booked revenue, recognized revenue, invoiced revenue, or pipeline value. That meaning has to come from finance, sales operations, data owners, or approved documentation.

So OKF generation should be semi-automated.

Technical metadata should be generated automatically. Business meaning should be reviewed and approved by the right domain owners.

For every critical SQL table, the knowledge file should capture:

Business purpose
Source system
Source-of-truth status
Data owner
Data steward
Refresh frequency
Key columns
Approved joins
Common usage patterns
Prohibited usage
Data quality rules
Sensitivity classification
Agent usage guidance
Known caveats

A table-level knowledge file should not simply describe the table. It should tell an agent how to use that table safely.

Example:

---
id: sales.crm.customer_account
kind: table
system: aurora-postgresql
cloud: aws
database: crm_prod
schema: sales
table: customer_account
domain: sales
owner: revenue-operations
steward: data-platform-team
environment: production
classification: confidential
pii: true
freshness_sla: "15 minutes"
source_system: salesforce
agent_usage: allowed_with_row_level_controls
approval_status: approved
---

# customer_account

## Business Meaning

This table represents customer account records synchronized from Salesforce into the CRM production database.

It is the approved source for account ownership, account segment, customer lifecycle stage, and sales territory mapping.

## Agent Usage Guidance

Use this table for customer account analysis, sales ownership, account segmentation, and lifecycle stage reporting.

Do not use this table for audited revenue reporting, invoice reconciliation, or financial close reporting.

For revenue reporting, use the approved finance revenue table.

## Important Caveats

Some legacy accounts may have missing segment values. Agents must not infer missing segment values without confirmation.

This table contains confidential customer information. Agents must apply row-level access controls and masking rules where applicable.

This is much more useful than a raw schema.

The schema tells the agent what exists.

The knowledge file tells the agent how to use it.

Generating Open Knowledge Format from AWS NoSQL Databases

NoSQL systems need even more care.

In DynamoDB, the table name and attributes rarely tell the full story. The real design is usually in the access patterns.

A DynamoDB table may store multiple entity types. It may use composite keys. It may depend on global secondary indexes. It may be optimized for specific queries and unsuitable for others.

If an agent does not understand this, it can misuse the table, trigger inefficient scans, produce incomplete answers, or misunderstand the business process.

For DynamoDB, the knowledge file should capture:

Table purpose
Partition key
Sort key
Item types
Common item shapes
Global secondary indexes
Local secondary indexes
Streams
TTL rules
Primary access patterns
Anti-patterns
Sensitive attributes
Agent permissions
Operational caveats

The most important part is access pattern documentation.

For example:

Access pattern 1:
Get all events for an order
PK = order_id
SK = event_timestamp

Access pattern 2:
Get latest order status
PK = order_id
Sort descending by event_timestamp
Limit = 1

Access pattern 3:
Investigate failed payment events
Use event_type index if available

This tells the agent how the table is actually meant to be used.

Example:

---
id: commerce.dynamodb.order_events
kind: nosql_table
system: dynamodb
cloud: aws
table: order_events
domain: commerce
owner: order-platform-team
environment: production
partition_key: order_id
sort_key: event_timestamp
billing_mode: PAY_PER_REQUEST
stream_enabled: true
classification: confidential
pii: false
agent_usage: allowed_read_only
approval_status: approved
---

# order_events

## Business Meaning

This table stores the event history of customer orders.

Each item represents an event in the order lifecycle, such as order created, payment completed, shipment initiated, shipment delivered, cancellation requested, or refund completed.

## Primary Access Pattern

Retrieve the event timeline for a specific order.

PK = order_id  
SK = event_timestamp

## Agent Usage Guidance

Agents should use this table to reconstruct order history, check operational status, and investigate order workflow issues.

Agents should not use this table as the financial source of truth for revenue, refunds, or payment settlement.

## Caveats

This table is append-only.

The latest operational event should not be treated as financial completion. For accounting status, agents must check the finance ledger.

This prevents an agent from treating NoSQL like a normal relational model.

Generating Knowledge from Document Databases

For document databases such as Amazon DocumentDB or MongoDB-compatible systems, the main challenge is flexible structure and nested sensitive data.

A support case document may contain customer messages. A customer profile may contain personal data. A workflow document may include internal comments, commercial terms, or escalation notes.

Agents need clear rules before reading, summarizing, or exposing this type of content.

For document collections, the knowledge file should capture:

Collection purpose
Common document structure
Required fields
Optional fields
Nested arrays
Indexes
Query patterns
Sensitive fields
Masking rules
Retention rules
Allowed agent use cases
Prohibited use cases

Example:

---
id: support.documentdb.customer_cases
kind: document_collection
system: documentdb
cloud: aws
database: support_prod
collection: customer_cases
domain: customer-support
owner: support-platform-team
environment: production
classification: restricted
pii: true
agent_usage: allowed_with_masking
approval_status: approved
---

# customer_cases

## Business Meaning

This collection stores customer support cases raised through web, email, account manager, and internal escalation channels.

It is used to view case history, identify recurring issues, and prepare escalation summaries.

## Agent Usage Guidance

Agents can use this collection for internal case summaries, issue classification, support briefings, and next-action recommendations.

Agents must not expose raw customer messages externally without masking sensitive information.

## Sensitive Fields

- customer_email
- phone_number
- messages.message
- account_id
- internal_notes

## Caveats

Free-text messages may contain sensitive personal or commercial information. Agents should summarize the issue and intent rather than copying raw text.

This is not just documentation. It is a guardrail.

The AWS Pipeline I Would Build

I would not make this a manual documentation exercise.

That will not scale.

I would build a pipeline that generates draft knowledge files automatically, then routes critical content for human review.

A practical AWS-based pipeline would look like this:

AWS data sources
   ↓
Metadata extraction
   ↓
Profiling and classification
   ↓
Business enrichment
   ↓
OKF draft generation
   ↓
Human review and approval
   ↓
Git-based version control
   ↓
Indexing into agent knowledge retrieval
   ↓
Agent execution
   ↓
Feedback loop

The extraction layer would pull from:

AWS Glue Data Catalog
RDS and Aurora metadata
Redshift catalog tables
DynamoDB DescribeTable
DynamoDB exports to S3
DocumentDB collection profiling
AWS tags
CloudWatch metrics
IAM and Lake Formation policies
Existing documentation and runbooks

The enrichment layer would add:

Business definitions
Source-of-truth mapping
Ownership
Usage guidance
Sensitivity classification
Approved joins
Known caveats
Agent-specific instructions

The governance layer would make sure the knowledge is trusted before agents rely on it.

This is important.

If we automate everything without review, we risk creating wrong knowledge at scale.

If we manually write everything, we will never scale.

The practical answer is auto-generation with human governance.

Governance Is Not Optional

Enterprise agents need trust boundaries.

Every knowledge file should have ownership and lifecycle metadata.

At minimum:

owner: finance-operations
steward: data-platform-team
classification: confidential
approval_status: approved
agent_usage: allowed_internal_only
last_reviewed_at: "2026-06-18"
next_review_due: "2026-09-18"
confidence: high

This gives the framework accountability.

If an agent uses a revenue definition, we should know who approved it.

If an agent queries a customer table, we should know whether it contains PII.

If an agent summarizes a support case, we should know what masking rules apply.

Governance is not bureaucracy here.

Governance is what allows agentic AI to move from demo to production.

The Feedback Loop

The best part of this approach is that the knowledge layer can improve over time.

Agents should not only consume knowledge. They should help identify gaps in it.

For example:

A data agent discovers that two tables have conflicting definitions.
A support agent identifies a recurring customer exception.
A delivery agent finds that a release checklist is outdated.
A FinOps agent identifies an untagged resource pattern.
A sales agent finds that a metric definition is ambiguous.

But agents should not directly overwrite approved knowledge.

They should propose updates.

The workflow should be:

Agent identifies reusable learning
   ↓
Agent creates OKF update proposal
   ↓
Domain owner reviews
   ↓
Approved change is merged
   ↓
Knowledge index is refreshed
   ↓
Future agents use improved context

This turns agentic AI into a learning operating model, not just task automation.

Where I Would Start

I would not start by documenting the whole enterprise.

That sounds ambitious, but it is usually a bad execution plan.

It becomes a documentation program, not an AI acceleration program.

I would start with one high-value domain where correctness matters.

Good candidates are:

Revenue analytics
Customer support
Delivery governance
Cloud cost optimization
Sales operations
Data quality monitoring

For the first MVP, I would select 10 to 20 high-value datasets and generate knowledge files around them.

The MVP should include:

AWS metadata extraction
Draft OKF generation
Manual business enrichment
Data owner approval
Git-based versioning
Agent retrieval integration
Measurement of answer quality and tool accuracy

The goal is not to create perfect documentation.

The goal is to make agents more accurate, more governed, and more useful.

How I Would Measure Success

The wrong metric is “number of OKF files created.”

That only measures documentation volume.

The right metrics are:

Reduction in incorrect SQL generation
Reduction in wrong source-of-truth usage
Increase in agent answer accuracy
Reduction in human corrections
Increase in approved knowledge reuse
Reduction in deprecated table usage
Improvement in data discovery time
Fewer governance violations
Higher user trust in agent outputs

The question should always be:

Did the knowledge layer make the agent better?

If not, we are just creating another documentation repository.

Final View

Google Open Knowledge Format is not interesting because markdown and YAML are new.

They are not.

It is interesting because it points to one of the most important problems in enterprise AI: how to make organizational knowledge usable by agents without locking it inside one platform.

In an AWS environment, we already have many of the raw signals: RDS schemas, Aurora metadata, Redshift catalogs, DynamoDB keys and indexes, Glue Data Catalog, S3 exports, CloudWatch metrics, tags, IAM policies, Lake Formation rules, and existing documentation.

The opportunity is to convert these scattered signals into a governed Enterprise Knowledge Layer.

That layer becomes the memory and context foundation for agentic AI.

My view is simple:

Models give agents reasoning power.

Tools give agents execution power.

Knowledge gives agents enterprise judgment.

Without that knowledge layer, agentic AI will remain impressive in demos and fragile in production.

With it, enterprises can build agents that do not just act fast, but act correctly, safely, and in alignment with how the business actually works.

Takeway from AWS Generative AI Lens

Amit Kayal — Thu, 11 Jun 2026 05:41:26 +0000

I was going through the AWS Generative AI Lens recently, especially the sections on agentic AI, real-world scenarios, and cost optimization.

My biggest takeaway was simple:

Enterprise AI is not about adding a chatbot everywhere.

It is about deciding where AI should assist, where it should reason, and where it should actually take action.

That difference is important.

A chatbot is mostly a conversation layer. It answers questions, summarizes information, or helps users find something faster. Useful, yes. But limited.

Agentic AI goes one step further. It can understand a goal, collect context, use tools, call APIs, check results, and continue the workflow. That is where things become powerful, but also risky.

And honestly, I think this is where many teams will make mistakes.

They will jump directly into “autonomous agents” because it sounds advanced. But in most enterprise systems, full autonomy should not be the first step.

A lot of use cases only need controlled AI-assisted workflows.

For example, if a support ticket comes in, AI can classify the issue, extract the important details, check the knowledge base, and suggest the next action. That does not need a fully autonomous agent. It needs a reliable workflow with a few intelligent decision points.

The same applies to document processing, internal knowledge search, compliance checks, report generation, and many operational tasks.

Start simple. Keep the flow predictable. Add autonomy only where the process genuinely needs it.

The AWS scenarios helped me think about this more practically. Use cases like autonomous call centers, generative BI, incident response, code review, Kanban workflows, and knowledge-worker copilots are not just “AI features.” They are business workflows being redesigned with AI inside them.

That is the right mental model.

Take generative BI as an example.

The value is not just that a user can ask a question in English and get a chart. The real value is that business users can get answers without always depending on analysts or knowing the database structure.

But this only works if the data layer is governed properly. Access control, semantic consistency, row-level security, auditability, and accuracy all matter. Otherwise, the system may produce a confident answer that is wrong, or worse, expose information to the wrong person.

So the hard part is not the natural language interface.

The hard part is trust.

Incident response is another strong example.

During an incident, engineers waste a lot of time moving between dashboards, logs, alerts, deployment history, tickets, and runbooks. An AI-assisted incident system can collect this context quickly, summarize what changed, compare with past incidents, and suggest likely causes.

That is a good use case for agentic behavior because incidents are rarely linear. The system needs to investigate, adjust, and reason through incomplete information.

But even there, I would not give the agent full production control on day one.

Let it investigate first.

Then let it recommend.

Then automate low-risk actions.

Only after enough confidence, allow higher-impact actions with approval.

That is how autonomy should mature in the enterprise.

Another important AWS scenario is the multi-tenant generative AI platform. I think this will become very relevant for larger organizations.

Without a shared platform, every team starts building its own AI stack. One team builds its own RAG pipeline. Another team creates its own prompt management. Another team handles model access differently. Cost tracking becomes scattered. Guardrails become inconsistent. Security reviews become repetitive.

That does not scale.

A central AI platform can solve this by providing reusable capabilities: model access, retrieval, evaluation, guardrails, monitoring, cost visibility, and deployment patterns.

But it should not become a slow central gatekeeper.

The better approach is platform plus autonomy. The central team provides the foundation. Business and product teams build specific use cases on top.

That is how companies can move fast without creating AI chaos.

The cost optimization section was also a good reminder.

Generative AI cost behaves differently from traditional software cost.

In a normal application, one user action may trigger one API call or one database query.

In an agentic system, one user request can trigger multiple model calls, retrieval steps, tool calls, retries, evaluations, and follow-up reasoning loops.

If you do not design boundaries, cost can grow silently.

This is why model selection matters. Not every task needs the largest model. A classification task, summarization task, routing task, and deep reasoning task may need different models.

The mature question is not:

“Which model is best?”

The better question is:

“Which model is good enough for this task at the right cost, latency, and reliability?”

That is a very different engineering mindset.

Cost-aware AI design also means keeping prompts clean, controlling response length, optimizing vector stores, caching repeated results, setting iteration limits, and defining clear exit conditions for agents.

Especially for agents, workflow boundaries are critical.

An agent should not keep thinking, retrying, searching, or calling tools without a clear stop condition. That is not intelligence. That is bad engineering.

My final takeaway is this:

AI should be treated as part of the production architecture, not as a side experiment.

The companies that succeed will not be the ones with the flashiest demos. They will be the ones that understand where AI fits into the actual workflow, how much autonomy is safe, what controls are needed, and whether the cost is justified by the outcome.

For me, the real question is no longer:

“How do we add AI to this product?”

The better question is:

“Which workflow can become smarter, faster, or more scalable if AI is designed into it properly?”

That is where agentic AI becomes useful.

Not as hype.

As architecture.

Hosting MCP Gateway Registry on AWS ECS: A Practical Blueprint for Enterprise Agentic AI Systems

Amit Kayal — Sun, 24 May 2026 09:13:03 +0000

Hosting MCP Gateway Registry on AWS ECS: A Practical Blueprint for Enterprise Agentic AI Systems

AI agents are no longer just demo applications that answer questions.

They are slowly becoming systems that can take action: search customer records, update opportunities, generate quotes, create tickets, check inventory, read contracts, trigger workflows, and interact with business applications.

That is where the real enterprise problem begins.

When an AI agent only chats, the risk is limited. But when an agent starts using tools, APIs, and enterprise systems, we need a much stronger operating model. We need to know what the agent can access, who approved that access, what data it can touch, and how we can monitor every action.

This is exactly where an MCP Gateway and Registry becomes important.

The MCP Gateway Registry gives us a central place to register MCP servers, discover available tools, manage authentication, control access, and observe how agents interact with enterprise capabilities.

In this blog, I will walk through how we can host an MCP Gateway Registry on AWS using ECS Fargate, based on the Terraform AWS ECS deployment model from the MCP Gateway Registry project. This blog is based on the repo https://github.com/agentic-community/mcp-gateway-registry/tree/main and all credit goes to repo contributors.

Why This Problem Matters

In early AI agent projects, the architecture usually starts simple.

One agent connects to one or two tools.

For example:

Sales Agent
   |
   |-- Salesforce MCP Server
   |-- Knowledge Base MCP Server

This works well for a proof of concept.

But after some time, more teams start building agents.

The sales team wants Salesforce and quote tools.
The support team wants ticketing and knowledge base tools.
The finance team wants billing and contract tools.
The delivery team wants Jira, project reports, and document search tools.
The leadership team wants reporting and analytics agents.

Very quickly, the environment starts looking like this:

Agent 1 ---> MCP Server A
Agent 1 ---> MCP Server B
Agent 2 ---> MCP Server A
Agent 2 ---> MCP Server C
Agent 3 ---> MCP Server D
Agent 4 ---> MCP Server B
Agent 5 ---> MCP Server E

At this stage, the issue is no longer just technical integration.

The real problems are:

Who owns each MCP server?
Which agent is allowed to use which server?
What permissions does each tool have?
How do we prevent duplicate MCP servers?
How do we audit tool usage?
How do we onboard new tools safely?
How do we remove old or risky tools?
How do we monitor failures?
How do we stop agents from accessing sensitive systems without approval?

If we do not solve this early, the MCP layer can become another uncontrolled integration layer.

And in enterprise systems, uncontrolled integration always becomes a risk.

What an MCP Gateway Registry Actually Does

An MCP Gateway Registry acts as a control plane between AI agents and MCP servers.

Instead of letting every agent directly connect to every MCP server, we introduce a managed gateway and registry layer.

The architecture becomes cleaner:

AI Agents / Developers / Applications
              |
              v
      MCP Gateway and Registry
              |
              v
        Approved MCP Servers
              |
              v
      Enterprise Applications

This gives us a much better operating model.

The registry helps maintain information about available MCP servers:

Server name
Owner
Description
Capabilities
Available tools
Security scopes
Environment
Version
Health status
Approval status
Discovery metadata

The gateway helps control and route access:

Authentication
Authorization
Tool discovery
Request routing
Policy enforcement
Logging
Monitoring
Access control

This is important because enterprise agents should not randomly discover and use tools. They should use approved tools with approved scopes through a governed access path.

Why Hosting This on AWS ECS Makes Sense

There are multiple ways to host an MCP Gateway Registry.

You can run it on virtual machines.
You can deploy it on Kubernetes.
You can run it on ECS.
You can even start with a simple Docker Compose deployment for local testing.

But for an enterprise-grade AWS deployment, ECS Fargate is a very practical option.

It gives us a managed container runtime without the operational overhead of managing EC2 worker nodes or a full Kubernetes control plane.

For this type of gateway, ECS Fargate gives a good balance between simplicity and production readiness.

Key benefits include:

No EC2 server management
Container-based deployment
Built-in integration with IAM
Easy logging through CloudWatch
Service-level health checks
Integration with Application Load Balancer
Auto-scaling support
Good fit for Terraform automation
Lower operational complexity than Kubernetes

In my view, unless an organization already has a mature EKS platform and Kubernetes operating model, ECS Fargate is a better first choice for hosting this kind of control-plane service.

Kubernetes gives more flexibility, but it also adds more operational responsibility. For many teams, that is not needed on day one.

Target AWS Architecture

A production-style AWS architecture for MCP Gateway Registry can look like this:

Users / Agents / Developers
          |
          v
Route 53 Custom Domain
          |
          v
CloudFront
          |
          v
AWS WAF
          |
          v
Application Load Balancer
          |
          v
ECS Fargate Services
   |          |           |
Registry   Auth Server   Keycloak
   |          |           |
   |          |           v
   |          |      Aurora PostgreSQL
   |
   v
Amazon DocumentDB

Supporting Services:
- AWS Secrets Manager
- CloudWatch Logs
- CloudWatch Alarms
- ECR
- IAM
- ACM
- Optional Prometheus and Grafana

This is not just about running containers.

This architecture gives us:

Secure external access
Managed container hosting
Central authentication
Registry persistence
Secret management
Observability
Certificate management
Custom domain support
Infrastructure automation

That is the difference between a demo deployment and an enterprise deployment.

Core AWS Components

1. Amazon ECS Fargate

ECS Fargate runs the containerized services.

The deployment can include multiple services such as:

MCP Gateway Registry
Authentication server
Keycloak
MCP gateway service
Sample MCP servers
Sample agents
Observability components

Each service runs as an ECS task.

In production, I would recommend separating these into clear services rather than bundling too much into one container. This gives better control over scaling, logging, deployments, and troubleshooting.

For example:

Registry service       --> Handles MCP server metadata and discovery
Auth service           --> Handles authentication flow
Keycloak service       --> Identity and access management
Sample MCP services    --> Optional, mostly for demo or validation

For production, sample agents and sample MCP servers should be disabled or deployed only in a non-production environment.

2. Application Load Balancer

The Application Load Balancer exposes the ECS services through HTTPS endpoints.

It performs routing to the correct ECS target group.

For example:

/registry  --> Registry service
/auth      --> Auth service
/keycloak  --> Keycloak service

Or, in a cleaner production model:

registry.company.com  --> Registry service
auth.company.com      --> Auth service
kc.company.com        --> Keycloak

This domain-based separation is better for enterprise usage because it improves clarity, security boundaries, and operational ownership.

3. CloudFront

CloudFront can sit in front of the ALB.

For production, this is useful because it gives:

Global edge access
Better TLS handling
Additional protection layer
Integration point for WAF
Cleaner public access pattern
Potential performance benefits

For internal-only deployments, CloudFront may not always be required. But if the registry is accessed by distributed teams, external developers, or cloud-hosted agents, CloudFront becomes useful.

4. AWS WAF

I would strongly recommend using AWS WAF in front of internet-facing endpoints.

The MCP gateway is a sensitive entry point because it controls access to tools. So it should not be exposed casually.

Useful WAF controls include:

Rate limiting
AWS managed rule groups
IP restrictions
Bot protection
Geo restrictions if required
SQL injection protection
Cross-site scripting protection

This is especially important if agents, developers, or external systems access the gateway over the internet.

5. Route 53 and ACM

Route 53 manages DNS records.

ACM provides SSL/TLS certificates.

This gives us clean URLs such as:

registry.company.com
auth.company.com
kc.company.com

For enterprise adoption, this matters more than people think. Clean domain names make the platform feel like a real internal product rather than a temporary engineering setup.

6. Amazon Aurora PostgreSQL

Aurora PostgreSQL is used for Keycloak data.

Keycloak needs a relational database to store identity-related information, including:

Users
Realms
Clients
Roles
Sessions
Identity provider configuration
Authentication settings

Using Aurora gives better reliability than running a database inside a container.

For production, I would avoid containerized databases for this type of platform. Identity is too important to treat casually.

7. Amazon DocumentDB

DocumentDB is used by the registry layer.

This is where MCP server and agent metadata can be stored.

Example records may include:

MCP server name
MCP server URL
Tool list
Tool descriptions
Security scopes
Server health
Owner team
Environment
Version
Approval state
Risk classification

Over time, this registry becomes the enterprise catalog for agent-accessible capabilities.

This is very valuable.

It allows teams to search and discover what tools already exist instead of rebuilding the same MCP servers again and again.

8. AWS Secrets Manager

Secrets Manager should be used for:

Database credentials
Keycloak admin credentials
JWT secrets
Client secrets
Service credentials
API keys

No production credential should be hardcoded inside Terraform files, Docker images, or environment files stored in Git.

This is basic, but it is often missed in early AI platform projects.

9. CloudWatch Logs and Alarms

Every ECS service should write logs to CloudWatch.

At minimum, we should monitor:

Container startup failures
Authentication failures
Registry API errors
Tool discovery failures
Database connection errors
ECS task restarts
ALB 4xx errors
ALB 5xx errors
High latency
Memory pressure
CPU pressure

But for an MCP gateway, infrastructure logs are not enough.

We also need agent activity logs.

For example:

Which agent requested tool discovery?
Which MCP server was selected?
Which tool was invoked?
Which scope was used?
Was the request allowed or denied?
What was the response status?
How long did the tool call take?
Was sensitive data involved?

This is where the MCP gateway starts becoming a governance system, not just a routing layer.

Deployment Options

The Terraform setup supports different deployment modes.

Option 1: CloudFront Only

This is useful for a quick POC.

You do not need a custom domain. You get a CloudFront-generated URL.

This is suitable for:

Internal demo
Engineering validation
Architecture exploration
Short-term sandbox

This is not my preferred option for production, but it is a good way to start quickly.

Option 2: Custom Domain Only

In this model, Route 53 and ACM are used, but CloudFront may not be enabled.

You get URLs like:

registry.company.com
kc.company.com

This is better than a random generated URL, but it may not give enough edge protection if exposed publicly.

This can work well for private/internal deployments.

Option 3: CloudFront + Custom Domain

This is the best production model.

Traffic flows like this:

User / Agent
    |
    v
Custom Domain
    |
    v
CloudFront
    |
    v
WAF
    |
    v
Application Load Balancer
    |
    v
ECS Fargate Service

This gives a stronger production posture.

My recommendation:

Use CloudFront + Route 53 + WAF for production.
Use CloudFront-only for demo.
Use custom domain-only only for controlled internal environments.

Practical Deployment Flow

The deployment flow can be divided into clear stages.

Stage 1: Prepare AWS Account

Before starting, we should decide:

AWS region
VPC strategy
Domain name
Environment name
Access model
CIDR restrictions
Secrets strategy
Terraform state backend

For production, I would not deploy this into a random shared AWS account.

Better model:

Separate AWS account for dev
Separate AWS account for staging
Separate AWS account for production

At minimum, use separate environments and separate Terraform state.

Stage 2: Build and Push Images to ECR

The services need to be built as Docker images and pushed to Amazon ECR.

A simplified flow:

export AWS_REGION=us-east-1
make build-push

The result is a set of ECR image URIs.

Example:

123456789012.dkr.ecr.us-east-1.amazonaws.com/mcp-gateway-registry:v1.0.0
123456789012.dkr.ecr.us-east-1.amazonaws.com/mcp-gateway-auth:v1.0.0
123456789012.dkr.ecr.us-east-1.amazonaws.com/mcp-gateway:v1.0.0

For production, avoid using latest.

Use versioned immutable tags.

Bad:

mcp-gateway-registry:latest

Better:

mcp-gateway-registry:v1.0.3

Best:

mcp-gateway-registry:v1.0.3-build-20260524

This helps with rollback, audit, and release traceability.

Stage 3: Configure Terraform Variables

The terraform.tfvars file is where we configure the deployment.

Important values include:

aws_region = "us-east-1"

enable_cloudfront  = true
enable_route53_dns = true

base_domain = "company.com"

session_cookie_domain = ".company.com"
session_cookie_secure = true

ingress_cidr_blocks = [
  "YOUR_OFFICE_IP/32",
  "YOUR_VPN_IP/32"
]

Database and admin passwords should be handled carefully.

In a strong production model, these should come from a secure secret injection process rather than being manually placed in local files.

Stage 4: Initialize Terraform

Run:

terraform init -upgrade

For production, Terraform state should be stored remotely.

Recommended backend:

S3 bucket for state
DynamoDB table for locking
KMS encryption
Restricted IAM access

Do not use local state for production.

Local state is acceptable for learning, but not for enterprise infrastructure.

Stage 5: Create Certificates First

ACM certificates often require DNS validation.

That is why the deployment may need a first targeted apply for certificates.

Conceptually:

terraform apply \
  -target=aws_acm_certificate.keycloak \
  -target=aws_acm_certificate.registry \
  -target=aws_acm_certificate_validation.keycloak \
  -target=aws_acm_certificate_validation.registry

This allows certificates to be created and validated before the rest of the infrastructure depends on them.

Stage 6: Deploy Full Infrastructure

After certificate validation:

terraform apply

This deploys the full stack:

Networking
Security groups
ECS cluster
ECS services
ALB
Target groups
CloudFront
Route 53 records
Aurora PostgreSQL
DocumentDB
Secrets
CloudWatch logs
IAM roles
Optional observability stack

At this point, the infrastructure is created, but the application may still need initialization.

Stage 7: Run Post-Deployment Setup

Post-deployment setup is very important.

This step usually performs:

Terraform output extraction
DNS validation
ECS service health checks
Keycloak realm setup
Client setup
Admin user setup
DocumentDB collection initialization
Registry indexes
Scope setup
Service restart
Endpoint validation

This step converts infrastructure into a usable platform.

Without this, the containers may be running, but the gateway may not be fully ready.

How the Gateway Should Be Used After Hosting

Once deployed, teams can start registering MCP servers.

A good MCP server registration should include:

Server name
Business capability
Owner team
Technical owner
Environment
Base URL
Supported tools
Required scopes
Risk level
Data classification
Health check endpoint
Approval status
Version

For example:

Name: Salesforce Opportunity MCP Server
Owner: Sales Platform Team
Environment: Production
Tools:
- searchOpportunity
- updateOpportunityStage
- getAccountDetails
Scopes:
- salesforce.read
- salesforce.opportunity.update
Risk: High
Data: Customer and revenue data
Approval: Required

This level of metadata is important.

Without it, the registry becomes just another technical catalog. With it, the registry becomes a real enterprise control plane.

Enterprise Governance Model

For enterprise usage, I would define a clear lifecycle for MCP servers.

Suggested MCP Server Lifecycle

Draft
   |
Submitted for Review
   |
Security Review
   |
Approved for Dev
   |
Approved for Production
   |
Monitored
   |
Deprecated
   |
Retired

Every MCP server should have an owner.

Every high-risk tool should have approval.

Every production MCP server should have monitoring.

Every deprecated server should have a retirement date.

This may sound heavy, but it is necessary once agents start touching real systems.

Access Control Model

The gateway should not allow all agents to use all MCP servers.

That is a weak design.

A better model is scope-based access.

Example:

Agent: Sales Copilot
Allowed scopes:
- salesforce.read
- quote.read
- product.search

Not allowed:
- discount.approve
- contract.delete
- customer.export

Another example:

Agent: Deal Desk Agent
Allowed scopes:
- quote.read
- quote.update
- discount.request
- contract.read

Requires approval:
- discount.approve
- final_quote.submit

This is how we prevent agents from becoming over-permissioned.

One of the biggest risks in agentic AI systems will be excessive tool permission. If we give one agent too many tools and too much authority, it becomes hard to control behavior and impact.

Observability for Agentic Systems

Traditional application monitoring is not enough here.

We need both system observability and agent observability.

System Observability

Track:

CPU
Memory
Container restarts
Task failures
ALB errors
Request latency
Database connections
Authentication errors

Agent and Tool Observability

Track:

Agent ID
User ID
Tool requested
MCP server used
Scope used
Decision outcome
Policy result
Execution latency
Failure reason
Data classification
External system touched

For example, a useful audit log may look like this:

{
  "agent": "sales-copilot",
  "user": "john@company.com",
  "mcp_server": "salesforce-opportunity-server",
  "tool": "updateOpportunityStage",
  "scope": "salesforce.opportunity.update",
  "decision": "allowed",
  "timestamp": "2026-05-24T10:15:00Z",
  "latency_ms": 450,
  "status": "success"
}

This type of logging becomes extremely important when something goes wrong.

If an agent updates the wrong opportunity or calls a pricing tool incorrectly, we should be able to reconstruct exactly what happened.

CI/CD Model

For production, deployment should not be manual.

A good CI/CD pipeline should look like this:

Developer raises PR
        |
Code review
        |
Build Docker images
        |
Run unit tests
        |
Run container security scan
        |
Push image to ECR
        |
Terraform plan
        |
Manual approval for production
        |
Terraform apply
        |
Run post-deployment setup
        |
Smoke test
        |
Notify platform team

This keeps the deployment controlled and auditable.

For rollback, the team should be able to redeploy a previous image tag quickly.

Recommended Environment Strategy

I would recommend at least three environments.

Development
Staging
Production

Development

Used for engineering testing.

Can have relaxed settings.

Sample MCP servers allowed
Lower database capacity
CloudFront-only mode acceptable
Limited monitoring

Staging

Used for pre-production validation.

Should be close to production.

Custom domain
WAF enabled
Production-like IAM
Production-like secrets
Observability enabled

Production

Used for real enterprise agents.

Should be hardened.

Separate AWS account
CloudFront + WAF
Private subnets
Strict ingress
Immutable images
Centralized logs
Audit trail
Backup enabled
Approval workflow

Production Hardening Checklist

Before calling this production-ready, I would validate the following.

Remote Terraform state enabled
Terraform state encrypted
DynamoDB locking enabled
Separate AWS accounts or environments
Secrets stored in Secrets Manager
No secrets in Git
CloudFront enabled
WAF enabled
Ingress restricted
Keycloak admin access restricted
ECS tasks in private subnets
ALB security groups reviewed
Aurora backups enabled
DocumentDB backups enabled
CloudWatch alarms configured
Container image scanning enabled
Immutable image tags used
IAM least privilege applied
Audit logging enabled
MCP server ownership defined
Tool scopes defined
Production approval process defined
Runbook created
Rollback process tested

The most common mistake is to stop after the Terraform deployment succeeds.

That only means infrastructure exists.

It does not mean the platform is secure, governed, observable, or ready for production.

Operational Runbook

For a serious enterprise setup, the platform team should maintain a simple runbook.

The runbook should answer:

How do we onboard a new MCP server?
How do we approve a production MCP server?
How do we revoke access?
How do we rotate secrets?
How do we check service health?
How do we debug registry failures?
How do we debug authentication failures?
How do we rollback a release?
How do we retire an old MCP server?
How do we investigate suspicious tool usage?

This is where platform maturity comes in.

An MCP gateway is not a one-time deployment. It becomes part of the agentic AI platform.

Where This Fits in an Enterprise Agent Architecture

In a broader enterprise agentic AI architecture, the MCP Gateway Registry sits between orchestration and enterprise tools.

A practical model:

User Interface
      |
      v
Agent Orchestrator
      |
      v
Policy / Guardrail Layer
      |
      v
MCP Gateway Registry
      |
      v
MCP Servers
      |
      v
Enterprise Systems

The orchestrator decides what needs to be done.

The policy layer checks whether the action is allowed.

The MCP gateway provides controlled tool discovery and access.

The MCP server performs the actual system interaction.

This separation is important.

Do not put all responsibilities into one big agent.

That becomes hard to scale, hard to debug, and dangerous to govern.

My Practical Recommendation

For a real enterprise deployment, I would host the MCP Gateway Registry with this setup:

AWS ECS Fargate for services
CloudFront in front
AWS WAF enabled
Route 53 custom domains
ACM certificates
Application Load Balancer
Private subnets for ECS tasks
Aurora PostgreSQL for Keycloak
DocumentDB for registry metadata
Secrets Manager for credentials
CloudWatch for logs and alarms
Optional Grafana and Prometheus for deeper observability
S3 backend for Terraform state
DynamoDB for Terraform locking
CI/CD for image build and deployment
Immutable ECR image tags
Strict admin access
Scope-based authorization
Audit logs for all tool usage

For a POC, I would keep it simple.

For production, I would not compromise on security, logging, and access control.

Key Lessons Learned

The biggest lesson is this:

Hosting the MCP Gateway Registry is not only an infrastructure activity. It is the beginning of an operating model for enterprise agents.

If agents are going to use real tools, then organizations need:

Tool ownership
Tool approval
Tool discovery
Tool scopes
Tool observability
Tool lifecycle management
Tool risk classification

Without this, agentic AI systems may work technically but fail operationally.

And in enterprises, operational failure is usually what blocks adoption.

Final Thought

MCP is making tool integration more standard for AI agents. That is a very important shift.

But standardization also creates scale.

And once we scale the number of agents and tools, we need governance.

That is why an MCP Gateway Registry should be treated as a core platform capability, not as a side component.

It gives engineering teams a structured way to expose tools.
It gives security teams a way to control access.
It gives platform teams a way to monitor usage.
It gives business teams more confidence that agents are not directly and blindly touching enterprise systems.

In my view, this is one of the important building blocks for production-grade agentic AI systems.

The future will not be one agent directly connected to many tools.

The future will be governed agent ecosystems, where tools are registered, discoverable, monitored, secured, and lifecycle-managed through a central control plane.

When One AI Agent Is Not Enough: A Practical Delegation Pattern for Enterprise Systems

Amit Kayal — Sat, 23 May 2026 17:50:08 +0000

When One AI Agent Is Not Enough: A Practical Delegation Pattern for Enterprise Systems

A lot of enterprise AI systems start the same way.

One agent.
One big prompt.
A bunch of tools.
A lot of hope.

At first, it looks great. The agent can answer questions, call a few systems, maybe even complete a useful workflow. But once the use case gets more realistic, cracks start to show.

The agent has to understand too much.
It has to access too many systems.
It has to make too many different kinds of decisions.
And when something goes wrong, it is hard to tell where the problem actually is.

That is usually the point where the issue stops being “prompt quality” and starts becoming “system design.”

One pattern I’ve found especially useful is delegation across agents and subagents.

Not because it sounds advanced.
Because it is often the more practical way to build enterprise AI.

The real problem with a single large agent

There is an appealing simplicity in saying, “Let one agent handle the whole thing.”

But enterprise workflows are rarely that clean.

Take something simple on the surface, like a customer escalation.

To handle it well, the system may need to:

pull ticket history
understand product context
check support policy
review account state
recommend next actions
trigger an internal workflow
draft a reply

Yes, one agent can try to do all of that.

But in practice, the more responsibilities you pile into one agent, the more fragile it becomes.

You usually end up with:

too much context going into one step
too many tools available to one component
weaker predictability
weaker governance
and much harder debugging

The system may still “work,” but it becomes difficult to trust.

A better pattern: one lead agent, a few focused subagents

The cleaner pattern is this:

Primary agent -> specialist subagents -> final outcome

The primary agent owns the workflow.

Its job is to understand the request, decide what needs to happen, delegate the right pieces of work, and then combine the results.

The subagents each do one thing well.

For example:

a retrieval subagent gets the right context
a policy subagent checks rules or entitlements
an analysis subagent recommends next steps
an execution subagent handles approved downstream actions
a communication subagent drafts the final message

That is a much healthier design than asking one broad agent to do everything in one pass.

Why this pattern works better

The first reason is simple: focus.

A retrieval subagent can focus on retrieval.
A policy subagent can focus on policy.
An execution subagent can focus on action.

You are not forcing one component to juggle too many responsibilities.

The second reason is control.

Different subagents can have different permissions, different tools, and different operating boundaries. That is much easier to govern in enterprise systems.

The third reason is observability.

If the outcome is wrong, you have a better shot at knowing where it went wrong:

bad retrieval
wrong policy interpretation
weak action selection
poor response generation

That is a huge advantage once the system moves beyond demo stage.

What the primary agent should actually do

One mistake I see is treating the primary agent like a simple router.

That is not enough.

The primary agent should behave more like a coordinator.

It should:

understand the incoming request
decide what subtasks are needed
choose the right subagents
pass only the necessary context
review what comes back
and decide whether to continue, retry, escalate, or stop

In other words, it owns the workflow logic.

It should not blindly trust every subagent output.
It should have judgment.

That is what makes delegation useful rather than just decorative.

What makes a good subagent

A good subagent is narrow.

That is probably the single most important design rule.

Each subagent should ideally have:

one clear job
limited tools
limited context
a defined output format
clear boundaries on what it should not do

If a subagent is doing retrieval, analysis, execution, and communication together, it is no longer a real specialist.

It is just another general-purpose agent with a different label. And once you do that, the value of delegation starts disappearing.

A sharper example

Let’s go back to the customer escalation example.

Bad design

One large agent receives the case and tries to:

read the issue
search past history
check policy
assess severity
decide the next action
update internal systems
draft the reply

This may work sometimes.

But it is too much responsibility in one place.

Better design

Primary agent
Owns the overall case flow.

Retrieval subagent
Gathers ticket history, account context, product details, and related documentation.

Policy subagent
Checks entitlement, SLA, escalation rules, and any support constraints.

Analysis subagent
Looks at the combined context and suggests the best next step.

Execution subagent
Triggers the approved workflow, creates tasks, or updates systems.

Communication subagent
Drafts the customer-facing or internal message.

Now the workflow is clearer.
Each step is easier to test.
And if the result is weak, you can usually tell why.

When delegation is worth it

Not every use case needs this pattern.

Sometimes one well-designed agent is enough.

Delegation becomes useful when:

the workflow crosses different domains
different systems or permissions are involved
some work can happen in parallel
one agent is becoming overloaded
governance starts getting messy
you want better testing and failure isolation

If the workflow is small and bounded, keep it simple.

The point is not to add more agents for the sake of it.
The point is to use delegation when specialization clearly improves the system.

Practical rules that help

1. Start with a small number of subagents

Do not build a maze.

Start with one primary agent and maybe two or three specialists. That is usually enough to prove whether the pattern is helping.

2. Keep context tight

Do not pass everything to every agent.

Each subagent should get only the context it actually needs. Too much context often makes outputs worse, not better.

3. Use structured outputs

Subagents should return something predictable:

a decision
a label
a ranked list
a JSON object
a recommendation plus confidence

Not vague prose that another component has to guess at.

4. Design low-confidence paths

If a subagent is not confident, that should trigger something explicit:

retry
clarification
fallback logic
human review

Do not let weak outputs quietly flow into the rest of the chain.

5. Log the handoffs

You need to know:

what task was delegated
what context was passed
what came back
what happened next

Without that, debugging becomes painful very quickly.

6. Control tools by role

A retrieval subagent should not have broad execution rights.
An execution subagent should not have unnecessary access to everything.
Different responsibilities should have different permissions.

That is one of the easiest ways to keep governance strong.

Common mistakes

A few patterns show up again and again.

Too many agents too early
More moving parts do not automatically make the design better.

Subagents with overlapping jobs
If roles are fuzzy, delegation becomes noisy.

Passing all context everywhere
That weakens specialization fast.

No fallback design
One failed subtask should not silently break the whole workflow.

This is an architecture pattern.

Final thought

Delegation across agents and subagents is one of the more practical patterns in enterprise AI.

Not because it is clever.
Because it reflects how real systems usually need to operate.

The strongest setups are usually not the ones with the most agents.

They are the ones where:

the primary agent clearly owns the workflow
the subagents are genuinely specialized
the context is controlled
the outputs are structured
and the operating model is easy to debug and govern

That is what turns a multi-agent design from an interesting idea into something you can actually run in production.

A Scaling Lesson Building Production-Grade Agentic AI Systems

Amit Kayal — Tue, 19 May 2026 18:30:50 +0000

A Scaling Lesson Building Production-Grade Agentic AI Systems

One of the early observations we had while designing enterprise AI agents was this:

Giving an agent more tools does not necessarily make it smarter.

In theory, it sounded correct.

If an agent had access to customer systems, payment systems, inventory, shipping, reporting, ticketing, email, scheduling, analytics, and internal knowledge bases — it should become more powerful and autonomous.

But what we observed in real implementations was very different.

The more tools we added, the more unstable the system became.

Not because the model was weak.

Not because the tools were poorly built.

But because the agent’s decision space became too large.

For every user request, the agent had to evaluate all available tools, compare descriptions, infer intent, decide sequencing, and determine the best execution path.

Now imagine doing this with 18 tools:

Customer lookup
Order search
Refund processing
Inventory checking
Shipping tracking
Email sending
Ticket creation
Knowledge base search
Sentiment analysis
Language translation
Calendar scheduling
Report generation
Data export
User authentication
Payment processing
Discount application
Feedback collection
Escalation routing

Initially, everything looked manageable.

But as workflows became more dynamic, we started observing:

wrong tool selection,
unnecessary tool chaining,
higher latency,
increased token usage,
inconsistent execution paths,
and occasional hallucinated actions.

The problem was not intelligence.

The problem was cognitive overload inside the orchestration layer.

Over time, one pattern became very clear:

Agents perform significantly better when their responsibility boundaries are smaller.

In our experience, once an agent moves beyond roughly 4–5 actively usable tools, reliability starts dropping rapidly. Similar enterprise orchestration patterns are now recommending smaller, specialized agents instead of monolithic “super agents.”

That observation changed how we started designing AI systems.

Instead of building one massive “do everything” agent, we moved toward specialized agents with tightly scoped responsibilities.

For example:

A support agent handles:

customer lookup,
ticket creation,
escalation routing,
knowledge retrieval.

A commerce agent handles:

orders,
refunds,
discounts,
payments.

An operations agent handles:

shipping,
inventory,
reporting,
exports.

This immediately improved:

tool accuracy,
execution consistency,
observability,
debugging,
latency,
and operational trust.

But another important learning came later.

Even after distributing tools properly, systems still degraded when too many agents were active simultaneously.

This is something many teams underestimate.

As the number of agents increases, coordination overhead also increases:

more inter-agent communication,
more memory synchronization,
more orchestration reasoning,
more retries,
more conflict resolution,
and more state tracking.

At lower scale, this is manageable.

At enterprise scale, it becomes a serious engineering challenge.

We observed cases where:

agents started waiting on each other,
orchestration layers became bottlenecks,
duplicate reasoning increased token burn,
cascading retries created operational instability,
and observability became extremely difficult.

Multi-agent systems introduce their own scaling complexity around coordination, governance, and orchestration overhead. Most production-grade architecture guidance today recommends keeping orchestration layers as simple as possible.

Over time, we established a few practical thumb rules internally.

Some Practical Thumb Rules We Follow Now

1. Keep Tool Count Small Per Agent

Our practical guideline today is:

3–5 tools → ideal
6–8 tools → manageable with careful prompting
10+ tools → requires routing/filtering layers
15+ tools → usually an architectural warning sign

The issue is not model capability.

It is decision dilution.

2. Every Agent Must Have One Clear Business Responsibility

We avoid mixing domains.

For example:

payments + support,
analytics + execution,
reporting + approvals,
inventory + customer engagement.

The narrower the responsibility boundary, the more predictable the behavior.

3. Start With the Lowest Complexity Possible

One important learning from enterprise orchestration patterns is this:

Do not introduce multi-agent architecture unless the workflow genuinely requires it.

Sometimes:

a prompt is enough,
sometimes a single agent is enough,
sometimes workflows are better handled through deterministic orchestration.

Not every problem needs “AI teamwork.”

4. Avoid Excessive Agent-to-Agent Conversations

Agent collaboration sounds powerful in demos.

But in production:

every interaction increases latency,
every message consumes tokens,
every dependency creates failure paths.

We now aggressively reduce unnecessary conversations between agents.

5. Retrieval Before Reasoning

Instead of exposing all tools to all agents, we first narrow candidates through:

semantic routing,
metadata filtering,
RAG-based retrieval,
workflow classification.

This significantly improves tool selection accuracy and reduces reasoning load.

6. Observability Is Mandatory

Once systems become multi-agent, debugging becomes one of the hardest engineering problems.

We now treat the following as first-class requirements:

distributed tracing,
token tracking,
step-level logging,
execution replay,
agent health monitoring,
retry visibility,
and orchestration graphs.

Without observability, production support becomes nearly impossible.

7. Human Escalation Is Still Critical

One thing we intentionally avoid is trying to automate every decision.

We now introduce human checkpoints for:

financial operations,
policy-sensitive actions,
low-confidence reasoning,
and customer-impacting workflows.

Autonomy without governance becomes operational risk.

What I increasingly believe is that the future of enterprise AI is not one giant super-agent.

It is orchestrated systems of smaller specialized agents collaborating through routing, delegation, memory sharing, and controlled execution.

The real engineering challenge is no longer:
“How many tools can an agent use?”

The better question is:
“How effectively can we reduce the decision burden for each agent while keeping orchestration manageable?”

That has become one of the most important scaling lessons for us while building production-grade agentic AI systems.

How We Are Thinking About This in Cloud Architecture

One important realization for us was that multi-agent systems should not be treated as a single application deployment.

They should be treated as distributed cloud-native systems.

That changes the architecture significantly.

Today, the architecture pattern we increasingly follow looks something like this:

Specialized Agents as Independent Services

Each agent runs independently with:

isolated APIs,
dedicated scaling,
separate observability,
isolated memory/context,
and domain-level permissions.

This reduces blast radius and improves operational governance.

In AWS, this naturally aligns very well with:

Lambda,
ECS/EKS,
event-driven services,
queues,
Bedrock,
and serverless orchestration patterns.

What I personally liked while evaluating newer AWS patterns is how Amazon Bedrock AgentCore is trying to standardize several production concerns around agents. Instead of teams writing custom orchestration glue repeatedly, AgentCore is introducing managed capabilities around:

runtime isolation,
observability,
memory,
identity,
tool gateways,
and orchestration patterns.

One thing I strongly relate to from practical experience is this:

Building the reasoning layer is usually not the hardest part anymore.

The harder part is:

orchestration,
debugging,
tracing,
retries,
governance,
and operational scalability.

That is where systems usually become unstable at scale.

AWS AgentCore Observability is also moving in an interesting direction by treating agent execution visibility as a first-class production capability with:

execution tracing,
token monitoring,
latency tracking,
tool usage visibility,
and CloudWatch integration. ()

When you have multiple agents collaborating dynamically, you need visibility into:

why a tool was selected,
which agent delegated the task,
what context was shared,
where retries happened,
and why execution paths changed.

Without this, production debugging becomes extremely difficult.

Another pattern we increasingly prefer is asynchronous orchestration.

Instead of tightly coupling agents synchronously, we now lean more toward:

queues,
events,
workflow engines,
and loosely coupled communication.

It improves:

resilience,
scalability,
retry handling,
and fault isolation.

Most importantly, it prevents one overloaded agent from slowing down the entire system.

What I increasingly believe is that the future of enterprise AI is not one giant super-agent.

Technical debt handling

Amit Kayal — Mon, 11 May 2026 13:15:12 +0000

Over the years, my opinion on technical debt has changed a lot. Earlier, I used to think technical debt meant bad engineering decisions.

Now I think differently.

In product companies, especially fast-moving SaaS and AI products, some level of technical debt is unavoidable. If teams try to make everything perfect from day one, they usually move too slowly.
The real problem is not technical debt.
The real problem is when nobody knows:

why the shortcut was taken
how long it can survive
what impact it will create later

Personally, I look at technical debt in 3 broad categories:

Strategic debt : Shortcuts taken consciously to move faster, validate ideas, or release quickly.
Operational debt: Things that slowly start hurting deployments, production stability, debugging, support effort, and developer productivity.
Architectural debt: This is the one that becomes dangerous over time. Scaling becomes harder, integrations become messy, releases become slower, and every new feature starts feeling more expensive to build.

I feel AI products make this even more complicated. In normal SaaS systems, debt usually impacts engineering speed. But in AI systems, technical debt can directly affect:

response quality
hallucination handling
latency
observability
model cost
evaluation consistency

And because AI systems are probabilistic, debugging becomes much harder compared to traditional software.

I’ve also seen SaaS platforms suffer heavily from invisible debt because of:

multi-tenant complexity
customer-specific customizations
integrations
deployment dependencies
security and compliance requirements

One weak architectural decision early on can create pain for years.

That’s why I personally prefer making technical debt visible and measurable instead of treating it as a future problem.

Some of the signals I usually watch:

deployment friction
rollback frequency
incident trends
onboarding difficulty for new engineers
release confidence
overall engineering velocity

One pattern I’ve noticed repeatedly:
When team size keeps increasing but delivery speed keeps dropping, technical debt is already affecting the organization.

Learnings while working with long-running AI agents

Amit Kayal — Mon, 11 May 2026 13:12:53 +0000

One of my biggest learnings while working with long-running AI agents is that logging and progress reporting are not optional features when the agent is tightly coupled with a UI — they are part of the product experience itself.

Initially, I used to think of logging mainly from a debugging or engineering perspective. But with agentic systems, especially long-running workflows involving multiple tools, reasoning steps, APIs, retries, or multi-agent coordination, I realized users experience “silence” very differently than traditional applications.
When an agent takes 30 seconds, 2 minutes, or longer without visible progress, users immediately start questioning:

Is the system stuck?
Did my request fail?
Is it doing the wrong thing?
Should I refresh or retry?

That uncertainty destroys trust very quickly.
I learned that users do not just want the final answer — they want confidence that the system is actively working toward the answer. Progress visibility creates psychological assurance. Even simple updates like:

“Analyzing uploaded documents…”
“Fetching data from CRM…”
“Generating recommendations…”
“Validating final response…”

dramatically improve user confidence and patience.
Another major realization was that long-running agents are fundamentally non-deterministic systems. Unlike traditional APIs, agents can:

take different execution paths,
loop through reasoning,
invoke tools dynamically,
retry failed steps,
or spend time resolving ambiguity.

Without structured logging and traceability, debugging becomes extremely difficult because the same input may not always produce the same internal execution path. Modern AI observability emphasize tracing tool calls, reasoning paths, latency, token usage, and execution flow because agent behavior is inherently complex and probabilistic.

I also learned that progress reporting is not only for users — it becomes equally important for engineering and operational visibility. Once agents move into production, observability helps teams identify:

where workflows slow down,
which tool calls fail,
why latency spikes happen,
and where hallucinations or execution deviations originate.

One practical lesson I learned is that UI-integrated agents should expose execution state intentionally, not dump raw logs. There is a difference between:

engineering telemetry,
operational traces,
and user-friendly progress communication.

Users need understandable milestones, while engineers need deep execution traces.
Another important learning was around perceived performance. In many cases, improving progress visibility improved user satisfaction more than reducing actual latency. A 90-second process with clear step-by-step reporting often feels faster and more reliable than a silent 40-second execution.

Today, I strongly believe that for long-running AI agents:

logging is part of reliability,
progress reporting is part of UX,
and observability is part of trust.

Building a Hybrid AWS Microservices Platform with API Gateway, Lambda, ECS, and Load Balancers

Amit Kayal — Mon, 20 Apr 2026 18:41:39 +0000

Building a Hybrid AWS Microservices Platform with API Gateway, Lambda, ECS, and Load Balancers

Introduction

When teams start splitting a large backend into smaller services, the first infrastructure question is usually not "How do we build a microservice?" but "How do we expose many different services safely, consistently, and without creating a networking mess?"

Our architecture provides a practical answer to that problem using a hybrid AWS design:

API Gateway as the front door
Lambda for lightweight serverless capabilities and supporting workflows
ECS Fargate for containerized business services
Internal load balancers for private service routing
Terraform for repeatable, staged infrastructure delivery

The important architectural idea is separation of concerns. Public access, authentication, routing, container execution, and service discovery are all handled by different layers. That keeps the platform easier to scale and much easier to evolve as the number of services grows.

The Core Pattern

At a high level, the platform follows this flow:

A client sends an HTTPS request to API Gateway.
API Gateway applies request-level controls such as API key enforcement, CORS behavior, and route matching.
The request is sent either to a Lambda-backed endpoint or to a private containerized service.
For ECS services, traffic goes through a VPC Link into internal load balancing.
The load balancer forwards the request to the correct ECS service based on path rules.
ECS Fargate runs one or more healthy tasks for that service and returns the response.

This gives a single API surface to consumers while allowing the backend implementation to vary by use case.

Why Combine Lambda and ECS?

A platform like this benefits from using both compute models rather than forcing every workload into one.

Lambda is a strong fit for:

lightweight request handlers
event-driven tasks
simple orchestration
platform support functions
endpoints that do not need a full container lifecycle

ECS Fargate is a better fit for:

long-lived HTTP microservices
containerized frameworks and dependencies
services that need more predictable runtime behavior
APIs that benefit from load balancing, health checks, and horizontal scaling

In our architecture, the design supports both. Some APIs are routed to Lambda-based services, while others are routed to ECS services defined through service configuration. That hybrid model is useful in real organizations because all services do not have the same runtime needs.

A Three-Stage Infrastructure Model

One of the strongest ideas in our architecture is the staged Terraform layout. Instead of deploying everything together, the infrastructure is split into three layers.

Stage 1: Networking

The first stage establishes the network foundation:

VPC selection or creation
public and private subnet discovery or provisioning
internal Network Load Balancer
internal Application Load Balancer
VPC Link for API Gateway
ECS task security group
ALB log storage and network observability components

This stage is intentionally infrastructure-only. No application services are deployed here.

Stage 2: Compute

The second stage provisions the actual execution environment:

ECS cluster on Fargate
ECR repositories for service images
target groups per service
ALB listener and listener rules
ECS service definitions
CloudWatch log groups
Lambda functions used by the platform

This stage consumes outputs from the networking stage so the compute layer never hardcodes network assumptions in its own design.

Stage 3: API Gateways

The third stage exposes services through API Gateway:

a public API for internet-facing consumption
a private API for VPC-only access
route creation from service metadata
VPC Link integrations for containerized services
Lambda proxy integrations for Lambda-backed services
API keys, usage plans, and stage configuration

This split is operationally important. Teams can change routing without rebuilding networking, and they can add services without redesigning the entire platform.

The Request Path for ECS Services

For containerized microservices, the implementation follows a private ingress model.

The path is:

Client -> API Gateway -> VPC Link -> internal NLB -> internal ALB -> ECS service -> ECS task

That may look like one hop too many at first, but each layer has a purpose.

API Gateway

API Gateway is the public control plane. It handles:

TLS termination at the edge
route exposure
API key enforcement
request and header mapping
CORS handling
stage-based deployment

It gives consumers a stable API contract while keeping the backend private.

Why a VPC Link Is Used

ECS services are not exposed directly to the internet. Instead, API Gateway connects privately into the VPC using a VPC Link. That allows the public API layer to reach internal services without making the services themselves public.

This is a strong security pattern because the application runtime stays inside the VPC, but consumers still get a clean managed API endpoint.

Why the Repository Uses Both NLB and ALB

A useful implementation detail in our architecture is that the VPC Link targets an internal Network Load Balancer, and that NLB forwards to an internal Application Load Balancer.

This arrangement provides two separate benefits:

The NLB is used as the stable target for the API Gateway VPC Link.
The ALB performs path-based routing to the actual microservices.

The ALB is what makes many ECS services practical behind one internal entry point. Each service gets its own listener rule and target group, so the platform can route based on URL path rather than provisioning a separate load balancer per service.

How Load Balancing Works

The load-balancing model is service-oriented.

Each ECS microservice contributes:

a base API path
an ALB path pattern
a listener rule priority
a container port
a health check definition

From that metadata, Terraform creates:

one target group per service
one listener rule per service
one ECS service per service

This means the routing layer is not manually duplicated for every new microservice. The service declares its path and runtime settings, and the platform generates the infrastructure around it.

Target Groups

Each target group points to ECS tasks using IP targets. That is the correct choice for Fargate because tasks run with their own elastic networking interfaces rather than on shared EC2 hosts.

The target groups in this repository also use application-level health checks. A task is considered healthy only when its service endpoint responds successfully on the configured health path.

That matters because container startup is not the same as application readiness. A service may be running from ECS's perspective but still not ready to receive traffic.

Listener Rules

The ALB listener is configured once, and each service gets a path-based rule. For example, a service under a quoting path can be matched independently from a service under a product-pricing path.

This keeps the routing layer centralized and avoids deploying a dedicated ALB per service, which would become expensive and operationally noisy as the platform grows.

Health Checks and Traffic Protection

The repository uses health checks in multiple places:

API health endpoints at the application level
ALB target group health checks
ECS service health grace periods
container health checks inside the task definition

That layered approach improves resilience:

unhealthy tasks are removed from target groups
ECS replaces failed tasks
API Gateway continues to route through the same private entry point

The result is a platform that can recover from instance-level failures without changing the public API contract.

How ECS Is Structured

The ECS side of the platform is built for repeatability rather than one-off service definitions.

ECS Cluster

The platform provisions a shared ECS cluster per environment. That allows multiple microservices to run within the same operational boundary while still being isolated at the task and service level.

The cluster uses Fargate, which removes the need to manage EC2 worker nodes. This simplifies operations significantly:

no patching of container hosts
no cluster capacity management at the instance level
easier scaling by task count

Reusable ECS Service Module

Instead of defining each ECS service from scratch, the repository uses a reusable Terraform module for service deployment.

That module is responsible for:

task definition creation
container logging configuration
IAM role wiring
ECS service creation
target group attachment
subnet and security group placement
optional capacity provider strategy

This is a strong platform choice. It makes service onboarding consistent and reduces drift between services.

Task Definitions

Each service runs as a Fargate task with:

a named container image from ECR
CPU and memory settings
environment variables
a health check command
CloudWatch logging

The repository also includes support for an additional X-Ray sidecar container in the task definition pattern, which is useful for distributed tracing in a microservice environment.

Network Mode

Tasks run with awsvpc networking, which gives each task its own network interface and private IP. This is the standard model for ECS on Fargate and is what allows ALB target groups to use IP mode cleanly.

Subnet and Security Group Design

This repository supports both existing/default VPC usage and a more segmented custom VPC model.

That flexibility matters because many teams start in a default-VPC or dev-friendly setup and later move to stricter network isolation for staging and production.

Subnet Placement

The network layer discovers public and private subnets where available. In a custom VPC, the design supports proper private subnet deployment. In a simpler default VPC setup, the platform can fall back to available public subnets when private ones are not present.

This is an important operational nuance:

development environments often optimize for simplicity
higher environments usually optimize for stricter isolation

The repository is built to handle both.

Security Groups

The security model follows least-privilege intent:

ECS tasks accept application traffic from the internal load-balancing layer
services are not directly internet-facing
API Gateway reaches backend services through private network integration

This keeps the application tier out of direct public exposure while still allowing a public API facade.

Config-Driven Service Onboarding

One of the most scalable ideas in our architecture is that services are registered through configuration rather than by handcrafting infrastructure every time.

There is a master service registry that lists enabled services per environment, and each service provides its own deployment metadata, including:

service identity
container port
desired task count
CPU and memory
API base path
ALB path pattern
listener priority
health check behavior
logging retention
autoscaling preferences

This creates a platform model rather than a collection of unrelated microservices.

Adding a new service becomes a repeatable process:

Create the service.
Define its configuration.
Register it in the service catalog.
Build and publish the image.
Apply Terraform stages.

That is much easier to maintain than cloning infrastructure blocks over and over.

Container Delivery with ECR

For ECS workloads, the container supply chain is straightforward:

Build the service image.
Push it to an ECR repository.
Reference the tagged image in the ECS task definition.
Update the ECS service to roll out the new task definition.

Our platform provisions one ECR repository per service, with image scanning enabled. That is a good baseline for a microservices platform because it keeps artifacts separated by service while still following a common naming convention.

There is also an explicit deployment phase between infrastructure provisioning and API exposure where container images are built and pushed. That is a practical real-world step many diagrams omit, but it is essential because ECS cannot run a service until the image exists in the registry.

How Lambda Fits into the Platform

Lambda is used here as a first-class platform option, not as an afterthought.

There are two useful Lambda patterns in our architecture.

1. Lambda as an API Backend

Some services can be exposed through API Gateway using Lambda proxy integration. This is ideal for capabilities that are naturally event-driven, lightweight, or operationally simpler as functions than as always-on containers.

In this model:

API Gateway owns the route
Lambda executes the business logic
API Gateway returns the Lambda response directly

This avoids unnecessary load-balancer and container overhead for smaller workloads.

2. Lambda as a Platform Support Function

Our architecture also provisions Lambda functions that support the overall platform, such as authentication-related or onboarding-related workflows.

This is a smart use of Lambda in a hybrid platform because not every supporting concern needs to run inside ECS.

Authentication and API Protection

Our architecture clearly treats API protection as an API Gateway concern.

The current public API implementation enforces API key usage through API Gateway methods, API keys, and usage plans. The codebase also provisions a supporting API key validation Lambda function and related permissions, which shows the platform is designed to accommodate Lambda-based validation flows where needed.

From a blog perspective, the important architectural takeaway is this:

keep authentication and traffic governance at the gateway layer
keep service containers focused on business logic
keep private workloads private

That separation keeps the platform easier to secure and easier to reason about.

Public and Private API Models

Another strength of our architecture is that it supports both public and private APIs.

Public API

The public API is intended for internet-facing access. It handles:

external client access
API keys and usage plans
CORS behavior
Lambda and ECS route exposure

Private API

The private API is intended for internal or VPC-scoped access. It is useful when services should only be reachable from trusted network boundaries such as internal AWS workloads, integration environments, or enterprise connectivity paths.

This split is helpful when some capabilities should be public and others should remain internal even though they share the same service platform underneath.

Observability and Operations

A microservices platform is only as good as its operational visibility.

Our architecture includes observability at several levels:

CloudWatch log groups for ECS services
CloudWatch logs for Lambda functions
API Gateway stage logging
ALB logging support
VPC flow logging
X-Ray-friendly task patterns

That combination helps answer the most common production questions:

Did the request reach the gateway?
Was it routed to the right backend?
Was the target healthy?
Did the service fail or time out?
Was the problem in networking, routing, or application logic?

Without that layered visibility, hybrid platforms become difficult to troubleshoot.

Scaling Characteristics

This architecture scales well because each layer can evolve somewhat independently.

API Layer Scaling

API Gateway absorbs public traffic without requiring the backend to manage edge-facing concerns directly.

ECS Scaling

ECS services scale by task count. Each service can define:

desired count
minimum and maximum capacity
CPU and memory sizing
autoscaling thresholds

That means heavily used services can scale out without affecting lighter services.

Platform Growth

As more services are added, the platform does not need a new ingress pattern each time. The same path-based routing model continues to work as long as route definitions and listener priorities stay clean.

Alignment with AWS Well-Architected Best Practices

This architecture also aligns well with AWS best-practice design principles, especially the AWS Well-Architected mindset.

Operational Excellence

We have structured the platform so that it is operated as a system rather than as a collection of one-off deployments.

This is reflected in:

staged Terraform deployments for clearer ownership and safer changes
configuration-driven service onboarding
consistent ECS service patterns through reusable modules
standardized logging and deployment workflows

This reduces manual drift and makes operational changes more repeatable.

Security

Security is addressed through layered controls rather than a single protection point.

We have adhered to good AWS security practices by:

placing ECS services behind private networking rather than exposing them directly
using API Gateway as the controlled ingress layer
applying API-level protection at the gateway
using security groups to limit east-west traffic
supporting encrypted log and storage patterns
separating public access from internal service routing

This follows the AWS principle of strong boundaries, least privilege, and defense in depth.

Reliability

Reliability comes from designing for failure at the service and routing layers.

We have incorporated that through:

multi-AZ subnet placement
load balancer health checks
ECS task replacement behavior
target group isolation per service
decoupled gateway and backend layers
staged infrastructure dependencies with clear outputs between layers

This means a failing task or unhealthy target does not require the API surface itself to change.

Performance Efficiency

The architecture chooses the right compute model for the right workload.

That is an AWS best practice because it avoids treating all traffic the same.

Examples include:

Lambda for lighter, event-oriented, or supporting workflows
ECS Fargate for containerized services that need steady HTTP handling
ALB path-based routing for efficient multi-service consolidation
service-specific CPU, memory, and scaling settings

This lets us tune services independently instead of overprovisioning everything at the platform level.

Cost Optimization

Cost optimization is also visible in the design choices.

We are not multiplying infrastructure unnecessarily. Instead, the architecture encourages shared but controlled platform components:

one API layer for many services
one internal routing layer for many ECS workloads
shared ECS cluster patterns per environment
service-level scaling instead of blanket scaling
support for Fargate and optional capacity-provider strategies where appropriate

That is much closer to AWS best practice than provisioning separate ingress and compute stacks for every small service.

Sustainability and Maintainability

Even when sustainability is not called out directly, maintainable designs usually consume fewer engineering and infrastructure resources over time.

The architecture helps here by:

reducing duplicated infrastructure definitions
making service onboarding metadata-driven
encouraging reuse of shared platform components
keeping the public contract stable while backend services evolve

That leads to lower long-term complexity, which is a practical form of architectural efficiency.

Why This Pattern Works Well

This AWS pattern is effective because it balances standardization with flexibility.

It standardizes:

deployment stages
ingress architecture
service registration
load-balancer behavior
logging and health checks
ECS service creation

It stays flexible by allowing:

Lambda-backed endpoints
ECS-backed endpoints
public and private APIs
different service-level scaling and runtime settings
multiple environments with different networking strategies

That is exactly what a growing microservices platform needs.

Practical Implementation Advice

If you want to implement a similar architecture, a good sequence is:

Build the networking foundation first.
Keep all service backends private.
Put API Gateway in front of everything external.
Use ECS Fargate for containerized APIs that benefit from long-lived service behavior.
Use Lambda for support functions and lightweight endpoints.
Register services through metadata, not repetitive infrastructure definitions.
Use path-based ALB routing so many services can share one internal ingress layer.
Add strong health checks and centralized logs before traffic grows.

The key is not just choosing AWS services, but assigning each AWS service a clear responsibility.

Conclusion

Our architecture demonstrates a mature way to implement Lambda and ECS-based microservices through API Gateway without exposing backend services directly.

The architecture uses:

staged Terraform for separation of concerns
API Gateway as the public and private API facade
Lambda where serverless execution makes sense
ECS Fargate for containerized microservices
NLB and ALB together for private, path-aware routing
config-driven onboarding for scale

For teams building an enterprise microservices platform, this is a strong pattern because it supports security, operational clarity, and service growth without forcing every workload into the same runtime model.

Most importantly, it turns infrastructure into a reusable platform. Once that platform is in place, adding the next service becomes much easier than adding the first one.

Lessons Learned

Keeping API Gateway as the front door and backend services private makes the architecture easier to secure and easier to evolve.
Using both Lambda and ECS is more practical than forcing every use case into a single compute model.
Path-based routing through shared internal load balancing scales better than creating isolated ingress infrastructure for every service.
Service onboarding becomes significantly easier when routing, health checks, scaling, and runtime settings are driven by configuration.
Health checks, logging, and observability need to be designed from the beginning; adding them later is much harder in a distributed system.
A staged infrastructure model reduces operational risk because networking, compute, and API exposure can be changed independently.
Standardizing platform patterns early saves substantial effort as the number of microservices grows.

Building a Practical Lambda Capacity Provider Platform: Lessons Learned from Warm Pools, Version Hygiene, and CI/CD Reality

Amit Kayal — Mon, 20 Apr 2026 18:25:06 +0000

Building a Practical Lambda Capacity Provider Platform: Lessons Learned from Warm Pools, Version Hygiene, and CI/CD Reality

There is a big difference between a slide-deck architecture and an operating system you can trust on a Monday morning.

This implementation captures that difference well. On paper, the idea is simple: create a shared AWS Lambda Managed Instances capacity provider, run latency-sensitive workloads on ARM64, keep the pool warm with EventBridge, prune old Lambda versions before they become operational debt, and wrap the whole thing in a GitHub Actions plus CodeBuild delivery model. In practice, each of those choices changes how you think about performance, cost, blast radius, and developer discipline.

What follows is not a generic cloud post. It is the kind of write-up you produce after actually building and living with the system.

The Real Problem We Were Solving

Traditional Lambda is excellent when you want abstraction and convenience. It becomes less elegant when your workload is sensitive to startup time, carries heavier dependencies, or needs more predictable execution behavior under bursty load.

That is where a Lambda capacity provider changes the discussion.

In this implementation, the platform is built around a shared aws_lambda_capacity_provider that uses ARM64 Graviton instances and auto scaling. The core idea is straightforward: instead of leaving execution placement entirely to the default Lambda fleet, we deliberately provide a managed compute pool that multiple functions can share. That gives us more control over cost-performance characteristics and lets us design around cold-start pain rather than merely complain about it.

The choice is visible in the Terraform:

The provider runs on arm64
Allowed instance types are constrained to m6g.large, m6g.xlarge, m7g.large, and m7g.xlarge
Scaling is set to Auto
The maximum pool ceiling is set to 64 vCPU
The capacity provider is placed in the default VPC, with unsupported Availability Zones filtered out

That last point matters more than it first appears. The code explicitly excludes unsupported AZs such as us-east-1e, which is a good example of operational maturity: the happy path is not enough when the service itself has placement constraints.

How We Actually Created the Capacity Provider

One thing I wanted this platform to avoid was "concept architecture" with no implementation backbone. So the capacity provider here is not described abstractly. It is provisioned directly in Terraform and wired into the Lambda lifecycle in a fairly intentional way.

The build starts in terraform_file/agent_core_sync_cp.tf.

First, the capacity provider itself is created with aws_lambda_capacity_provider. The naming pattern ties it to the service and environment, which is the right instinct for multi-environment operation. The provider is tagged as shared compute for agent workloads, which matters later for discoverability and platform governance.

Second, the provider is placed inside the default VPC, but not blindly. In terraform_file/data.tf, the code:

discovers the default VPC
fetches the default subnets
inspects subnet Availability Zones one by one
excludes unsupported zones such as us-east-1e
optionally caps how many subnets are used

This is a subtle but important design choice. Lambda Managed Instances often create one placement footprint per subnet or AZ. If you do not control subnet spread, you can end up creating more infrastructure surface area than you intended.

Third, the provider uses a dedicated security group rather than inheriting something vague and accidental. The current implementation keeps outbound traffic fully open and allows inbound HTTPS. That is permissive, but it is at least explicit and repeatable. Early-stage platforms benefit from that kind of clarity.

Fourth, the capacity provider gets its own operator role through AWSLambdaManagedEC2ResourceOperator. That is a critical detail. Capacity providers are not just Lambda resources; they need AWS to manage the EC2-backed execution infrastructure on your behalf. If you miss that role, the platform does not really exist no matter how nice your Terraform looks.

Fifth, the instance requirements are opinionated. The code forces arm64 and narrows the fleet to supported Graviton M-family instance types. That is one of the better engineering decisions in this implementation because it converts an architectural preference into an enforceable runtime rule.

Finally, the Lambda function is attached to the capacity provider in terraform_file/lambda_clm_router_agent.tf through capacity_provider_config. That is where the abstraction becomes real. We are not just provisioning a pool and hoping someone uses it later. We are explicitly binding a published Lambda to that pool and tuning:

memory GiB per vCPU
max concurrency per execution environment
ARM64 runtime alignment
published versioning through Lambda aliases

That is the full loop: provision shared compute, constrain placement, grant AWS the operator role it needs, attach live functions to the pool, and then manage the resulting version sprawl with automation. That is what makes this feel like a platform artifact rather than a loose Terraform experiment.

Lesson 1: A Capacity Provider Is Not a Tuning Knob. It Is an Operating Model.

Teams often talk about capacity providers as if they are just a performance optimization. That framing is too shallow.

The moment you move Lambda onto managed instances, you are no longer only buying faster startup. You are adopting a new operating model with very clear implications:

You now care about instance family compatibility
You need to think about subnet strategy and AZ support
You have to reason about pool scaling ceilings, concurrency, and memory per vCPU
You are effectively blending serverless ergonomics with infrastructure accountability

This implementation shows that transition clearly. The CLM router Lambda is not just declared with a runtime and handler. It is attached to the shared capacity provider and explicitly tuned with:

execution_environment_memory_gib_per_vcpu
per_execution_environment_max_concurrency
publish = true
architectures = ["arm64"]

That is the tell. Once we start specifying how execution environments should behave, we are no longer simply "deploying a Lambda." We are shaping compute economics.

The practical lesson here is simple: if you adopt Lambda Managed Instances, treat it like platform engineering, not like a runtime checkbox.

Lesson 2: ARM64 Delivers Real Value, but Only if You Respect Service Constraints

One of the strongest decisions in this implementation is the bias toward Graviton. For Python-heavy agent workloads, ARM64 is usually the right default. The economics are better, and the performance-per-dollar story is often compelling.

But there is an important nuance that the Terraform comments correctly capture: not every EC2 family you might expect is supported in the way you assume. This implementation explicitly avoids unsupported combinations and narrows the fleet to supported M-family Graviton instances.

That is a good lesson in cloud architecture generally: cloud products market flexibility, but production systems survive on constraint management.

The teams that do well with modern AWS services are not the ones that assume every SKU works. They are the ones that encode the service's real boundaries in Terraform so no one has to rediscover them during an incident window.

Lesson 3: Warmup Is Not a Hack. It Is a Deliberate Control Loop.

There is a tendency in engineering circles to treat "warming" as a slightly embarrassing workaround. I think that is the wrong mindset.

This implementation schedules the CLM router Lambda every five minutes through EventBridge. The handler itself is intentionally lightweight and effectively acts as a keep-alive mechanism. That is not laziness. It is an explicit decision to keep the shared pool alive for latency-sensitive traffic.

More specifically, the warmer exists to reduce the probability that the capacity provider has to spin up fresh managed instance capacity for a new invocation path after a quiet period. That is the practical point of the EventBridge rule in terraform_file/eventbridge_cp_arm.tf. By invoking the Lambda on a steady rate(5 minutes) schedule, the platform keeps the execution path warm enough that the shared capacity provider is less likely to fall all the way back to a cold, scale-from-zero posture right before a real request arrives.

The important insight is this: once you care about cold-start predictability, you need a control loop.

That control loop can be:

Provisioned concurrency
Scheduled warmers
Request shaping
A shared managed instance pool

In this design, the team chose scheduled warm invocation plus a shared capacity provider. That is a sensible middle ground. It is cheaper and simpler than overcommitting always-on infrastructure, while still materially reducing the first-hit penalty.

In plain English: the EventBridge warmer is being used here so the capacity provider does not need to spin up a brand-new server footprint every time traffic reappears after idle time. For interactive or latency-sensitive agent workloads, that is a very practical optimization.

The strategic lesson is that warmup should be measured against business latency, not ideological purity. If a five-minute EventBridge schedule protects user experience and keeps cost acceptable, it is doing its job.

Lesson 4: Shared Pools Create Efficiency, but They Also Create Coupling

The capacity provider here is intentionally shared across platform agents and automation services. That is the right move early in a platform journey because it improves utilization and prevents every Lambda from inventing its own isolated infrastructure story.

But shared pools always introduce two forms of coupling:

Technical coupling, because multiple workloads compete for the same execution substrate
Organizational coupling, because one team's deployment patterns can affect another team's cost and performance envelope

That is why the concurrency controls here matter. The CLM router function uses a per-execution-environment concurrency setting, and the environment-specific .tfvars files pin that concurrency to 4. That is more than a performance number. It is a fairness policy.

If I were advising a platform team scaling this pattern, I would say this clearly: shared capacity providers are excellent, but they need quota thinking from day one. Otherwise the first successful workload becomes the first noisy neighbor.

Lesson 5: If You Publish Versions Aggressively, You Need Lifecycle Hygiene on Day One

This implementation makes another good call: the Lambda functions are published, aliased, and then cleaned up with an automated version pruner.

That matters because version sprawl is one of those quiet operational problems that teams ignore until it becomes annoying enough to disrupt deployments. Published versions accumulate quickly when CI/CD is active. If you do not manage them, you eventually pay in clutter, confusion, or hard service limits.

The lambda_version_pruner implementation is stronger than a simplistic cleanup script because it preserves what actually matters:

It scans all Lambda functions
It filters only functions associated with the target capacity provider
It lists all aliases and protects aliased versions
It keeps the latest N published versions
It deletes everything older that is neither current nor aliased

This is exactly the kind of automation mature teams invest in. Not glamorous. Very valuable.

There is also an understated platform principle here: rollback is not just about keeping artifacts. It is about keeping the right artifacts. By preserving aliased versions, the pruner respects deployment intent rather than blindly optimizing for tidiness.

There is also a more practical capacity-provider reason for doing this, and it deserves to be stated directly.

When you run a shared Lambda Managed Instances pool, you want the platform to spend its effort on the versions that are actually serving traffic, warming correctly, or remaining available for safe rollback. If old published versions keep accumulating forever, three unhealthy things tend to happen:

operators lose clarity on which versions are still meaningful
rollback and alias management become noisier than they should be
the shared platform carries more deployment residue than useful runtime intent

Strictly speaking, deleting old Lambda versions does not magically increase CPU on the capacity provider. What it does do is improve platform hygiene around the shared pool. It ensures that the versions attached to aliases, warmup patterns, and deployment workflows remain deliberate and limited. In other words, it improves capacity-provider utilization indirectly by reducing version sprawl around the workloads that consume that shared capacity.

That matters in real operations. The healthier the deployment surface is, the easier it is to reason about what is warming, what is active, what can be rolled back, and what should no longer influence the platform at all.

So the version pruner is not just a cleanup utility. It is part of making the shared capacity provider operationally efficient. Not by adding raw compute, but by reducing noise, protecting the versions that matter, and keeping the platform focused on live execution paths instead of historical leftovers.

Lesson 6: GitHub Actions Should Orchestrate. CodeBuild Should Execute.

Architecturally, the CI/CD model here is sensible.

GitHub Actions is used as the control plane:

branch-based triggering
security scanning
environment selection
AWS credential injection
build orchestration

AWS CodeBuild is used as the execution plane:

Terraform install
terraform init
terraform validate
terraform plan
terraform apply

I like this split. It keeps GitHub Actions lightweight and makes AWS the place where the actual infrastructure mutation happens. That usually gives better access control, cleaner auditability, and fewer surprises around long-running plan or apply steps.

The buildspecs pin Terraform 1.12.2, install the CLI explicitly, and then execute plan/apply flows with environment-specific variable files. That is exactly the kind of boring repeatability you want in infrastructure delivery.

This is one of the most practical lessons from the implementation: do not force GitHub Actions to be your full deployment runtime if AWS-native execution gives you better control.

Lesson 7: CI/CD Maturity Is Not About Having a Pipeline. It Is About Where the Gates Actually Are.

The implementation also reveals a harder truth: CI/CD design is won or lost not by YAML volume, but by trigger discipline.

There are some good instincts here:

Dev deployment is chained off a successful security workflow
Security scanning runs on push and PR for dev
PR security review is scoped only to actual code and infrastructure changes
Environment-specific secrets are used for AWS access

That said, the current implementation also shows the kinds of issues every fast-moving team encounters:

The dev deploy workflow is triggered by Security Checks (Push), not by a broader quality gate such as tests plus security plus static analysis
The QA workflow is currently triggered on pull_request to qa, yet it also includes an apply stage, which is a risky combination
The sanity workflow references a different CodeBuild project naming pattern, which looks like copy-forward drift from another implementation
One dev apply step mixes generic and environment-specific secrets in a way that deserves tightening

This is not a criticism of the team. It is actually the most authentic part of the system.

Real pipelines evolve through reuse, renaming, urgency, and partial migration. The useful engineering habit is not pretending they are pristine. It is recognizing that pipeline drift is itself a production concern.

My blunt lesson here is this: CI/CD is software. It needs the same review rigor as application code.

Lesson 8: Documentation Drift Is a Reliability Signal

The README here is ambitious and useful, but parts of it clearly describe a broader or earlier architecture than the exact files currently present. That mismatch is more important than most teams realize.

When documentation and implementation diverge, three things happen:

new engineers learn the wrong system
reviewers approve changes with outdated mental models
incidents take longer to resolve because operators trust stale diagrams

One of the best engineering habits is to treat documentation drift as an operational bug, not as a cosmetic issue.

This implementation makes that case well. The code is the source of truth. The docs are directionally strong, but some names, workflow descriptions, and file references have clearly moved over time. That is normal. What matters is catching it before the next engineer builds decisions on old assumptions.

Lesson 9: The Default VPC Is Fine for Speed, but It Should Be a Conscious Temporary Convenience

The Terraform intentionally uses the default VPC and default subnets, then layers in filtering and a custom security group. For early velocity, that is an acceptable choice. It removes friction and makes the first deployment much easier.

But teams should be honest about the tradeoff.

Using the default VPC accelerates setup. It does not provide the same clarity, segmentation, or policy hygiene that a dedicated workload VPC eventually should. The inbound HTTPS rule from 0.0.0.0/0 is another example of where a practical early-stage decision should later be revisited with a more opinionated security posture.

My view is simple: default VPC usage is fine when it is a speed decision. It becomes dangerous when it silently hardens into architecture.

Lesson 10: Least Privilege Usually Loses the First Battle. Do Not Let It Lose the War.

The Lambda IAM policy for the router function is broad. Very broad.

That is common when a platform team is trying to unblock integration work quickly across S3, SQS, SNS, DynamoDB, Bedrock, AppSync, logs, X-Ray, and secrets. The version pruner is noticeably tighter, which is encouraging. But the broader pattern remains familiar: the first version of a system usually over-grants.

The lesson is not "never do that." The lesson is "know when you are doing it, and schedule the hardening work while the platform is still comprehensible."

Security debt compounds. The longer a wide-open policy survives, the more invisible it becomes.

What This Repo Gets Right

If I strip away the drift and focus on the platform instincts, this implementation gets a lot right:

It treats capacity provider infrastructure as shared platform capability, not one-off function plumbing
It optimizes for ARM64 economics instead of defaulting to x86 out of habit
It acknowledges cold starts as a business problem and addresses them operationally
It preserves rollback safety with aliases while still pruning version sprawl
It separates orchestration from execution in CI/CD
It encodes AWS service constraints in Terraform comments and defaults, which reduces tribal knowledge

That is a strong foundation.

What I Would Improve Next

If I were turning this into the next version of a production-grade internal platform, I would prioritize the following:

Tighten naming consistency across the implementation.
The capacity provider name appears in slightly different forms across resources. That is how automation misses its target. Shared naming locals should eliminate this class of error.
Make QA and production promotion rules stricter.
A PR-triggered apply path should be removed. Plan on PR, apply on protected branch or approved environment gate is the cleaner model.
Run Terraform from a single explicit working directory.
The current layout places Terraform under terraform_file/, while some buildspec commands read like root-level execution. That ambiguity should be eliminated.
Move from broad IAM toward intent-based policies.
Especially for the router Lambda, policy scope should narrow as the workload stabilizes.
Revisit networking posture.
The default VPC is fine for speed; a dedicated VPC model is better for longevity, auditability, and controlled ingress.
Add stronger deployment quality gates.
Security review is useful, but infrastructure promotion should also hang off validation, tests, linting, and explicit approval where appropriate.
Add platform observability as code.
CloudWatch alarms, dashboarding, and cost visibility for the capacity provider should be treated as first-class Terraform resources, not follow-up tasks.

The Bigger Technical Lesson

The biggest takeaway from this implementation is not about Lambda specifically.

It is about how modern platform teams should build.

We should absolutely chase better cost-performance curves. We should use managed primitives aggressively. We should automate the boring work. But we also need the discipline to encode what we learn while the system is still small enough to reason about.

What makes this useful is that it shows both halves of real engineering:

the architectural intent
the implementation scars

That combination is where credible engineering judgment comes from.

Anyone can present a clean target state. The harder and more useful skill is building systems that survive contact with deployment friction, service constraints, naming drift, and operational reality.

That is what this implementation is doing. And that is why the lessons here matter.

Closing Thought

Capacity providers, warmers, version pruning, and GitHub-driven delivery are not separate topics. They are all answers to the same technical question:

How do we make cloud systems faster, cheaper, safer, and more repeatable without turning every application team into a specialized infrastructure group?

In this implementation, the answer was to centralize the hard platform decisions, automate the hygiene, keep the runtime warm where it matters, and stay honest about the places where the system still needs tightening.

That is not just good infrastructure work.

That is good engineering practice.

Lessons I learned building a memory-aware agent with Amazon Bedrock AgentCore Runtime

Amit Kayal — Mon, 20 Apr 2026 18:10:16 +0000

Lessons I learned building a memory-aware agent with Amazon Bedrock AgentCore Runtime

When I started building an agent with Amazon Bedrock AgentCore Runtime, I thought the difficult parts would be model selection, tool wiring, and deployment. Those certainly mattered, but the part that shaped the quality of the agent most was memory.

The first version of the agent could answer single prompts well enough, but it did not behave like a real multi-turn system. Follow-up questions were brittle. The agent lost short-range intent. Tool usage worked, but only within the narrow boundaries of the current prompt. As soon as the conversation depended on what happened one or two turns earlier, the system started to feel less like an agent and more like a stateless inference endpoint.

That experience changed how I approached the design. I stopped thinking about memory as a convenience feature and started treating it as part of the runtime architecture itself. This article is a distillation of the most important lessons I learned while building a short-term-memory-aware agent with Amazon Bedrock AgentCore Runtime and Strands.

Lesson 1: An agent is not really multi-turn until memory is part of the lifecycle

One of the first things I learned is that conversational continuity does not emerge automatically just because the application calls the same runtime repeatedly.

Without short-term memory, the agent only sees the current prompt unless the application keeps reconstructing and replaying history manually. That creates several problems:

previous instructions are easy to lose,
tool chains become fragile across turns,
users have to restate identifiers and intent,
the system becomes increasingly prompt-shaped rather than interaction-shaped.

What became clear to me is that short-term memory is not about storing everything forever. It is about preserving enough recent state for the current conversation to remain coherent.

That distinction matters. I was not trying to build a knowledge base or semantic fact store. I was trying to answer a simpler question: how do I help the agent remember what we were just doing?

Once I framed the problem that way, the architecture became much clearer.

Lesson 2: The cleanest pattern is explicit memory, not implicit transcript magic

Another lesson I learned quickly is that I did not want memory to be hidden behind vague runtime behavior. I wanted the agent code to make memory use explicit:

where memory comes from,
when it is read,
when it is written,
which user it belongs to,
which conversation it belongs to.

That led me to a pattern built around MemoryClient and hooks.

Instead of treating memory like a passive transcript that somehow appears at the edge of the request, I found it much more reliable to think about it as a lifecycle-managed dependency:

create a short-term memory resource,
pass the memory identity into the runtime,
read recent turns when the agent initializes,
write new messages as events when the conversation changes.

The important shift for me was this: memory worked best when it was part of the agent object model, not just part of request handling glue code.

Lesson 3: Hooks are where memory belongs

This was probably the biggest implementation insight.

Once I had a Strands-based agent running inside AgentCore Runtime, I needed to decide where the memory logic should live. I could have put everything directly into the entrypoint and manually stitched together request parsing, history retrieval, message persistence, and prompt injection. That would have worked, but it would have made the agent lifecycle harder to reason about.

What worked better was using hooks tied to the agent lifecycle itself:

AgentInitializedEvent
MessageAddedEvent

That structure gave me a much cleaner mental model.

On initialization, the agent needs context before it reasons. That is the right moment to retrieve the most recent turns from memory and inject them into prompt context.

When a new message is added, the conversation state has changed. That is the right moment to persist the latest user or assistant message back into memory.

The core interaction looks like this:

recent = memory_client.get_last_k_turns(
    memory_id=memory_id,
    actor_id=actor_id,
    session_id=session_id,
    k=5,
)

memory_client.create_event(
    memory_id=memory_id,
    actor_id=actor_id,
    session_id=session_id,
    messages=[(text, role)],
)

What I like about this model is that it is deterministic.

memory load happens before reasoning,
memory write happens when conversation state changes,
both operations use the same identity boundaries,
the entrypoint stays focused on request extraction rather than conversation orchestration.

That made the system easier to debug, easier to extend, and much easier to explain.

Lesson 4: Identity is the real memory boundary

Before building this, I thought of memory mostly as a storage problem. In practice, I learned it is just as much an identity problem.

The two identifiers that mattered most were:

actor_id
session_id

This separation ended up being foundational.

Why `actor_id` matters

actor_id is the user boundary. If that identifier is unstable, absent, or inconsistent, memory quality degrades immediately.

What I learned is that a memory system is only as good as the application identity you feed into it. If the same user appears under multiple IDs, the agent cannot retrieve a coherent conversational history. If two users are accidentally mapped to the same identity, memory becomes unsafe.

So one of my strongest takeaways is that actor_id should always come from a stable authenticated user identity, not from an incidental client-generated value.

Why `session_id` matters

session_id turned out to be just as important. A single user does not have just one conversation. They may have multiple active threads:

one troubleshooting flow,
one transcript analysis request,
one abandoned conversation from earlier,
one brand-new task.

Without a session boundary, all of that collapses into one memory stream. The agent might technically “remember,” but it remembers too much of the wrong thing.

That was a key lesson for me: useful memory is not just preserved memory. It is correctly scoped memory.

Lesson 5: The agent should be rebuilt per request, but memory should persist across requests

This was an architectural point that became clearer as I implemented the runtime flow.

The Strands agent instance itself is created per request. That makes sense because each invocation carries request-specific state:

the current user prompt,
the active user identity,
the active conversation session,
the active tool and runtime context.

But memory should not behave like request-local state. Memory has to outlive the agent instance and remain keyed to the same user and conversation across invocations.

That split was important for me to internalize:

agent instance lifecycle is short,
conversation memory lifecycle is longer,
the link between them is established through state and hooks.

Once I started thinking in those terms, the design felt much more natural.

Lesson 6: Deployment is part of the memory design

I originally thought of deployment as a separate concern from conversational behavior. Building this agent convinced me that the two are tightly connected.

The runtime needs to know which memory resource it should use, but I did not want that decision hardcoded in application logic. The better pattern was to resolve the correct memory resource during deployment and pass that identity into the runtime as configuration.

In practice, that meant the runtime received environment-specific values such as:

AGENT_NAME=<agent-name>
MEMORY_ID=<memory-id>

That gave me a few benefits immediately:

the same application code could move across environments,
memory resources stayed aligned with environment boundaries,
the runtime remained configurable without source changes,
the control plane remained the primary place where resource binding happened.

One of the clearest lessons here is that memory should be treated like any other environment-bound infrastructure dependency. If it is not part of deployment, it tends to become a hidden assumption.

Lesson 7: Short-term memory and long-term memory solve different problems

I found it helpful to stop using the word “memory” as if it meant one thing.

Short-term memory answered the question:

"What was happening in this conversation recently?"

Long-term memory answers a different question:

"What durable information should the system remember beyond this immediate interaction?"

For the agent I was building, the short-term problem came first. I needed:

recent-turn continuity,
bounded replay,
session-scoped context,
predictable event retention.

I did not need semantic fact retrieval in the first phase. I did not need vector search for historical knowledge. I needed the agent to remain coherent across adjacent turns.

That was an important design simplification. It kept the first version of the memory architecture focused on event continuity instead of overextending into knowledge retrieval prematurely.

Lesson 8: Recent-turn replay should be bounded

Once I had memory retrieval working, the next question was how much of it to inject back into the agent context.

My lesson here was simple: more memory is not always better memory.

If too much prior conversation is replayed:

prompt size grows,
token cost grows,
stale context starts competing with the current task,
reasoning quality can actually decline.

I found the most practical pattern was to retrieve the last few turns and inject them into prompt context in a compact representation. In this design, that replay window was bounded at five turns.

That gave me a good balance:

enough recent context for continuity,
small enough context for predictable prompt growth,
simple enough formatting to inspect and debug.

This also reinforced another lesson: short-term memory should be operationally understandable. I want to know what context the model saw, not just trust that some opaque memory layer handled it correctly.

Lesson 9: Memory becomes more valuable when tools are involved

The agent I built was not just a conversational shell. It had tools, including domain-specific behavior such as transcript retrieval and AWS interactions.

That is where the value of short-term memory became even more obvious.

In a tool-using workflow, the user often does not repeat the full context every turn. They say things like:

"use the same meeting"
"what did the second speaker say?"
"now summarize that"
"check the S3 output from before"

Without memory, the agent has to reconstruct working state from a single prompt. With memory, the agent has a much better chance of preserving:

the active object under discussion,
the prior user instruction,
the last tool result,
the intended next step.

One of my strongest takeaways is that memory is not just a conversational improvement. It is a workflow improvement. It makes tool orchestration across turns materially more coherent.

Lesson 10: Failure modes need to be designed, not discovered in production

Building this also made me think much more carefully about degraded behavior.

If memory resolution fails and the runtime cannot find a memory resource, the agent may still run. That sounds convenient, but it also means the system may silently shift from stateful to stateless behavior.

That taught me to treat the following as first-class operational conditions:

memory enabled,
memory disabled,
memory load succeeded,
memory write succeeded,
memory resolution failed,
identity inputs were missing or malformed.

The same thing applies to identity mistakes.

If actor_id is unstable, memory becomes fragmented.

If session_id is reused incorrectly, unrelated conversations bleed into each other.

If replay windows grow without discipline, prompt quality degrades.

These are not edge cases. They are part of the normal operating surface of a memory-aware agent.

Lesson 11: Retention, privacy, and compliance show up earlier than expected

Short-term memory sounds lightweight, but it is still stored interaction data.

That means retention policy is not just a platform setting. It is part of the product design. While building this, I became much more aware that memory decisions quickly intersect with:

data handling policy,
privacy expectations,
deletion and retention requirements,
security review,
production observability.

The technical implementation can be elegant, but if these operational questions are not addressed early, the design will be incomplete.

Lesson 12: AgentCore became more useful to me when I treated it as a runtime system, not just a hosting target

This may be the broadest lesson of all.

At first, I thought of AgentCore Runtime mainly as the place where the agent container would run. But while building with memory, I started appreciating it more as a runtime environment with clear operational boundaries:

the runtime executes the agent,
the framework manages reasoning and tools,
the memory plane manages event continuity,
the deployment workflow binds the right resources together.

That view helped me move beyond “deploy a model wrapper in a container” toward “operate an agent system with state, identity, and lifecycle.”

For me, that was the real shift.

The technical pattern I would reuse

If I were building the same class of agent again, I would reuse the same high-level pattern:

Create a dedicated short-term memory resource.
Resolve the correct memory resource during deployment.
Pass memory identity into the runtime explicitly.
Build the agent per request with user and session state.
Load recent turns during agent initialization.
Persist new messages when they are added.
Keep replay windows bounded.
Treat actor_id and session_id as core correctness boundaries.

I would also keep the same mental model:

short-term memory is for continuity,
long-term memory is for durable recall,
hooks are the right place for memory orchestration,
deployment is part of memory architecture,
observability should make degraded memory behavior visible.

Closing thought

The biggest lesson I learned while building with Amazon Bedrock AgentCore Runtime is that memory is not something you sprinkle onto an agent once the rest of the system works. Memory changes the shape of the system.

It affects:

request lifecycle,
identity boundaries,
prompt construction,
deployment,
observability,
privacy,
and tool coherence across turns.

Once I accepted that, the architecture became much more disciplined. The agent became easier to reason about, easier to operate, and much more capable in real multi-turn interactions.

That is the lesson I would carry into any future AgentCore build: if the experience is meant to feel conversational, memory has to be designed as a first-class runtime concern from the beginning.

API Gateway as Websocket

Amit Kayal — Tue, 21 Jan 2025 07:49:42 +0000

API Gateway as websocket

API Gateway as WS Components

Websocket provides bidirectional session aware communication between caller and receiver and a crucial component for realtime application.

Setup API Gateway for WebSocket
- Create a WebSocket API in the Amazon API Gateway console or through IAC.
- Define the WebSocket API route selection expression. Routes here are simply like a bridge to connections e.g.,
  - $request.body.action.
  - Define the following WebSocket routes:
  - $connect: Triggered when a client establishes a connection.
  - $disconnect: Triggered when a client disconnects.
  - Custom routes, e.g., sendMessage, to handle specific actions.
Create an Integration with AWS Lambda
- For each route ($connect, $disconnect, custom routes), integrate a Lambda function to handle the respective logic.
- Use the Lambda function's handler to process:
  - $connect: Store the connection in DynamoDB.
  - $disconnect: Remove the connection from DynamoDB.
  - Custom routes: Process the message and forward it to SQS.
DynamoDB for Connection Management
- Create a DynamoDB table to store:
  - Connection ID (Primary Key).
  - Session ID or other metadata for grouping connections.
- This table allows tracking active WebSocket connections for broadcasting messages.
Configure SQS for Message Queue
- Use an SQS FIFO queue for guaranteed order and deduplication.
- Messages processed in Lambda (custom routes) are sent to SQS for downstream services.
IAM Roles and Permissions
- Assign an IAM role to the API Gateway to invoke the integrated Lambda functions.
- Grant Lambda permissions to read/write from DynamoDB and send messages to SQS.
Client Connection and Messaging
- Use WebSocket-compatible libraries (e.g., ws in Node.js or WebSocket API in browsers) to:
- Establish a WebSocket connection to the API Gateway endpoint.
- Send and receive messages using the WebSocket protocol.

Architecture of Websocket mechanism

WebSocket Client:
- Initiates WebSocket connection and communicates via send() and onmessage().
API Gateway (WebSocket API):
- Manages WebSocket connections and invokes Lambda functions for defined routes.
Route Integration (Lambda Functions):
Every route should have an integration. There are 3 types — Mock, HTTP and Lambda.
- $connect: Adds connection metadata to DynamoDB.
- $disconnect: Removes connection metadata from DynamoDB.
- $default route: selected when route cant be evaluated against message
- Custom Routes: Processes messages to invoke integration based on message content and forwards them to SQS.
DynamoDB:
- Maintains active connection records, including connectionId and associated metadata.
SQS FIFO Queue:
- Queues messages for downstream processing, ensuring delivery order and deduplication.
Downstream Services:
- Processes messages from SQS and performs actions like notifications, data updates, or storage.

Security

Authentication and Authorization

Custom Authorizer (Lambda Authorizer)
It can only be used for the $connect route.
- Create a Lambda Authorizer to validate custom tokens or headers sent during connection attempts.
- Example:
  - Validate a JWT token from an identity provider (e.g., Cognito, Auth0).
  - Check the token against allowed users or roles.
Amazon Cognito:
- Use Amazon Cognito for user authentication.
- Configure API Gateway to use Cognito to validate tokens in connection requests.
- Best suited for applications with user pools.

Secure WebSocket Connections

Always use the secure WebSocket protocol (wss://). API Gateway enforces HTTPS/TLS, ensuring encrypted communication.
Associate a custom domain with API Gateway WebSocket endpoint. We should AWS Certificate Manager (ACM) to manage SSL/TLS certificates.

IP Whitelisting and Blacklisting

IP Whitelisting and Blacklisting: We should Attach AWS WAF to API Gateway and Block/allow requests based on IP addresses or CIDR ranges. we should also use rate limit to protect from DDoS attack ### API Gateway Throttling
We can Set rate and burst limits on API Gateway routes to limit the number of connections per client.
We can create API keys and associate them with usage plan and then we Limit the number of allowed requests per API key

Environment-based Access Control:

We should always use distinct stages (e.g., dev, prod) and restrict connections to the production API through IP rules.

Tools to test

There are following tools which we can explore to test websocket.

Piesocket
Postman

S3 table & S3 Metadata table

Amit Kayal — Mon, 09 Dec 2024 18:26:23 +0000

Open table format and its architecture

OpenTable formats, such as Apache Iceberg, Apache Hudi, and Delta Lake, have gained popularity in the data analytics mainly because:

ACID Transactions: OpenTable formats (e.g., Apache Iceberg, Delta Lake) ensure reliable and consistent data updates, even with concurrent access.
Schema Evolution: They allow seamless updates to schemas without disrupting existing pipelines, simplifying data management. metadata tracks the changes to the dataset. The files held in the Data layer are captured by the metadata files held in the Metadata layer. As the files change, the metadata files attached to them track these changes.
Optimized Queries: Partitioning and indexing enable faster queries by scanning only relevant data, improving performance and cost-efficiency.
Time Travel: Users can access historical versions of data for debugging, compliance, or analytics.
Interoperability: These formats integrate seamlessly with big data tools like Spark, Flink, and Presto, making them versatile and widely adopted.

Open file format

S3 table

Key Features

Amazon S3 Table is optimized for analytics workloads. It is designed to continuously enhance query performance and reduce storage costs for tabular data. This solution looks very promising if you are working with LakeHouse architecture. It’s a new type of bucket that organizes tables as sub-resources.
A new bucket type s3 table has been introduced to support this. As liked any other aws resoyrce, it has ARN, can take resource policy and as an unique feature it has dedicated endpoint.

S3 Tables are intended explicitly for storing data in a tabular format, such as daily purchase transactions, streaming sensor data, or ad impressions. This data is organized into columns and rows like a database table.
Table buckets support storing tables in the Apache Iceberg format. You can query these tables using standard SQL in query engines that support Iceberg.
Read/write allowed on datafiles and metadata files. Delete and update not allowed to save data integrity.
Compatible query engines include Amazon Athena, Amazon Redshift, and Apache Spark.
S3 Table automatically performs maintenance tasks like compaction and snapshot management to optimize your tables for querying, including removing unreferenced files.
S3 Table offers access management for both table and bucket
Fully managed apache icebarg tables in S3
It supports automatic compaction of underlying files to improve query performance and tune then further for better latency.

S3 Table buckets namespace

Namespace logically groups related s3 table together and thus allowing us to have greater control based on namespace of s3 tables. It helps us for following:

logical segmentation of data and multi tenancy
- supporting of multi tenancy by having separate namespace. Supports compliance with data isolation requirements in regulated industries.
- separate tables based on application, project etc
prevent naming conflicts
- Each namespace acts like a "container," allowing tables with the same name in different namespaces without conflicts.
Better Access Control
- Policies can grant or restrict access to specific namespaces, ensuring data security and compliance. It also reduces the risk of unauthorized access to unrelated tables in the same bucket.
Easy data management
- Makes our life easier to query, update, or delete related tables in bulk.
- Makes easy metadata management for tables grouped under a namespace.
Advanced workflows based on namespace
- It helps to simplify automation for data pipelines or real-time analytics applications.

S3 table opertaion & management

Table Operation
They are quite similar to CRUD operation.

list tables
create tables
Get table metadata location
Update table metadata location
Delete Table

Table Management

Put Table Policy
Put Table Bucket Policy
Put Table Maintenance Config
Put Table Bucket Maintenance Config

Policies related to S3 table operation

Allow access to create and use table buckets

Here Action Lists the specific actions the policy allows.

These actions are S3 Tables-specific:

s3tables:CreateTableBucket: Grants permission to create a table bucket in S3 Tables.
s3tables:PutTableBucketPolicy: Allows setting or updating the bucket policy for a table bucket.
s3tables:GetTableBucketPolicy: Allows retrieving the bucket policy associated with a table bucket.
s3tables:ListTableBuckets: Allows listing all table buckets within the specified scope.
s3tables:GetTableBucket: Grants permission to access the metadata of a specific table bucket.
Resource Defines the scope of the resources these actions can apply to.
"arn:aws:s3tables:region:account_id:bucket/*": Specifies all table buckets in the account (account_id) and region (region).
The * after bucket/ indicates that permissions apply to all buckets under this account and region.

{
    "Version": "2012-10-17",
    "Statement": [{
        "Sid": "AllowBucketActions for user",
        "Effect": "Allow",
        "Action": [
            "s3tables:CreateTableBucket",
            "s3tables:PutTableBucketPolicy",
            "s3tables:GetTableBucketPolicy",
            "s3tables:ListTableBuckets",
            "s3tables:GetTableBucket"
        ],
        "Resource": "arn:aws:s3tables:region:account_id:bucket/*"
    }]
}

Allow access to create and use tables in a table bucket

Here Action Lists the specific actions allowed by the policy, related to S3 Tables. Please note that The first policy focused on creating and managing table buckets and associated metadata, but it did not include granular operations like managing tables within namespaces. The first policy did not include actions such as creating tables, querying data, or updating metadata at the table level. These are the operations where namespaces become relevant.

s3tables:CreateTable: Allows creating new tables in the specified table bucket.
s3tables:PutTableData: Grants permission to write data to tables within the table bucket.
s3tables:GetTableData: Allows reading data from tables in the bucket.
s3tables:GetTableMetadataLocation: Allows retrieving metadata location information for a table.
s3tables:UpdateTableMetadataLocation: Grants permission to update the metadata location of a table.
s3tables:GetNamespace: Allows retrieving namespace information associated with the table bucket.
s3tables:CreateNamespace: Grants permission to create namespaces for organizing table data.

Resource section specifies

Grants permissions on the bucket named amzn-s3-demo-table-bucket
Grants permissions on all tables within the amzn-s3-demo-table-bucket

{
     "Version": "2012-10-17",
     "Statement": [ 
         {
             "Sid": "AllowBucketActions",
             "Effect": "Allow",
             "Action": [
                 "s3tables:CreateTable",
                 "s3tables:PutTableData",
                 "s3tables:GetTableData",
                 "s3tables:GetTableMetadataLocation",
                 "s3tables:UpdateTableMetadataLocation",
                 "s3tables:GetNamespace",
                 "s3tables:CreateNamespace"
             ],

             "Resource": [
               "arn:aws:s3tables:region:account_id:bucket/amzn-s3-demo-table-bucket",
               "arn:aws:s3tables:region:account_id:bucket/amzn-s3-demo-table-bucket/table/*"
            ]
         }
     ]
 }

Table bucket policy to allows read access to the namespace

This policy allows to read s3 tables from a namespace. Here Action Lists the specific actions allowed by the policy, related to S3 Tables.

s3tables:GetTableData: Allows reading data from tables in the bucket.
s3tables:GetTableMetadataLocation: Allows retrieving metadata location information for a table. The resource section allows all s3 tables under bucket amzn-s3-demo-table-bucket1 but then s3tables:namespace restrict to only hr related s3 tables.

{
     "Version": "2012-10-17",
     "Statement": [ 
         {
             "Effect": "Allow",
             "Action": [
             "Principal": {
               "AWS": "arn:aws:iam::123456789012:user/Jane"
             },
             "Action": [
                  "s3tables:GetTableData", 
                  "s3tables:GetTableMetadataLocation"
             ],
             "Resource":{ "arn:aws:s3tables:region:account_id:bucket/amzn-s3-demo-table-bucket1/table/*”}
             "Condition": { 
                  "StringLike": { "s3tables:namespace": "hr" } 
             }
     ]
 }

S3 table automatic maintenance

It provides automated maintenance through configurations that help simplify table management, optimize performance, and reduce operational overhead.

Table Lifecycle Management
- we can add S3 Table configurations that includes lifecycle policies that automatically handle data expiration, transitions, or archival.
- automatic snapshot expiration can be configured easily.
Data Compaction
- S3 Tables automatically compact small files (often produced by incremental writes) into larger, optimized files. It helps to have faster query and reduce storage cost.
Schema Evolution
- Automated checks ensure compatibility between new and existing data.
Metadata Optimization
- Indexing of metadata for faster querying and retrieval of table details.

All these can be policy based configuration.

Policy for snapshot management

By configuring the maximumSnapshotAge, we can specify the retention period for table snapshots. The following example ensures S3 Table will automatically retain only the snapshots from the last 30 days

MinimumSnapshots: Ensures that at least one snapshot is always retained, regardless of age.
MaximumSnapshotAge: Specifies the maximum age (in hours) for snapshots to be retained.

aws s3tables put-table-maintenance-configuration \
    --table-arn arn:aws:s3tables:region:account_id:bucket/bucket_name/table/table_name \
    --maintenance-configuration '{
        "SnapshotManagement": {
            "MinimumSnapshots": 1,
            "MaximumSnapshotAge": 720
        }
    }

S3 Table Integration with AWS Analytics

S3 Tables integrate seamlessly with AWS analytics services to enable querying, processing and insight generation.

Amazon Athena - Run serverless SQL queries on S3 Tables

Use AWS Glue to create a Data Catalog for S3 Tables.
Query data directly using SQL in Athena.
Leverage table formats like Apache Iceberg or Parquet for optimized performance.

AWS Glue - Automate ETL processes for S3 Tables

Use Glue Crawlers to discover table metadata.
Create ETL jobs to transform and load data into S3 Tables or other destinations.

S3 Metadata table

It includes system metadata including object tags and user defined metadata
stored into s3 table
generated in near real time during data creation so that it can be used in mins during query

Use case for S3 metadata table

Real-Time Analytics
- efficient query execution on metadata to identify relevant data partitions.
Machine Learning Pipelines
- metadata tables to filter, select, and partition data for model training.
Governance and Compliance
- Track data retention and enforce lifecycle policies via metadata.
Multi-Tenant Data Applications
- Use namespaces within metadata tables to logically isolate tenant data.
Data Cataloging and Discovery
- Use metadata queries to identify datasets matching specific criteria.

Here is the sample python based function which uses metadata table query from athena.

def query_metadata_table(criteria):

    query = f"""
        SELECT *
        FROM {DATABASE}.{TABLE}
        WHERE {criteria}
    """

    print(f"Running query: {query}")

    # Start Athena query
    response = athena_client.start_query_execution(
        QueryString=query,
        QueryExecutionContext={'Database': DATABASE},
        ResultConfiguration={'OutputLocation': S3_OUTPUT}
    )

    query_execution_id = response['QueryExecutionId']

    # Wait for query completion
    print("Waiting for query to complete...")
    while True:
        status = athena_client.get_query_execution(QueryExecutionId=query_execution_id)
        state = status['QueryExecution']['Status']['State']
        if state in ['SUCCEEDED', 'FAILED', 'CANCELLED']:
            break
        time.sleep(2)

    if state != 'SUCCEEDED':
        raise Exception(f"Query failed with state: {state}")

    # Retrieve results
    results = athena_client.get_query_results(QueryExecutionId=query_execution_id)
    datasets = []
    for row in results['ResultSet']['Rows'][1:]:  # Skip the header row
        datasets.append([col['VarCharValue'] for col in row['Data']])

    print(f"Query returned {len(datasets)} datasets matching the criteria.")
    return datasets