Build an Agentic AI Workflow on AWS: Generative AI for Production Automation

Can AI help investigate production incidents without giving it unrestricted access to your AWS environment?

Imagine your production API starts returning HTTP 503 errors. Latency increases, CPU utilization spikes, and your logs show database connection pool timeouts.

A developer usually needs to open CloudWatch, inspect logs, compare metrics, investigate dependencies, and determine what to do next.

What if an AI-assisted workflow could collect the evidence, summarize the incident, recommend troubleshooting steps, and prepare a remediation plan for human approval?

In this tutorial, we'll explore how to build that workflow using AWS generative AI services and a small Python project.

We'll cover:

• Core AWS services for generative AI and agentic workflows.
• A reference architecture for production incident triage.
• A runnable Python demo.
• Integrating Amazon Bedrock.
• Adding approval gates and security controls.
• Measuring the real impact on production operations.

1. Core AWS Services for Agentic AI

Each AWS service has a different role in the workflow.

Amazon Bedrock: Foundation models

Amazon Bedrock provides managed access to supported foundation models through APIs.

You can use it to summarize application logs, classify incidents, explain deployment errors, generate reports, and recommend troubleshooting steps.

Amazon Bedrock Agents: Tool-based workflows

Amazon Bedrock Agents can coordinate foundation models with configured tools and knowledge bases to support multi-step tasks.

For example, an operations agent could collect approved metrics, retrieve a runbook, analyze an incident, and recommend a next step.

An agent should never receive unrestricted AWS permissions just because it can reason about operational problems.

Amazon Bedrock Knowledge Bases: Organizational context

Knowledge Bases support retrieval-augmented generation (RAG).

You can provide relevant context from internal runbooks, architecture documentation, incident reports, and deployment procedures.

This helps the model use your organization's documentation when generating recommendations, although its responses still need validation.

Amazon Bedrock Guardrails: Safety controls

Bedrock Guardrails provides configurable controls for supported use cases, including content filtering and sensitive-information protection.

Guardrails complement application validation and IAM controls; they do not replace them.

AWS Lambda: Controlled tools

AWS Lambda can provide specific functions for collecting metrics, retrieving approved logs, generating reports, or executing an authorized operation.

Keep the AI's recommendations separate from the code that performs the action.

AWS Step Functions: Workflow orchestration

AWS Step Functions can coordinate the process, handle execution states, and support workflows that wait for human approval.

Amazon CloudWatch: Operational evidence

Amazon CloudWatch provides metrics, logs, dashboards, and alarms.

It supplies the operational evidence that the AI-assisted workflow can analyze.

2. Architecture: From Incident Detection to Approved Remediation

Here's a possible architecture for an AI-assisted incident-triage system:

Production Application
         |
         v
Amazon CloudWatch
  Metrics, Logs, Alarms
         |
         v
AWS Step Functions
         |
         v
AWS Lambda
Collect approved evidence
         |
         v
Amazon Bedrock
Analyze incident and recommend actions
         |
         +----> Knowledge Base / Runbooks
         |
         v
Validate recommendation
         |
         v
Human approval
         |
         v
AWS Lambda
Execute approved action
         |
         v
CloudWatch
Verify outcome

This is a reference architecture, not an automatically configured integration. You must implement the relevant API calls, IAM roles, validation logic, and workflow states.

The important design decision is that the model recommends an action, but the application decides whether that action is allowed.

3. Build a Python Incident-Triage Demo

Let's create a small project that identifies symptoms in simulated production telemetry.

Our sample checkout-api has:

• CPU utilization: 92%
• Memory utilization: 78%
• HTTP 5xx errors: 45
• P95 latency: 2,400 ms
• Database connection pool timeouts

For the first version, we'll use deterministic Python rules. This makes the project runnable without AWS credentials or model access.

Step 1: Create the project

aws-ai-incident-triage/
└── incident_triage.py

Step 2: Add the code

Create incident_triage.py:

import json

incident = {
    "service": "checkout-api",
    "environment": "production",
    "cpu_utilization_percent": 92,
    "memory_utilization_percent": 78,
    "http_5xx_count": 45,
    "p95_latency_ms": 2400,
    "recent_logs": [
        "Database connection pool timeout",
        "Request latency exceeded 2000 ms",
        "HTTP 503 returned by checkout-api"
    ]
}


def investigate(data):
    findings = []
    recommendations = []

    if data["cpu_utilization_percent"] >= 85:
        findings.append("CPU utilization is elevated.")
        recommendations.append(
            "Inspect CPU-intensive requests and task capacity."
        )

    if data["memory_utilization_percent"] >= 85:
        findings.append("Memory utilization is elevated.")
        recommendations.append(
            "Inspect memory usage and possible memory leaks."
        )

    if data["http_5xx_count"] >= 10:
        findings.append("HTTP 5xx errors exceed the demo threshold.")
        recommendations.append(
            "Inspect application errors and dependency failures."
        )

    if data["p95_latency_ms"] >= 1000:
        findings.append("P95 latency exceeds the demo threshold.")
        recommendations.append(
            "Investigate slow dependencies and request bottlenecks."
        )

    if any(
        "connection pool timeout" in message.lower()
        for message in data["recent_logs"]
    ):
        findings.append(
            "Logs indicate database connection pool timeouts."
        )
        recommendations.append(
            "Inspect database connections, pool limits, "
            "and query latency."
        )

    return {
        "service": data["service"],
        "environment": data["environment"],
        "findings": findings,
        "recommendations": recommendations,
        "proposed_action": (
            "Review service capacity and database connection behavior."
        ),
        "approval_required": True,
        "execution_status": "NOT_EXECUTED"
    }


report = investigate(incident)
print(json.dumps(report, indent=2))

Step 3: Run it

python incident_triage.py

No third-party Python packages are required for this first version.

Step 4: Review the output

The output will look similar to this:

{
  "service": "checkout-api",
  "environment": "production",
  "findings": [
    "CPU utilization is elevated.",
    "HTTP 5xx errors exceed the demo threshold.",
    "P95 latency exceeds the demo threshold.",
    "Logs indicate database connection pool timeouts."
  ],
  "recommendations": [
    "Inspect CPU-intensive requests and task capacity.",
    "Inspect application errors and dependency failures.",
    "Investigate slow dependencies and request bottlenecks.",
    "Inspect database connections, pool limits, and query latency."
  ],
  "proposed_action": "Review service capacity and database connection behavior.",
  "approval_required": true,
  "execution_status": "NOT_EXECUTED"
}

This output is generated by deterministic rules, not by a live AI model.

The demo identifies symptoms and recommends next steps. It does not restart services, modify infrastructure, or make production changes.

4. Connect the Workflow to Amazon Bedrock

Next, replace the rule-based analysis with a foundation model.

First, configure AWS credentials and choose a model available to your account in your selected Region. Ensure the identity has the required Bedrock permissions.

Install the AWS SDK for Python if needed:

python -m pip install boto3

The following example assumes the incident dictionary from the previous section is available:

import json
import boto3

bedrock = boto3.client(
    "bedrock-runtime",
    region_name="us-east-1"
)

response = bedrock.converse(
    modelId="YOUR_SUPPORTED_MODEL_ID",
    system=[
        {
            "text": (
                "You are an incident-triage assistant. "
                "Use only the supplied evidence. "
                "Distinguish observed facts from hypotheses. "
                "Never claim to have executed a remediation. "
                "Recommend investigation steps and flag uncertainty."
            )
        }
    ],
    messages=[
        {
            "role": "user",
            "content": [
                {"text": json.dumps(incident)}
            ]
        }
    ],
    inferenceConfig={
        "maxTokens": 800,
        "temperature": 0
    }
)

print(
    response["output"]["message"]["content"][0]["text"]
)

Replace YOUR_SUPPORTED_MODEL_ID with an actual supported model ID.

This code sends the sample incident to Amazon Bedrock and prints the model's response. It does not automatically retrieve CloudWatch telemetry or execute remediation actions.

For a real implementation, add an authorized Lambda integration to collect metrics and logs, validate model output against a defined schema, and pass approved operations into a controlled workflow.

5. Add an Approval Gate

A model can suggest a valid-looking action that is inappropriate for the target environment.

Start with an application-level allowlist:

APPROVED_ACTIONS = {
    "inspect_logs",
    "collect_metrics",
    "create_incident_report"
}


def authorize_action(action):
    if action not in APPROVED_ACTIONS:
        return {
            "allowed": False,
            "reason": "Action is not in the approved allowlist."
        }

    return {
        "allowed": True,
        "reason": "Action is allowed by the demo policy."
    }


print(authorize_action("inspect_logs"))
print(authorize_action("delete_production_database"))

The first action is permitted by this example policy; the second is denied.

This is a basic demonstration, not a complete authorization system. Production code must also validate the target resource, parameters, caller identity, and IAM permissions.

For actions that can affect production availability or data, use a workflow that requires explicit human approval before execution.

Never rely on a prompt alone to enforce a security boundary.

6. How Can You Measure Production Impact?

AI adoption should be evaluated against a baseline.

Metric	Evaluation method
Investigation time	Compare time spent preparing incident summaries
Evidence completeness	Check whether required logs and metrics are included
Recommendation accuracy	Compare suggestions with expert-reviewed findings
Unsafe recommendations	Track policy violations in proposed actions
Cost per investigation	Include model, compute, storage, and operational costs
Recovery time	Compare actual incident recovery outcomes

Test the workflow with different cases, including high CPU, database timeouts, missing telemetry, misleading symptoms, and incidents that require more information.

These are evaluation goals, not measured results from the demo.

7. Production Risks to Address

Before deploying an agentic workflow, plan for:

• Hallucinations: Require evidence-based findings and validate recommendations.
• Prompt injection: Treat logs and retrieved documents as untrusted input.
• IAM security: Use least-privilege roles and narrowly scoped tools.
• Reliability: Add timeouts, bounded retries, and fallback behavior.
• Cost and latency: Limit unnecessary model calls and measure end-to-end performance.
• Observability: Record tool calls, errors, approval decisions, and outcomes.
• Human oversight: Require approval for risky or irreversible changes.

Bedrock Guardrails can provide useful additional controls, but they are only one layer of the security design.

8. Implementation Roadmap

A practical rollout can follow these steps:

1. Collect approved telemetry through CloudWatch.
2. Build a Lambda function to retrieve the required evidence.
3. Use Amazon Bedrock to summarize symptoms and recommend next steps.
4. Validate the response against an expected schema and application policy.
5. Orchestrate the workflow with Step Functions.
6. Require approval before production-changing actions.
7. Measure recommendation quality, latency, and cost.
8. Add a Knowledge Base when internal runbooks provide useful context.

Use Bedrock Agents when model-driven tool selection and multi-step task execution are valuable. A deterministic workflow containing a model call may be easier to secure and operate for simpler use cases.

Conclusion

Agentic AI is not just about connecting a language model to tools. It is about combining useful reasoning with reliable evidence, controlled execution, and clear authorization boundaries.

Amazon Bedrock provides model access, Lambda supplies controlled tools, CloudWatch provides operational evidence, and Step Functions coordinates the workflow.

Our Python demo illustrates the first step: transform telemetry into structured findings and recommendations without executing production changes.

The next step is to connect the workflow to real telemetry, test it against known incidents, and measure whether it improves the investigation process.

My principle for production AI: Let the model recommend, let the application validate, and let authorized people approve consequential changes.

References

• Amazon Bedrock documentation
• Amazon Bedrock Agents
• Amazon Bedrock Guardrails
• AWS Step Functions human approval workflow
• Amazon CloudWatch