DEV Community: Yoshiki Fujiwara(藤原善基)@AWS Community Builder

42 Patterns, Category Architecture, and HA LifeKeeper Monitoring — FSx for ONTAP S3 Access Points, Phase 18

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 21 Jun 2026 16:54:22 +0000

TL;DR

Phase 18 restructures the entire repository from 41 flat directories into a categorized solutions/ hierarchy, adds HA LifeKeeper Monitoring as a new pattern category, introduces 5 category-specific architecture diagrams, and establishes modern Python project infrastructure. The repository now contains 42 deployable patterns organized for discoverability at scale.

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

Why Restructure?

With 41 pattern directories at the repository root, navigating the project had become unwieldy. New contributors could not quickly find patterns by domain, the README required extensive scrolling, and adding new categories (HA, GenAI) had no clear placement convention.

Before (flat):

legal-compliance/  financial-idp/  semiconductor-eda/  sap-erp-adjacent/
flexcache-anycast-dr/  genai-kb-selfservice-curation/  event-driven-fpolicy/
... (41 directories at root)

After (categorized):

solutions/
├── industry/           # 28 UC patterns (UC1-UC28)
├── flexcache/          # 7 FlexCache/FlexClone patterns
├── genai/              # 2 GenAI patterns (UC29-UC30)
├── sap/                # SAP/ERP pattern
├── ha/                 # HA monitoring (new)
├── event-driven/       # 2 FPolicy event-driven patterns
└── edge/               # CDN/edge delivery

The Move: git mv with History Preservation

All 41 directories were moved using git mv, preserving full commit history. Key renames include:

Old Path	New Path	Reason
`sap-erp-adjacent/`	`solutions/sap/erp-adjacent/`	Category grouping
`dynamic-flexcache-render-workflow/`	`solutions/flexcache/dynamic-render-workflow/`	Shorter name
`genai-kb-selfservice-curation/`	`solutions/genai/kb-selfservice-curation/`	Strip prefix
`devops-flexclone-cicd/`	`solutions/flexcache/devops-cicd/`	Strip prefix
`content-edge-delivery/`	`solutions/edge/content-delivery/`	Category grouping

Use git log --follow <file> to trace history across the move.

HA LifeKeeper Monitoring — New Pattern

SIOS LifeKeeper is a Linux/Windows HA clustering solution that can be used on Amazon EC2 for application-aware failover scenarios. With FSx for ONTAP Multi-AZ as shared storage (NFS/iSCSI, depending on OS and configuration), this pattern focuses on observing LifeKeeper logs without putting monitoring agents on the HA nodes.

The new HA pattern (solutions/ha/lifekeeper-monitoring/) provides non-intrusive log analysis:

graph TB
    subgraph "HA Cluster"
        LK1[LifeKeeper Node 1<br/>Active]
        LK2[LifeKeeper Node 2<br/>Standby]
    end

    subgraph "Shared Storage"
        FSXN[FSx for ONTAP Multi-AZ]
        S3AP[S3 Access Point<br/>Read-only log access]
    end

    subgraph "Analysis Pipeline"
        SFN[Step Functions]
        DISC[Discovery Lambda<br/>Log classification]
        PROC[Processing Lambda<br/>Bedrock Root Cause Analysis]
        RPT[Report Lambda<br/>Health score + alerts]
    end

    LK1 -->|Log write| FSXN
    FSXN --> S3AP -->|Non-intrusive read| DISC
    SFN --> DISC --> PROC --> RPT
    PROC -->|Nova Pro| BEDROCK[Amazon Bedrock]
    RPT --> SNS[SNS Alert]

Key Design Decisions

Non-intrusive to HA nodes: No monitoring agent is installed on HA nodes. The S3 AP read path avoids host-level changes, while still consuming FSx/S3 API throughput like any other read workload.
Human-in-the-loop: AI analysis is advisory only. LifeKeeper's own health checks handle failover decisions.
Health Scoring: 0-100 score with deductions for failover events, comm path latency, and resource state anomalies.
Root Cause Analysis: Bedrock Nova Pro analyzes state transitions (ISP→OSF, ISS→ISP) to identify likely causes.

Implementation note: This pattern observes LifeKeeper logs and produces advisory analysis. It does not replace LifeKeeper cluster design, quorum/witness configuration, split-brain prevention, protocol-specific recovery kit setup, or application-level failover testing.

Example Operational Metrics

These are suggested evaluation metrics for future validation. Phase 18 verifies the DemoMode pipeline, not real-cluster failover triage time.

Category	Metric	Demo Target
Operations	Time from workflow start to structured triage report	< 10 min
Technical	Log discovery completeness	All configured LifeKeeper log paths in scope
Quality	False positive alert rate	< 5% (requires real-cluster validation)
Cost	Monthly monitoring cost	< $15 (5-min polling, 1 cluster)

Deploy (DemoMode)

DemoMode=true deploys without FSx for ONTAP — uses a regular S3 bucket with sample LifeKeeper logs:

cd solutions/ha/lifekeeper-monitoring
sam build && sam deploy --guided \
  --parameter-overrides DemoMode=true \
    S3AccessPointAlias=your-demo-bucket \
    OutputBucketName=your-output-bucket

DemoMode verification confirms: SAM template deploys successfully, Step Functions workflow executes all states, Discovery Lambda classifies sample logs correctly, and Processing Lambda generates a health score report. Bedrock analysis produces structured, advisory observations based on log patterns; failover decisions remain outside the AI workflow.

Verified in ap-northeast-1 on 2026-06-21. The sample log set intentionally contains failure-like events, resulting in a demo health score of 40/100. The score is not a benchmark for LifeKeeper or FSx for ONTAP; it demonstrates that the scoring pipeline detects and reports anomalies from sample logs.

Coming in Phase 19: Full E2E verification with a real SIOS LifeKeeper HA cluster (AWS Marketplace + FSx for ONTAP Multi-AZ) including live failover testing, actual state transition detection, and Bedrock RCA quality assessment against real-world failure scenarios.

Category Architecture Diagrams

Phase 18 adds 5 mermaid architecture diagrams to README.md (both JA and EN), each in a collapsible <details> block:

Category	Key Components
🏭 FlexCache	ONTAP REST API → HealthCheck → RouteDecision → DynamoDB routing → Create/Cleanup lifecycle
🤖 GenAI	FPolicy → SQS → EventBridge → Bedrock KB → RetrieveAndGenerate / Agentic tools
🛡️ HA	S3 AP non-intrusive read → Bedrock RCA → Health score → SNS alerts
⚡ Event-Driven	FPolicy Engine → ECS Fargate TCP → SQS → EventBridge rule routing
🌐 Edge/CDN	3 delivery modes (ORIGIN_PULL, OAC, PUBLISH_PUSH) → vendor-neutral CDN

All diagrams include accTitle and accDescr for screen reader accessibility.

Project Infrastructure Improvements

pyproject.toml (PEP 621)

Modern Python project metadata with unified tool configuration:

[tool.ruff]
target-version = "py312"
line-length = 120

[tool.pytest.ini_options]
addopts = "-v --tb=short --import-mode=importlib"

[tool.coverage.report]
fail_under = 80

Dependency Pinning

# requirements.txt — exact versions for reproducibility
boto3==1.43.29
urllib3==2.7.0
jsonschema==4.17.3

# requirements-dev.txt
pytest==9.1.0
hypothesis==6.155.2
moto==5.2.2
ruff==0.15.17
cfn-lint==1.51.4

.cfnlintrc

Project-wide cfn-lint configuration that discovers all templates under solutions/:

templates:
  - "solutions/**/template.yaml"
  - "solutions/**/template-deploy.yaml"
ignore_checks:
  - W3002  # Local CodeUri (sam build handles upload)
  - W1031  # Fn::Sub false positive with Secrets Manager ARNs
regions:
  - ap-northeast-1

Additional Tooling

File	Purpose
`.gitattributes`	Consistent line endings, language-specific diff drivers
`.github/PULL_REQUEST_TEMPLATE.md`	Project-specific PR checklist
`solutions/README.md`	Category navigation index
`CHANGELOG.md`	Keep a Changelog format, all releases
`CONTRIBUTING.md`	"Adding a New Pattern" section with category guide

CI/CD Changes

The CI pipeline was split to avoid pytest importlib mode namespace collisions when multiple patterns with identically-named handler.py files are collected together:

# Before: single pytest invocation (collision risk)
pytest shared/tests/ solutions/**/tests/ --cov=shared

# After: isolated invocations
- name: Run shared tests with coverage
  run: pytest shared/tests/ --cov=shared
- name: Run pattern tests
  run: pytest solutions/industry/*/tests/ solutions/flexcache/*/tests/ ...

Also added persist-credentials: false to all actions/checkout steps (zizmor security hardening).

Multi-Perspective Review

The restructuring and HA monitoring pattern were reviewed from partner delivery, storage architecture, HA operations, security, CI/CD, accessibility, and contributor onboarding perspectives. The review resulted in wording changes around neutrality, operational caveats, connector validation, HA safety boundaries, and repository discoverability.

What This Means for You

Role	Impact
Partner / SI / delivery team	Pattern selection is now intuitive by category — pick `solutions/industry/` for industry PoCs, `solutions/flexcache/` for distributed workloads
New contributor	`CONTRIBUTING.md` now includes "Adding a New Pattern" with required checklist and category selection guide
Existing repository users	`git log --follow <file>` still works. `sam build` and `sam deploy` are unchanged (run from the pattern directory)
CI/CD	Zero changes needed in your `samconfig.toml` or deployment scripts — CodeUri is relative to template.yaml

Infrastructure cost: Zero. This is a repository organization change; it does not deploy AWS resources by itself.

What's Next

Phase 16-17 blog articles (GenAI patterns) — ready to publish
dev.to series updated for directory restructuring (Phase 13 link fixed)
Next pattern candidates under evaluation

Getting Started

git clone https://github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns.git
cd FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

# Browse patterns by category
ls solutions/

# Deploy HA LifeKeeper monitoring (DemoMode)
cd solutions/ha/lifekeeper-monitoring
sam build && sam deploy --guided --parameter-overrides DemoMode=true

# Run tests
make test-quick PYTHON=.venv/bin/python

Yoshiki Fujiwara

Amazon Quick Agentic Workspace Powered by FSx for ONTAP S3 Access Points — Phase 17

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 21 Jun 2026 15:54:23 +0000

TL;DR

UC30 bridges the gap between file-based business data and AI-powered actions. Business users maintain structured and unstructured data on an FSx for ONTAP SMB share, while Amazon Quick (Index / Sight / Flows) consumes it through S3 Access Points and a serverless Action API — providing search, BI, and governed action workflows from a single workspace.

Where UC29 focuses on "self-service knowledge ingestion into Bedrock KB," UC30 focuses on unifying search, analytics, and action execution behind Quick Suite's agentic interface.

New in this release: generate_brief_with_web action augments internal context with real-time web search results via AgentCore Web Search Tool (GA June 2026), enabling briefs that combine primary internal data with current public context.

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns (see solutions/genai/quick-agentic-workspace/ and samconfig.toml.example)

Amazon Quick × S3 AP Data Mapping

Quick Feature	Role	S3 AP Data	Implementation
Quick Index / Research	Unstructured file search	`index/<role>/` (md/pdf)	S3 AP as data source
Quick Sight	Structured BI & visualization	`analytics/<role>/` (csv)	Glue/Athena (Athena Query Lambda)
Quick Flows	Action automation	`flows/<role>/` (json)	Action API (API Gateway + Lambda + Bedrock)
Quick Flows + Web	Web-augmented briefs	`flows/<role>/` + web	Action API + AgentCore Web Search (opt-in)

Seven roles (sales / marketing / finance / IT / operations / legal / developers) share the same AI-dedicated volume — reusable from UC29.

Design note: FSx for ONTAP S3 Access Points are useful as an integration boundary, but they do not remove the need to validate each consuming service connector. The access path combines S3/IAM policy evaluation with file-system-level identity authorization.

Architecture

Amazon Quick provides a unified workspace — search (Index), BI (Sight), and action automation (Flows) — powered by FSx for ONTAP data via S3 Access Points.

Windows Explorer (drag & drop into quick-workspace/ SMB share)
  ├── index/<role>/ → Quick Index (unstructured search)
  ├── analytics/<role>/ → Glue/Athena → Quick Sight (BI)
  └── flows/<role>/ → Action API → Quick Flows (actions)

Action API (6 actions):
  API Gateway (IAM auth / SigV4)
  → Lambda (per-action authorization + HITL gate)
  → generate_brief           → Bedrock Converse (internal context only)
  → generate_brief_with_web  → Bedrock Converse + AgentCore Web Search (hybrid)
  → create_action_item       → SNS notification
  → request_approval         → DynamoDB (HITL entry)
  → approve                  → DynamoDB (admin only)
  → execute_approved         → DynamoDB check + execution (enforced HITL)

Hybrid RAG Flow (generate_brief_with_web)

Quick Flows request: {"action": "generate_brief_with_web", "params": {...}}
  ├─→ [1] Internal context (from params.context — FSx for ONTAP file content)
  ├─→ [2] AgentCore Web Search (us-east-1, MCP protocol)
  │       query from params.web_query or params.title
  │       → Amazon web index → snippets + URLs + titles + dates
  └─→ [3] Bedrock Converse (ap-northeast-1)
           system prompt: internal = primary, web = supplementary, untrusted
        → Unified brief with [Internal: ...] + [Web: title](URL) citations

Security Design

Authentication + Per-Action Authorization

The Action API uses IAM authentication (SigV4). The handler extracts the authenticated caller identity (requestContext.identity) — not self-declared body fields — and performs per-action authorization:

ACTION_AUTH_MODE=open (default/demo): No enforcement; audit fields still bound to authenticated caller

For production, use ACTION_AUTH_MODE=enforce and explicitly define AUTHORIZED_PRINCIPALS and ADMIN_PRINCIPALS.

ACTION_AUTH_MODE=enforce (production):

Read-only actions (generate_brief, generate_brief_with_web): always allowed

Mutating actions: caller must match AUTHORIZED_PRINCIPALS

Admin actions (approve): caller must match ADMIN_PRINCIPALS

Mismatch → 403 Forbidden

Enforced Human-in-the-Loop (HITL)

High-risk operations are gated by a DynamoDB approval store:

request_approval → persists record as pending_approval (enforced=true)
approve → admin transitions to approved (ConditionExpression prevents race)
execute_approved → only executes if record is approved; otherwise 409

Verified live: execute pre-approval → 409, post-approval → 200, re-execute → 409 (no replay).

Security note: Approval records have a 7-day TTL (DynamoDB Time-to-Live). Stale pending approvals auto-expire, preventing indefinite accumulation of unreviewed requests. Expired records cannot be approved or executed.

Additional Controls

Prompt injection defense: Both generate_brief and generate_brief_with_web treat context as untrusted data with explicit delimiter boundaries (<internal_context>, <web_search_results>)
Web query safety: Only params.web_query or params.title is sent to Web Search — never internal document content
Raw SQL disabled by default: ALLOW_RAW_SQL=false; role-level data boundaries enforced via Lake Formation (LF-TBAC) in production
Results bucket hardening: PublicAccessBlock + TLS-only + 30-day lifecycle
API throttling: Rate/burst limits against denial-of-wallet
Web Search citation obligation: Source URLs + titles are always included in responses (Acceptable Use Policy compliance)

Web-Augmented Brief Generation (opt-in)

GA at AWS Summit NYC 2026 (June 17, 2026). Powered by AgentCore Web Search Tool.

The Problem

Business briefs based solely on internal documents lack current market context. A sales brief about a product launch needs both the internal product spec and awareness of relevant public announcements published recently. A legal compliance brief needs both the internal policy document and the latest regulatory guidance.

The Solution

A new action generate_brief_with_web combines internal context with real-time web search results. The internal context remains the primary source; web results are supplemental, cited, and treated as untrusted input.

Usage

{
  "action": "generate_brief_with_web",
  "params": {
    "title": "Q3 Data Protection Regulatory Update",
    "context": "Internal operations follow FISC safety standards...",
    "web_query": "data protection regulation 2026 Japan financial services"
  }
}

Response

{
  "status": "completed",
  "action": "generate_brief_with_web",
  "title": "Q3 Data Protection Regulatory Update",
  "brief": "Based on internal FISC compliance documentation... Additionally, [Web: FISC 2026 Revision Summary](https://example.com/fisc) published on 2026-06-10 introduces...",
  "web_citations": [
    {"source": "https://example.com/fisc", "title": "FISC 2026 Revision Summary", "publishedDate": "2026-06-10"}
  ],
  "web_search_enabled": true,
  "guardrail_applied": true
}

Design Properties

Property	Detail
Internal context priority	Internal documents are the primary source; web supplements
Graceful degradation	Web Search failure → behaves like `generate_brief` (internal only)
Citation separation	Internal sources and web sources are visually distinct in the brief
Query safety	Only the `web_query` (or `title`) is sent externally — never internal content
Cross-region	Gateway in us-east-1 (Web Search Tool constraint); adds ~100-200ms
Authorization	Read-only action (same tier as `generate_brief`)
Prompt injection defense	Web results wrapped in `<web_search_results>` as untrusted data

Activation

sam deploy --parameter-overrides \
  EnableWebSearch=true \
  AgentCoreGatewayId=<gateway-id> \
  AgentCoreGatewayRegion=us-east-1

Without these parameters, generate_brief_with_web still works but produces internal-only briefs (graceful degradation).

Verification Findings

Lake Formation + Athena

Athena queries running against Glue tables backed by S3 AP data — the foundation for Quick Sight analytics.

The UC30 CloudFormation stack with all resources (API Gateway, Lambda, DynamoDB ApprovalsTable, Athena WorkGroup) deployed.

The test account had Lake Formation governing the Data Catalog. The Athena Query Lambda's execution role required Lake Formation permission grants (DESCRIBE on DB, SELECT/DESCRIBE on tables) in addition to IAM. Production deployments should design LF-TBAC for role-based data visibility.

Amazon Quick × FSx for ONTAP S3 AP Integration Boundary

Amazon Quick's S3 KB connector accepts the S3 AP alias but authorization fails due to FSx for ONTAP's dual-layer auth — leading to the recommendation below.

Quick provides multiple data integration paths — for FSx for ONTAP data, Bedrock KB (UC29) or Athena-mediated access is the validated route in this repository.

Amazon Quick's S3 knowledge base connector accepts an FSx for ONTAP S3 AP alias as a "valid URL," but connection verification fails with "You do not have permissions to access the S3 bucket." This is not a missing IAM permission that can be simply added — it is a structural authorization path mismatch. Three factors block the standard connector path:

ARN format mismatch — FSx for ONTAP S3 APs use arn:aws:s3:{region}:{account}:accesspoint/{name} (not the arn:aws:s3:::{bucket} format). The connector's internal IAM evaluation likely targets the alias as a bucket name, which does not match IAM policy evaluation for the AP ARN.
AP resource policy rejects the principal — Adding Quick's data access role to the S3 AP resource policy returns MalformedPolicy: Invalid principal in policy, indicating a principal constraint on FSx for ONTAP S3 AP policies.
Layer 2: filesystem identity — Even if IAM layers were resolved, the ONTAP file-system identity (UNIX UID or AD user) associated with the S3 AP must independently permit read access to the target files.

For readers attempting this path: Use CloudTrail to capture the exact AccessDenied event and identify Quick's calling principal ARN. If the MalformedPolicy constraint can be resolved (potentially by configuring the S3 AP with an AD-based identity rather than UNIX root), the direct path may become viable. As of June 2026, Bedrock Knowledge Base → FSx for ONTAP S3 AP is the confirmed working route for RAG ingestion.

Evidence-based implementation guidance:

FSx for ONTAP → RAG: Bedrock KB (UC29) is the validated path in this repository
Quick Index: Stage to a standard S3 bucket for predictable connector behavior
Quick Sight (BI): Athena-mediated access works (QuickSight role needs Athena/Glue/LF/results-bucket permissions)
Direct Quick → FSx for ONTAP S3 AP (standard connector path): validated — does not work (tested with UNIX root identity; MalformedPolicy on AP resource policy). AD-based S3 AP identity configuration is tracked as a future hypothesis

Glue Tables

analytics/<role>/ CSVs are pointed to by Glue tables (sales_pipeline / it_incidents) created via Athena DDL. LOCATION uses S3 AP alias format: s3://<alias>/quick-workspace/analytics/<role>/. For scale, migrate to Parquet + partitioning to reduce Athena scanned costs.

Data Classification

Output	Classification	Rationale
Action API response (generate_brief)	INTERNAL	Source-derived summary; no external disclosure
Action API response (generate_brief_with_web)	INTERNAL	Contains internal citations; web portion is PUBLIC
Action API response (create/approve/execute)	INTERNAL	Business operation records
Athena query results (results bucket)	INTERNAL	Encrypted + 30-day lifecycle + TLS
DynamoDB ApprovalsTable	INTERNAL	Approval state metadata
SNS notifications	INTERNAL	Action summaries only; no file content
Web Search results (raw)	PUBLIC	External public information

Extend shared/data_classification.py for regulated workloads (CUI / FISC / HIPAA).

Cost

Component	Monthly estimate	Notes
Amazon Quick	Per-user/plan billing	Separate; unsubscribe when done
Athena	Scanned-data pricing	Reduce with Parquet
Lambda / API Gateway	Serverless pay-per-use	< $10 for moderate usage
Bedrock LLM (briefs)	Usage-based	Usage-based; verify the current model price
DynamoDB (approvals)	Pay-per-request	Minimal for approval records
AWS Budgets alarm	Free (SNS delivery cost only)	Created when `NotificationEmail` set
AgentCore Web Search (opt-in)	Per-query pricing (see AgentCore pricing)	Gateway invocation pricing
Cross-region transfer (opt-in)	< $0.02	us-east-1 ↔ ap-northeast-1

Teardown / rebuild: one-command idempotent scripts (scripts/teardown-uc29-uc30.sh / scripts/rebuild-uc29-kb.py)

Getting Started

git clone https://github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns.git
cd FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns/solutions/genai/quick-agentic-workspace

# Install dependencies
pip install -r requirements.txt        # or: uv pip install -r requirements.txt

cat samconfig.toml.example  # Review parameters

sam build && sam deploy --guided

# DemoMode=true runs without FSx for ONTAP (regular S3 bucket)

# Optional: Enable Web Search hybrid RAG
sam deploy --parameter-overrides \
  EnableWebSearch=true \
  AgentCoreGatewayId=<gateway-id> \
  AgentCoreGatewayRegion=us-east-1

Governance Note

This article is technical architecture guidance, not legal, compliance, or regulatory advice. Amazon Quick features, pricing, regional availability, and connector behavior are subject to change — verify with official documentation and your own account settings. S3 AP data source boundaries are at volume/prefix granularity. For per-user visibility control, use Quick's document-level ACL or Custom Permission-Aware RAG. Web Search Tool usage requires compliance with the Acceptable Use Policy (source citations must be retained and displayed in end-user output).

Yoshiki Fujiwara

Amazon Bedrock Knowledge Bases + FSx for ONTAP S3 Access Points: Self-Service AI Curation via Windows Drag & Drop — Phase 16

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 21 Jun 2026 15:19:58 +0000

TL;DR

"Put files in this Windows folder and your AI assistant can use them after the next governed sync" — UC29 reduces the handoff friction around enterprise AI knowledge updates. Business users maintain Amazon Bedrock Knowledge Base content through Windows Explorer drag & drop on an FSx for ONTAP SMB share. No S3 console, no ETL, no copy. Data remains in the file system as the operational source; the S3 Access Point provides a governed access path for ingestion without creating a separate object copy.

Three maturity stages + one opt-in enhancement:

Scenario A (manual): User places files → triggers sync from console/CLI
Scenario B (scheduled): EventBridge Scheduler + Step Functions poll every 15 minutes
Scenario C (event-driven): FPolicy detects file placement → real-time KB sync
Hybrid RAG (opt-in): Internal KB answers augmented with real-time web search via AgentCore Web Search Tool (GA June 2026)

Vector store: Amazon S3 Vectors — a managed, cost-aware vector store option for Bedrock Knowledge Bases, with metadata filtering available for permission-aware retrieval designs.

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns (see solutions/genai/kb-selfservice-curation/ and samconfig.toml.example for deployment parameters)

Why This Pattern?

Challenge	Traditional approach	This pattern
Knowledge updates blocked on IT	Ticket → manual ETL	Business user drags & drops
Dual management (NAS + S3 copy)	Source drifts from S3 replica	S3 AP reads the single source directly
Forgotten re-ingestion	Manual and easy to forget	Automatic (Scenario B/C)
Specialist skills required	ETL / S3 / Bedrock expertise	Familiar Windows folder operations
Answers limited to internal docs	No current external context	Hybrid RAG: internal + web search (opt-in)
Vector store cost	OpenSearch Serverless ~$175/month minimum	S3 Vectors: cost-aware pay-per-use profile

Architecture

Users drag files into an AI-dedicated NTFS volume (SMB share, ACL-separated by department) on FSx for ONTAP. The same volume is exposed via S3 Access Point as a Bedrock Knowledge Base data source:

Users see a familiar Windows folder structure — each department has its own folder with NTFS ACL separation.

Drag & drop a product spec into the sales folder — that is the only content-maintenance step for the user.

Windows Explorer (drag & drop)
  → FSx for ONTAP volume (ai_knowledge/)
  → S3 Access Point (read-only, internet-origin)
  → Bedrock Knowledge Base Data Source (inclusionPrefixes: ai-knowledge/)
  → StartIngestionJob → S3 Vectors (vector store) updated

Seven business roles (sales / marketing / finance / IT / operations / legal / developers) share the volume with NTFS ACL-based permission separation.

Design note: S3 Access Points for FSx for ONTAP let S3-compatible AWS services access file data without copying it to an S3 bucket. Access is authorized twice: first through AWS/IAM policies on the access point path, and then through the file-system identity and permissions associated with the access point.

Hybrid RAG Flow (opt-in)

When EnableWebSearch=true, the Query Lambda augments internal KB answers with real-time web information:

User question
  ├─→ [1] Bedrock KB RetrieveAndGenerate (internal docs via S3 AP → S3 Vectors)
  ├─→ [2] AgentCore Web Search (us-east-1, cross-region MCP call)
  │       Amazon-operated web index (tens of billions of docs, continuously updated)
  └─→ [3] Bedrock Converse → unified answer with dual citations
            [Internal: product-spec.pdf] + [Web: Market Report 2026](https://...)

Scenario B: Scheduled Automation (Step Functions)

The most common production deployment. EventBridge Scheduler triggers a Step Functions workflow every 15 minutes:

EventBridge Scheduler (rate(15 minutes))
  └─→ Step Functions State Machine
       ├─→ DetectAndStartIngestion Lambda
       │     • ListObjectsV2 via S3 AP → compare with last-known state
       │     • If changes detected → StartIngestionJob
       │     • If no changes → skip (cost-zero run)
       ├─→ Wait (30s) → CheckIngestionStatus Lambda (poll loop)
       └─→ NotifySuccess / NotifyFailure → SNS

Key design choices:

Differential detection: Compares current S3 AP listing against prior state; only triggers ingestion when files actually changed
Idempotent: Re-running the same schedule with no file changes is a no-op (no wasted Bedrock ingestion cost)
Observable: Each Step Functions execution is visible in the console with per-state timing and error details
Configurable interval: The ScheduleExpression parameter accepts any EventBridge rate/cron expression

This is the required safety net for Scenario C (see below) — it catches any files missed during the lost-update window.

Scenario C: FPolicy Event-Driven Real-Time Sync

Scenario B's 15-minute polling is replaced by FPolicy real-time detection:

Windows/NFS file operation
  → FPolicy instant detection (CREATE/WRITE/DELETE/RENAME)
  → FPolicy Server → SQS → Bridge Lambda → EventBridge custom bus
  → EventBridge Rule (file_path prefix = ai_knowledge)
  → KB Trigger Lambda (debounce) → StartIngestionJob
  → Bedrock KB → reflected in tens of seconds to minutes

The FPolicy → SQS → EventBridge front-end reuses the existing solutions/event-driven/fpolicy pattern infrastructure. UC29 adds only an EventBridge rule and the KB Trigger Lambda.

Lost-Update Window (Critical)

The EventBridge rule routes only FPolicy events matching the ai_knowledge volume path to the KB Trigger Lambda.

Bedrock Ingestion performs a full-source scan at job start time. Files added during a running job are not included in that execution. Scenario C alone does not guarantee zero missed files.

Mandatory: Always pair Scenario C with Scenario B (periodic reconcile sync) as a safety net. The KB Trigger Lambda skips when a job is already in progress (debounce + ConflictException handling + reserved concurrency = 2).

Namespace Pitfall

FPolicy reports ONTAP volume-path namespace (ai_knowledge/..., underscore). The KB S3 ingestion prefix (ai-knowledge/, hyphen) is a different namespace. Initial implementation confused the two, causing false-skip. The EventBridge rule and Lambda secondary filter now use a dedicated FPOLICY_PATH_FILTER parameter for the volume-path namespace.

Hybrid RAG: Internal KB + Web Search (opt-in)

GA at AWS Summit NYC 2026 (June 17, 2026). Powered by AgentCore Web Search Tool.

Enterprise knowledge from FSx for ONTAP is treated as the primary internal source, while public web context is supplemental and untrusted. For questions that benefit from current external context — regulatory updates, market trends, public market information — the Query Lambda can optionally augment answers with real-time web search results.

How It Works

Internal KB retrieval (always): Bedrock KB searches S3 Vectors for relevant chunks from FSx for ONTAP documents
Web search (opt-in): AgentCore Gateway invokes Amazon's purpose-built web index via MCP protocol
Unified answer: Bedrock Converse merges both contexts, with internal documents as primary source

Key Design Decisions

Decision	Rationale
Opt-in (`EnableWebSearch=false` default)	Most enterprise QA needs internal data only
Graceful degradation	Web Search failure → internal-only answer (no error surfaced to user)
Cross-region (us-east-1 Gateway)	Web Search Tool is us-east-1 only; adds ~100-200ms latency
Query safety	Only user's question text is sent to Web Search — never internal document content
Citation separation	`[Internal: filename]` vs `[Web: title](URL)` — users see exactly which source informed each claim
Prompt injection defense	Web results wrapped in `<web_search_results>` with explicit "untrusted data" instruction
Acceptable Use compliance	Source URLs and titles are always displayed (Web Search Tool TOS requirement)

Deployment

sam deploy --parameter-overrides \
  EnableWebSearch=true \
  AgentCoreGatewayId=<gateway-id> \
  AgentCoreGatewayRegion=us-east-1

Example Response

{
  "status": "completed",
  "query": "What are the latest FISC guidelines for cloud data protection?",
  "answer": "Based on internal documentation, our current FISC compliance posture covers... Additionally, [Web: FISC 2026 Guidelines Update](https://example.com/fisc-2026) published last month introduces...",
  "citations": [{"source": "s3://.../legal/compliance/fisc-overview.pdf"}],
  "web_citations": [{"source": "https://example.com/fisc-2026", "title": "FISC 2026 Guidelines Update", "type": "web"}],
  "web_search_enabled": true
}

Verification Highlights

Windows-Identity S3 Access Point with Dedicated AD

The Windows EC2 domain-joined to AWS Managed Microsoft AD, with the FSx SMB share mapped — proving the literal drag & drop experience works end to end.

To demonstrate the literal Windows drag & drop experience, we built a dedicated AWS Managed Microsoft AD + domain-joined Windows EC2 + AD-joined SVM:

AD-joined SVM OU: AWS Managed AD's OU=Computers lacks delegation rights → use the domain-name OU (OU=<domain>,DC=...)
CIFS share creation: Executes against the filesystem management LIF, not the SVM LIF
Windows-identity S3 AP: Works correctly with a running dedicated AD; files dropped in Explorer are readable via S3 AP

Deletion Lifecycle

The Bedrock KB data source connected to the FSx for ONTAP S3 AP alias. Click "Sync" for manual ingestion, or let Scenario B/C automate it.

Scenario B's Step Functions workflow: detect changes → start ingestion → poll status → notify on completion.

"User deletes a file → AI forgets it" verified end-to-end: file deletion → next sync → numberOfDocumentsDeleted=1 → re-query returns "no information found". Powered by dataDeletionPolicy=DELETE. For urgent revocation between syncs, call the Ingestion API directly.

Performance Considerations

Shared bandwidth: S3 AP reads share the FSx throughput capacity (128/256/512 MBps) with NFS/SMB workloads. Scenario B's 15-minute interval and Scenario C's reserved concurrency (2) throttle ingestion flow
Bulk re-index: For full re-ingestion (e.g., embedding model change), use a FlexClone volume as the Ingestion target — zero impact on production I/O, consistent point-in-time read
Tiering: Frequently accessed AI knowledge should remain on the SSD tier. Capacity Pool retrieval latency affects GetObject time during ingestion
Web Search latency: Cross-region call to us-east-1 adds ~100-200ms. Total hybrid query latency depends on KB size, model, and network conditions (KB retrieve + Web Search + Converse generation)

Access Control — Three Layers

S3 AP boundaries are volume/prefix-level. For per-user visibility:

Search narrowing = Bedrock KB metadata filters (this UC; not AWS authorization)
Document-level ACL = Amazon Quick S3 Knowledge Base (UC30; user/group-level)
Chunk-level permission filter = Custom Permission-Aware RAG (FC3; AD SID/NTFS ACL for regulated industries)

Web Search results are public information — no ACL filtering needed. However, the unified answer that combines internal + web sources is subject to the same access control as internal-only answers (the internal citations remain permission-scoped).

Vector Store: Why S3 Vectors

This pattern uses Amazon S3 Vectors as the Bedrock KB vector store. OpenSearch Serverless remains a valid option when its operational and latency profile fits the workload better.

Criterion	OpenSearch Serverless	S3 Vectors
Minimum monthly cost	~$175 (2 OCU)	Pay-per-use only
Cost at scale	OCU-based	Cost savings for large vector datasets (see AWS documentation)
Metadata filtering	Supported	Supported (department, owner, role)
Permission-Aware RAG compatibility	Supported	Compatible with metadata-filtered retrieval designs; authorization enforced by application layer
Infrastructure management	Managed but OCU scaling required	Managed vector operations
Scale	Millions of vectors	2 billion vectors per index
Query latency	Sub-100ms	Sub-100ms

For this project — 28 industry patterns + PoC-to-production lifecycle — S3 Vectors' pay-per-use model is the right fit. We evaluated Bedrock Managed Knowledge Base (GA June 2026, AWS Summit NYC) but chose Custom KB + S3 Vectors for cost control, ACL metadata flexibility, and FSx for ONTAP lifecycle integration (see ADR: docs/investigations/managed-kb-vs-custom-kb-s3vectors.md).

Data Classification

Output	Classification	Rationale
KB vectors + metadata	INTERNAL	Inherits source file classification
Ingestion job status / SNS	INTERNAL	Operational metadata only
CloudWatch Metrics / Logs	INTERNAL	Aggregate metrics, no file content
Web Search results	PUBLIC	External public information
Hybrid answer (internal + web)	INTERNAL	Contains internal document citations

For regulated workloads (CUI / FISC / HIPAA), extend shared/data_classification.py labels. If retention-period requirements apply, use dataDeletionPolicy=RETAIN and design a separate purge procedure.

Cost

Component	Monthly estimate	Notes
Lambda (sync + query)	< $5	Serverless pay-per-use
S3 API (ListObjects, GetObject)	< $1	S3 AP reads
EventBridge Scheduler	< $1	15-min interval
Bedrock KB Ingestion	Usage-based	Per-document embedding
S3 Vectors	Usage-based	Compare with OpenSearch Serverless for your query volume, latency, and operations requirements
Bedrock LLM (query)	Usage-based	Nova Pro: $0.0008/1K input tokens
FPolicy Server (Scenario C)	~$35	ECS Fargate (set desiredCount=0 when idle)
AgentCore Web Search (opt-in)	Per-query pricing (see AgentCore pricing)	Gateway invocation pricing
Cross-region transfer (opt-in)	< $0.02	us-east-1 ↔ ap-northeast-1

Getting Started

git clone https://github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns.git
cd FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns/solutions/genai/kb-selfservice-curation

# Install dependencies (shared modules used by Lambda handlers)
pip install -r requirements.txt        # or: uv pip install -r requirements.txt

# Review parameters
cat samconfig.toml.example

# Build and deploy (requires configured AWS credentials + FSx for ONTAP S3 AP)
sam build && sam deploy --guided

# DemoMode=true runs without FSx for ONTAP (regular S3 bucket)

# Optional: Enable Web Search hybrid RAG
sam deploy --parameter-overrides \
  EnableWebSearch=true \
  AgentCoreGatewayId=<gateway-id> \
  AgentCoreGatewayRegion=us-east-1

Governance Note

This article is technical architecture guidance, not legal, compliance, or regulatory advice. Pricing, regional availability, and benchmark numbers are time-sensitive; verify them against current AWS documentation before production use. S3 AP data source boundaries are at volume/prefix granularity — for per-user visibility control, consider Custom Permission-Aware RAG. If retention-period requirements (NARA / FISC) apply, use dataDeletionPolicy=RETAIN and design purge procedures separately. Web Search Tool usage requires compliance with the Acceptable Use Policy (source citations must be displayed).

Yoshiki Fujiwara

From Logs to Detection: Building a File Security Pipeline in Datadog for FSx for ONTAP

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 14 Jun 2026 16:44:19 +0000

TL;DR

For security teams: Four threshold monitors + one ML anomaly monitor + four Cloud SIEM detection rules catch mass file deletion, data exfiltration, permission tampering, and unusual geography in under 5 minutes. EventID codes become human-readable operation names. All created via Datadog API — no manual clicking required.

For engineers: The Log Pipeline (6 processors including GeoIP enrichment) transforms raw ONTAP XML events into searchable, alertable fields. Five Saved Views cover the most common investigation patterns. Total AWS cost remains ~$1.50/month.

For architects: This pattern demonstrates the full detection-to-response lifecycle — from raw file system audit events through enrichment, categorization, alerting, cross-service correlation, and automated remediation — entirely serverless, entirely as code.

FSx for ONTAP → S3 Access Point → Lambda → Datadog Logs API v2
                                              │
                                              ▼
                                   ┌──────────────────────┐
                                   │ Log Pipeline (6)     │
                                   │  • Category Processor│
                                   │  • Status Remapper   │
                                   │  • Date Remapper     │
                                   │  • Attribute Remapper│
                                   │  • GeoIP Enrichment  │
                                   └──────────┬───────────┘
                                              │
                          ┌───────────────────┼───────────────────┐
                          ▼                   ▼                   ▼
                ┌─────────────────┐ ┌─────────────────┐ ┌─────────────────┐
                │ Log Monitors (4)│ │ Cloud SIEM (4)  │ │ Log Archive     │
                │ + Anomaly (1)   │ │ Security Signals│ │ → S3 → Glacier  │
                └────────┬────────┘ └────────┬────────┘ └─────────────────┘
                         │                   │
                         ▼                   ▼
                ┌──────────────────────────────────────┐
                │ Workflow → Slack + Case + Lambda     │
                │          (ONTAP Snapshot remediation)│
                └──────────────────────────────────────┘

Reading guide: Sections 1-9 cover the core pipeline (30 min to deploy). Sections 10+ cover advanced SOC integration (additional 15 min). Jump to Deployment if you want to start immediately.

Business value: This pipeline reduces mean-time-to-detect (MTTD) for file-based threats from "hours/days" (batch log review) to under 5 minutes (real-time alerting). For regulated environments, it provides continuous compliance evidence with automated archival — replacing manual quarterly reviews with always-on monitoring.

This is Part 16 of Serverless Observability for FSx for ONTAP.

Why Post-Ingestion Processing Matters

In Part 1, we shipped raw audit events to Datadog. That's necessary but insufficient. Raw events look like this:

{
  "event_type": "4660",
  "user": "CORP\\contractor-ext-03",
  "path": "/share/engineering/designs/prototype-v3.dwg",
  "result": "Audit Failure"
}

Questions that raw logs can't answer quickly:

What is EventID 4660? (Answer: Object Delete)
Should this alarm? (Answer: depends on volume and context)
Who should investigate? (Answer: Storage team + SOC)
What's the user's baseline? (Answer: need faceted historical view)

This article builds the processing layers that transform raw data into actionable security intelligence.

Architecture

Why EventBridge Scheduler? FSx for ONTAP S3 Access Points do not support S3 Event Notifications or EventBridge object-level events. Lambda polls on a schedule and uses checkpointing to process only new files. XML format is chosen for Lambda-native parsing without binary dependencies (vs EVTX which requires Windows-specific libraries).

┌──────────────────────────────────────────────────────────────────┐
│ Lambda Handler (Python 3.12)                                     │
│  • Parse XML → Normalize → HEC-compatible JSON                   │
│  • POST to Datadog Logs API v2                                   │
│  • Fields: event_type, user, path, client_ip, svm, result        │
└──────────────────────────────────────────────────────────────────┘
        │  HTTP 202
        ▼
┌──────────────────────────────────────────────────────────────────┐
│ Datadog Log Pipeline: "FSx for ONTAP Audit Logs"                 │
│ Filter: source:fsxn                                              │
│                                                                  │
│ 1. Category Processor → @operation_name                          │
│    4663→Object Access, 4660→Object Delete, 4656→Handle Request   │
│                                                                  │
│ 2. Status Remapper → log severity from @result                   │
│    "Audit Success" → info, "Audit Failure" → error               │
│                                                                  │
│ 3. Date Remapper → @timestamp as official log time               │
│                                                                  │
│ 4. Attribute Remapper → @user→usr.id, @client_ip→network.client  │
└──────────────────────────────────────────────────────────────────┘
        │
        ▼
┌──────────────────────────────────────────────────────────────────┐
│ Datadog Security Monitors (3)                                    │
│                                                                  │
│ • [FSxN] Mass File Deletion    → >50 deletes/5min per user       │
│ • [FSxN] Abnormal Access Volume → >1000 accesses/1h per user     │
│ • [FSxN] Access Failure Spike  → >10 failures/15min per user     │
└──────────────────────────────────────────────────────────────────┘

The Log Pipeline

Why a Pipeline?

Without a pipeline, every query requires remembering that 4660 means "delete" and 4663 means "access." With a pipeline, you search @operation_name:Object Delete — and Datadog handles the translation at ingest time.

Creating the Pipeline via API

import json, urllib3, boto3

sm = boto3.client('secretsmanager', region_name='ap-northeast-1')
api_key = sm.get_secret_value(SecretId='fsxn-datadog-api-key')['SecretString']
app_key = sm.get_secret_value(SecretId='datadog/fsxn-app-key')['SecretString']

http = urllib3.PoolManager()

pipeline = {
    "name": "FSx for ONTAP Audit Logs",
    "is_enabled": True,
    "filter": {"query": "source:fsxn"},
    "processors": [
        {
            "type": "category-processor",
            "name": "EventID to Operation Name",
            "is_enabled": True,
            "target": "operation_name",
            "categories": [
                {"filter": {"query": "@event_type:4663"}, "name": "Object Access"},
                {"filter": {"query": "@event_type:4656"}, "name": "Handle Request"},
                {"filter": {"query": "@event_type:4660"}, "name": "Object Delete"},
                {"filter": {"query": "@event_type:4670"}, "name": "Permission Change"},
                {"filter": {"query": "@event_type:4658"}, "name": "Handle Close"},
                {"filter": {"query": "@event_type:5140"}, "name": "Share Access"},
                {"filter": {"query": "@event_type:5145"}, "name": "Share Check"},
                {"filter": {"query": "@event_type:4624"}, "name": "Logon"},
                {"filter": {"query": "@event_type:4634"}, "name": "Logoff"},
            ]
        },
        {
            "type": "status-remapper",
            "name": "Map result to log status",
            "is_enabled": True,
            "sources": ["result"]
        },
        {
            "type": "date-remapper",
            "name": "Use event timestamp",
            "is_enabled": True,
            "sources": ["timestamp"]
        },
        {
            "type": "attribute-remapper",
            "name": "Map user to usr.id",
            "is_enabled": True,
            "sources": ["user"],
            "source_type": "attribute",
            "target": "usr.id",
            "target_type": "attribute",
            "preserve_source": True,
            "override_on_conflict": False
        },
        {
            "type": "attribute-remapper",
            "name": "Map client_ip to network.client.ip",
            "is_enabled": True,
            "sources": ["client_ip"],
            "source_type": "attribute",
            "target": "network.client.ip",
            "target_type": "attribute",
            "preserve_source": True,
            "override_on_conflict": False
        }
    ]
}

resp = http.request("POST",
    "https://api.ap1.datadoghq.com/api/v1/logs/config/pipelines",
    body=json.dumps(pipeline).encode("utf-8"),
    headers={
        "Content-Type": "application/json",
        "DD-API-KEY": api_key,
        "DD-APPLICATION-KEY": app_key
    })
print(f"HTTP {resp.status}: Pipeline ID = {json.loads(resp.data)['id']}")

Site note: Replace api.ap1.datadoghq.com with your Datadog site's API endpoint (e.g., api.datadoghq.com for US1, api.datadoghq.eu for EU1).

EventID Mapping Table

EventID	Operation Name	MITRE ATT&CK	Description
4663	Object Access	T1005 (Data from Local System)	File read/write operation
4656	Handle Request	T1005	File handle opened
4660	Object Delete	T1485 (Data Destruction)	File deleted
4670	Permission Change	T1222 (File Permissions Modification)	ACL/permission modified
4658	Handle Close	—	File handle closed (low signal)
5140	Share Access	T1021.002 (SMB/Windows Admin)	SMB share connected
5145	Share Check	T1135 (Network Share Discovery)	Share permission checked
4624	Logon	T1078 (Valid Accounts)	Authentication event
4634	Logoff	—	Session ended

Security Monitors

Monitor 1: Mass File Deletion

mass_delete = {
    "name": "[FSxN] Mass File Deletion Detected",
    "type": "log alert",
    "query": 'logs("source:fsxn @event_type:4660").index("*").rollup("count").by("@user").last("5m") > 50',
    "message": """## Mass File Deletion Alert

A user has triggered more than 50 file deletion events within 5 minutes.

**User**: {{@user}}  |  **Count**: {{value}}

### Investigation Steps
1. Check affected paths: `source:fsxn @event_type:4660 @user:{{@user}}`
2. Verify if this is a scheduled cleanup or authorized bulk operation
3. Check the client IP for unexpected sources
4. Correlate with user's normal deletion patterns

@slack-storage-alerts""",
    "tags": ["source:fsxn", "team:storage", "severity:high"],
    "options": {
        "thresholds": {"critical": 50, "warning": 20},
        "notify_no_data": False,
        "renotify_interval": 60,
        "evaluation_delay": 60
    }
}

Why 50? In our test environment, normal daily deletions per user stay under 10. The threshold should be tuned based on your environment's baseline — run source:fsxn @event_type:4660 | stats count by @user over a week to establish what's normal.

Monitor 2: Abnormal Access Volume

abnormal_access = {
    "name": "[FSxN] Abnormal Access Volume",
    "type": "log alert",
    "query": 'logs("source:fsxn @result:\\"Audit Success\\"").index("*").rollup("count").by("@user").last("1h") > 1000',
    "message": """## Abnormal Access Volume Alert

More than 1000 successful file access events in 1 hour.
May indicate data exfiltration or unauthorized bulk access.

**User**: {{@user}}  |  **Count**: {{value}}

### Investigation Steps
1. Review patterns: `source:fsxn @user:{{@user}}`
2. Check if backup jobs or batch operations are running
3. Verify client IP and compare with known locations
4. Check accessed paths for sensitive content

@slack-security-alerts""",
    "options": {
        "thresholds": {"critical": 1000, "warning": 500},
        "evaluation_delay": 60
    }
}

Tuning note: Backup service accounts (svc-backup, svc-indexer) generate legitimate high-volume access. Exclude them with @user:-svc-* in the query or create a suppression rule.

Monitor 3: Access Failure Spike

failure_spike = {
    "name": "[FSxN] Access Failure Spike",
    "type": "log alert",
    "query": 'logs("source:fsxn @result:\\"Audit Failure\\"").index("*").rollup("count").by("@user").last("15m") > 10',
    "message": """## Access Failure Spike

More than 10 access failures in 15 minutes.
May indicate unauthorized access attempts or permission misconfiguration.

**User**: {{@user}}  |  **Count**: {{value}}

### Investigation Steps
1. Check failed paths: `source:fsxn @result:"Audit Failure" @user:{{@user}}`
2. Verify if permissions were recently changed
3. Check if user is accessing resources outside their scope
4. Correlate with AD group membership changes

@slack-security-alerts""",
    "options": {
        "thresholds": {"critical": 10, "warning": 5},
        "evaluation_delay": 60
    }
}

Saved Views

Five pre-configured views for common investigation patterns:

View	Query	When to use
FSxN File Deletions	`source:fsxn @event_type:4660`	Monitor triggered, investigating which files
FSxN Access Failures	`source:fsxn @result:"Audit Failure"`	Permission denied investigation
FSxN All Events	`source:fsxn`	General audit stream
FSxN Sensitive Share Access	`source:fsxn (@path:finance OR @path:hr OR @path:legal)`	Sensitive data access review
FSxN After-Hours Access	`source:fsxn` (filtered by time)	Off-hours activity detection

Each view includes pre-configured columns (user, path, client_ip, event_type) so analysts don't need to customize the table every time.

Facets for One-Click Filtering

Facets add clickable filters to the left sidebar. Create them from any log entry's detail panel:

Facet	Field	Why
Event Type	`@event_type`	Filter to specific operations (4660=delete, 4663=access)
Operation	`@operation_name`	Human-readable version (after pipeline applies)
User	`@user`	Filter by specific user under investigation
SVM	`@svm`	Isolate events to specific storage virtual machines
File Path	`@path`	Filter by directory or share
Client IP	`@client_ip`	Filter by source workstation/server
Result	`@result`	Quick split between success and failure events

Note: Fields are searchable with @field:value syntax even without Facets. Facets add UI convenience — they're not required for alerting or saved views.

Cost Reality

Component	Monthly Cost	Notes
Lambda (5-min schedule, ~1s execution)	~$0.50	8,640 invocations/month
Secrets Manager (2 secrets)	~$0.80	API key + APP key
EventBridge Scheduler	~$0.00	Free tier covers this
S3 AP reads	~$0.05	Depends on file count
Datadog Forwarder Lambda	~$0.10	CloudTrail event processing
S3 Archive storage	~$0.02	Glacier tier after 30 days
AWS total	~$1.50/month
Datadog log ingestion	~$0.10/GB	Conventional pricing
Total (1GB/month)	~$1.60/month

Datadog pricing caveat: Log ingestion pricing varies by plan (Conventional vs On-Demand) and contract. The ~$0.10/GB is the list price for conventional plans; actual cost depends on your contract, committed volume, and retention choices.

Cloud SIEM pricing: Cloud SIEM is a separate paid feature (~$0.20/GB analyzed for log detection). A 30-day free trial is available. Security Signals, Detection Rules, and the triage workflow require Cloud SIEM. Log Monitors (threshold alerts to Slack) work without it.

Cost optimization patterns

Filter before shipping — Drop low-value EventIDs (4658 Handle Close) at the Lambda level to reduce ingestion volume
Compress payloads — Enable gzip on the HTTP POST to reduce egress
Tune audit scope — Configure ONTAP vserver audit to monitor only the shares that matter
Index retention — Use Datadog's flexible retention (15/30/60/90 days) to match compliance needs vs cost

E2E Verification Results

Verified on Datadog AP1 (ap1.datadoghq.com) with paid plan, June 2026:

Component	Status
Lambda → Datadog Logs API v2	✅ HTTP 202
Log Pipeline (5 processors)	✅ Applied automatically
Category Processor (9 EventIDs)	✅ `@operation_name` populated
Status Remapper	✅ Failure events marked as error
Date Remapper	✅ Event timestamp used (not ingest time)
Attribute Remapper (usr.id)	✅ Enables Datadog user features
Mass Delete Monitor	✅ ALERT triggered (55 events > threshold 50)
Access Failure Monitor	✅ ALERT triggered (12 events > threshold 10)
Abnormal Access Monitor	✅ Active (OK — high threshold by design)
Anomaly Monitor (ML)	✅ Active (learning baseline)
Saved Views (5)	✅ Accessible from Views dropdown
Facets (8 custom)	✅ Created via UI
Dashboard (10 widgets)	✅ FSx ONTAP Audit Log Overview
Log-based Metrics (4)	✅ `fsxn.audit.*` in Metrics Explorer
Sensitive Data Scanner (5 rules)	✅ PII detection active
Cloud SIEM Detection Rules (4)	✅ Security Signals generated
CloudTrail Forwarder	✅ Logs arriving in Datadog
GeoIP Enrichment	✅ Country/city populated on client_ip
GuardDuty Correlation Rule	✅ Cross-service detection active
Workflow Automation	✅ Monitor → Slack + Case
Case Management (FSXN project)	✅ Project + template case
SOC Triage Runbook	✅ 7-step Notebook
Log Archive (S3 → Glacier)	✅ source:fsxn archived
Snapshot Remediation Lambda	✅ Deployed (fsxn-snapshot-remediation)

Log-based Metrics

Log-based metrics extract numerical values from logs at ingest time — you get metric-resolution dashboards and anomaly detection without paying for log retention beyond the evaluation window.

# Created via Datadog API (no UI needed)
metrics = [
    "fsxn.audit.delete_count",         # Delete events grouped by user/SVM
    "fsxn.audit.access_failure_count",  # Failures by user/SVM/client_ip
    "fsxn.audit.event_count",           # All events by event_type/SVM
    "fsxn.audit.unique_users",          # Active users
]

Use cases:

Anomaly detection on fsxn.audit.delete_count — Datadog's built-in anomaly monitor catches unusual deletion patterns without manual thresholds
SLO tracking — Define an SLO on access failure rate (fsxn.audit.access_failure_count / fsxn.audit.event_count)
Cost efficiency — Metrics are retained for 15 months at metric resolution (vs log retention at full-text cost)

Sensitive Data Scanner

Audit logs frequently contain PII in file paths (/hr/EMP-123456-performance-review.xlsx) and user contexts. Datadog's Sensitive Data Scanner catches these before they reach analysts.

Rule	What It Catches	Why It Matters
Employee ID (`EMP-\d{6}`)	Employee records in paths	PII exposure prevention
JP Phone (`0[789]0-\d{4}-\d{4}`)	Mobile numbers in filenames	Privacy protection
Email in Path	Customer emails in file names	Data minimization
Credit Card	CC numbers in documents	PCI-DSS technical control
My Number	Japanese national ID	Sensitive PII detection

Compliance note: Sensitive Data Scanner is a technical detection and redaction control. It does not constitute full APPI/GDPR/PCI-DSS compliance, which requires organizational measures (consent management, purpose limitation, data subject rights, etc.). Consult your legal/compliance team for regulatory assessment.

Matched patterns are partially redacted at ingest — the first 3 characters remain for identification, the rest is replaced with [REDACTED]. This preserves investigation capability while protecting the data subject.

Enhanced Dashboard

The SOC-focused dashboard provides at-a-glance visibility into:

Who is generating the most activity (top users, top IPs)
What operations dominate (sunburst by operation/SVM)
Where access is concentrated (hot paths)
When failures spike (timeline with per-user breakdown)
Trend from log-based metrics (delete rate over time)

Cloud SIEM Security Signals

Beyond log monitors (which notify via Slack/PagerDuty), Security Signals integrate with Datadog's SOC workflow — triage, investigation, and response all in one place.

Log Monitors vs Cloud SIEM Detection Rules

Aspect	Log Monitors	Cloud SIEM Detection Rules
Output	Alert notification (Slack/PagerDuty/email)	Security Signal (appears in Security Signals panel)
Triage	Manual (click alert → investigate)	Built-in triage workflow (Open → In Progress → Archived)
MITRE mapping	Manual tags	Native MITRE ATT&CK framework integration
Correlation	Single query	Cross-log-source correlation possible
Case integration	None	Direct "Create Case" from signal
Best for	Operational alerts (DevOps/SRE)	Security investigation (SOC/IR teams)

Recommendation: Use both. Log Monitors for immediate Slack notification to storage team. Cloud SIEM Detection Rules for SOC triage workflow and MITRE-mapped investigation.

Three detection rules generate Security Signals when FSxN audit logs match threat patterns:

Rule	MITRE ATT&CK	Trigger	Severity
Mass File Deletion	T1485 Data Destruction	>50 deletes/5min per user	Critical
Brute Force File Access	T1110 Brute Force	>20 failures/15min per user+IP	High
Permission Tampering	T1222 File Permissions Modification	>5 changes/10min per user	High

# Creating a Security Signal rule via API
rule = {
    "name": "FSxN: Mass File Deletion",
    "type": "log_detection",
    "queries": [{
        "query": "source:fsxn @event_type:4660",
        "groupByFields": ["@user"],
        "aggregation": "count",
        "name": "delete_events",
        "dataSource": "logs"
    }],
    "cases": [
        {"condition": "delete_events > 50", "status": "critical", "name": "Critical"},
        {"condition": "delete_events > 20", "status": "high", "name": "High"}
    ],
    "options": {
        "evaluationWindow": 300,
        "keepAlive": 3600,
        "maxSignalDuration": 86400,
        "detectionMethod": "threshold"
    },
    "message": "User {{@user}} deleted {{value}} files in 5 min. T1485.",
    "tags": ["source:fsxn", "technique:T1485-data-destruction"]
}

Each signal includes investigation steps and response guidance directly in the signal panel — analysts don't need to leave the Security Signals view to understand what happened.

Cloud SIEM Setup Steps

Navigate to Security → Cloud SIEM → Get Started
Skip Content Packs (FSxN is a custom source)
Select fsxn as a log source → Enable as Trial
Configure Cloud SIEM Index (default: 450 days retention)
Detection rules are automatically applied to incoming source:fsxn logs

Workflow Automation

When a critical monitor fires, the Workflow automatically triggers response actions:

Monitor Alert → @workflow-fsxn-security-alert-response → Slack notification + Investigation links

The workflow (fsxn-security-alert-response) is linked to monitors via the @workflow-<handle> mention syntax in the monitor message. No additional configuration needed — just add the mention to any monitor's notification body.

Extensions (via Datadog Workflow builder):

Auto-create Jira ticket with investigation context
Query Active Directory for user's manager
Invoke fsxn-snapshot-remediation Lambda for evidence preservation (deployed — see Automated Snapshot Remediation)

Investigation Notebook

A pre-built 5-step investigation template guides analysts through alert triage:

Step	Content	Widget
1. Identify Scope	Event type distribution timeline	Timeseries (bars)
2. User Timeline	Delete events by user over time	Timeseries (line)
3. Affected Files	Top 20 deleted paths	Top List
4. Client IP Analysis	Top 10 source IPs	Top List
5. Conclusion	Root cause, impact, action template	Markdown

Access at: Notebooks → "FSxN Audit Log Investigation Template"

Each step includes pre-configured queries — analysts just adjust the time window and user filter to match the alert context.

RBAC and Access Control

The FSxN Security Analyst role provides scoped access to audit logs:

Role	Permissions	Use Case
FSxN Security Analyst	`logs_read_data`, `logs_read_index_data`	SOC analysts investigating file access
Datadog Standard Role	Full access	Storage admins managing pipeline

Assign users to this role to grant log access without exposing infrastructure metrics or APM data. Combined with Saved Views, analysts see only the investigation tools relevant to their role.

OTel Collector Bridge

For teams already running OpenTelemetry Collector, an alternative delivery path routes FSxN logs through OTel with trace context injection:

# otel-bridge/collector-config.yaml (excerpt)
receivers:
  otlp:
    protocols:
      http: { endpoint: 0.0.0.0:4318 }

processors:
  resource:
    attributes:
      - key: service.name
        value: ontap-audit
        action: upsert
  transform:
    log_statements:
      - context: log
        statements:
          - set(attributes["ddsource"], "fsxn")

exporters:
  datadog:
    api:
      key: ${env:DD_API_KEY}
      site: ${env:DD_SITE:-ap1.datadoghq.com}

Benefits over direct Lambda→Datadog:

trace_id injection for distributed tracing correlation
Multi-backend fanout (Datadog + Grafana + S3 in parallel)
Attribute enrichment at collector level (team, environment tags)
Sampling for high-volume environments
Edge-side PII redaction — Mask sensitive fields before they leave your AWS account (stronger than relying solely on Datadog's Sensitive Data Scanner)

Edge redaction pattern: Use OTel's transform processor to hash user and client_ip and truncate path to directory level BEFORE export to Datadog. This ensures PII never crosses your network boundary — even if Datadog's SDS configuration is misconfigured or disabled, your data is already protected at the source.

Status: Config syntax verified. Integration testing requires deploying OTel Collector on ECS (see Part 7 for the ECS-based Collector pattern used with Grafana and Honeycomb).

Log Archives and Compliance

For regulatory retention (adjust based on your organization's audit policy), a dedicated CloudFormation template deploys an S3 archive with Glacier lifecycle:

aws cloudformation deploy \
  --template-file integrations/datadog/template-log-archive.yaml \
  --stack-name fsxn-datadog-archive \
  --parameter-overrides \
    DatadogExternalId=<from-datadog-integration-page> \
    RetentionDays=30 \
    GlacierRetentionDays=2555 \
  --capabilities CAPABILITY_NAMED_IAM

This separates concerns:

Detection — Hot logs in Datadog (15-30 day index retention)
Compliance — Cold archive in S3 → Glacier (7+ years)
Investigation — Rehydrate specific time ranges on demand

Anomaly Detection (ML-based)

Beyond static thresholds, the anomaly monitor on fsxn.audit.delete_count uses Datadog's agile algorithm to learn each user's baseline:

# Anomaly monitor — no manual threshold needed
"avg(last_4h):anomalies(
    sum:fsxn.audit.delete_count{*} by {user}.as_count(),
    'agile', 3, direction='above'
) >= 1"

When a user who normally deletes 5 files/day suddenly deletes 200, the alert fires — even though 200 is well below the static "50 in 5 minutes" threshold. This catches slow, sustained exfiltration that threshold-based monitors miss.

Baseline period: The anomaly algorithm needs ~2 weeks of data to build confidence. Deploy early — it stays silent during the learning period.

Cardinality Management

⚠️ Log-based metrics with group_by: user create one time series per unique user. For organizations with 10,000+ users:

Metric	Recommended group_by	Reason
`fsxn.audit.event_count`	svm only	Broad metric, low cardinality
`fsxn.audit.delete_count`	user, svm	Targeted, high signal
`fsxn.audit.access_failure_count`	user, svm, client_ip	Investigation-focused
`fsxn.audit.unique_users`	user	Intentionally per-user

Monitor cardinality in Metrics Summary (fsxn.audit.*). Datadog bills per unique tag-value combination — keep per-user grouping only on metrics where individual user behavior matters.

What's Different from Splunk/CrowdStrike?

Aspect	Datadog	Splunk (Part 8)	CrowdStrike (Part 15)
Delivery protocol	Logs API v2 (JSON)	HEC (JSON)	HEC (JSON)
Pipeline config	API-driven (Python)	props.conf / UI	LogScale parser YAML
Monitor creation	API (`POST /api/v1/monitor`)	Saved searches + alerts	CQL + alert actions
EventID mapping	Category Processor	eval/lookup	LogScale parser
Identity correlation	`usr.id` attribute	`user` field	Falcon Identity
Free tier	❌ (paid only for logs)	❌	❌
API management	✅ Full (Pipeline + Monitor + Dashboard)	Partial	Limited

The key Datadog advantage: everything is API-manageable. Pipeline, monitors, dashboards, and saved views can all be created, updated, and version-controlled through the Datadog API — making infrastructure-as-code for observability fully achievable.

Production Recommendations

Use the API, not the UI — Version-control your Pipeline, Monitor, and Dashboard definitions in a setup script. When you need to adjust thresholds or add EventIDs, it's a code change with review.
Separate APP key from API key — The API key ships logs; the APP key manages configuration. Store both in Secrets Manager with different IAM policies.
Exclude service accounts — Add @user:-svc-* to monitor queries or create Datadog suppression rules for known batch accounts.
Start with Warning, promote to Critical — Deploy monitors with Warning thresholds first, observe for 1-2 weeks, then tighten to Critical after confirming the baseline.
Connect to incident workflow — Route Critical monitors to PagerDuty/Slack and include investigation links (Saved View URLs) in the alert message.
Enable Log Archives for compliance — Configure S3 archive with Glacier lifecycle for FISC/SOC2 retention requirements. Rehydrate on demand for historical investigation.
Use Workflow Automation for response — Configure Datadog Workflows to automatically create Jira tickets, trigger Slack quick-actions, or invoke Lambda remediation when critical monitors fire.
Manage with Terraform for production — For enterprise/multi-org deployments, use the Datadog Terraform Provider to version-control Pipelines, Monitors, and Detection Rules. Separate API key (logs_write only) from APP key (admin-level, CI/CD pipeline only).
Apply IAM Permissions Boundary — In large organizations, apply a Permissions Boundary to the log-shipping Lambda role to prevent privilege escalation. The boundary should allow only s3:GetObject, s3:ListBucket, secretsmanager:GetSecretValue, and logs:* (CloudWatch).
Encrypt with customer-managed keys (CMK) — For regulated environments, enable Datadog Log Management Encryption with your KMS CMK, and configure the S3 archive bucket with SSE-KMS. This ensures audit logs are encrypted with keys you control at every stage.
Tune detection rules (anti-alert-fatigue) — Run all new Cloud SIEM rules in "Warning / dev" status for 2 weeks. Whitelist known service accounts with @usr.id:-svc-* in queries. Review false positives weekly and adjust thresholds or add suppression rules before promoting to Critical.
Data Integrity: Handle HTTP 429/5xx — Datadog Logs API v2 returns HTTP 202 (accepted, not indexed). On HTTP 429 (rate limit) or 5xx (transient), implement exponential backoff (base 1s, max 5 retries). After final failure, route the complete HEC payload to SQS DLQ for replay. Never drop logs silently.

Deployment

# 1. Deploy the CloudFormation stack (Lambda + Scheduler + DLQ + Alarms)
aws cloudformation deploy \
  --template-file integrations/datadog/template.yaml \
  --stack-name fsxn-datadog-integration \
  --parameter-overrides \
    FsxS3AccessPointArn=arn:aws:s3:<region>:<account>:accesspoint/<name> \
    DatadogApiKeySecretArn=<secret-arn> \
    DatadogSite=ap1.datadoghq.com \
  --capabilities CAPABILITY_NAMED_IAM

# 2. Deploy full observability (Pipeline + Monitors + Metrics + Scanner)
export DD_API_KEY_SECRET_ID="fsxn-datadog-api-key"
export DD_APP_KEY_SECRET_ID="datadog/fsxn-app-key"
export DD_SITE="ap1.datadoghq.com"
bash integrations/datadog/scripts/setup-full-observability.sh

# 3. (Optional) Deploy Log Archive for compliance
aws cloudformation deploy \
  --template-file integrations/datadog/template-log-archive.yaml \
  --stack-name fsxn-datadog-archive \
  --parameter-overrides DatadogExternalId=<external-id> \
  --capabilities CAPABILITY_NAMED_IAM

# 4. Create Facets (guided manual step)
bash integrations/datadog/scripts/setup-facets.sh

# 5. Enable Cloud SIEM (UI: Security → Cloud SIEM → Get Started → select fsxn)

Time from zero to full detection capability: ~45 minutes.

Pre-Deployment Checklist

Before deploying to production, confirm:

[ ] FSx audit configuration confirmed (XML format, rotation schedule)
[ ] S3 Access Point created and Lambda role authorized
[ ] Datadog API Key + APP Key stored in Secrets Manager
[ ] Datadog site region confirmed (AP1/US1/EU1)
[ ] Data classification sign-off: audit logs contain user PII (usernames, IPs, file paths) — confirm external transmission is approved per organization policy
[ ] Retention requirements defined (Datadog index retention + S3 archive lifecycle)
[ ] On-call/escalation path defined for Critical signals
[ ] Service accounts identified for monitor exclusion (svc-*)
[ ] Cloud SIEM trial or paid plan confirmed (for Security Signals)
[ ] Network path validated (VPC-external Lambda or NAT Gateway for S3 AP access)

Lessons Learned

1. API-first beats UI-first

Creating the Pipeline via API took 30 seconds. Doing it through the UI takes 5 minutes of clicking through dropdowns. More importantly, the API approach is repeatable, reviewable, and version-controlled.

2. Facets are UX, not functionality

Every field is searchable with @field:value whether or not you create a Facet. Facets just add sidebar convenience. Don't block on Facet setup — your monitors work without them.

3. Category Processor is the key transformation

Without it, analysts need to memorize Windows EventID codes. With it, @operation_name:Object Delete is immediately understandable. This single processor provides more investigative value than any other pipeline step.

4. Evaluation delay prevents false positives

The 60-second evaluation_delay on monitors accounts for ingestion latency. Without it, you get spurious alerts when log batches arrive slightly out of order.

5. Saved Views are investigation shortcuts

When a monitor fires at 3 AM, the on-call engineer needs to investigate immediately. Pre-built Saved Views with the right columns and filters eliminate setup time during incidents.

6. Anomaly detection needs baseline data

The anomaly monitor requires at least 2 weeks of historical data to build a reliable baseline. Deploy it early — it won't generate false positives during the learning period (it simply stays silent until confident).

7. Cloud SIEM and Log Monitors are complementary

Log Monitors give you fast Slack/PagerDuty notifications for the on-call team. Cloud SIEM Detection Rules give SOC analysts a structured triage workflow with MITRE mapping. Deploy both — they serve different audiences.

8. GeoIP is free threat intelligence

The GeoIP Processor is zero-cost and instantly useful. Even in a pure on-premises environment, unusual geography flags VPN misconfiguration, compromised credentials, or travel you weren't expecting.

9. Forwarder deployment enables an ecosystem

Once the Datadog Forwarder Lambda is deployed, enabling CloudTrail, GuardDuty, Lambda logs, or any other AWS log source is a single S3 notification configuration — no new infrastructure needed.

CloudTrail Correlation (Cross-Service Detection)

A dedicated detection rule correlates FSxN audit events with AWS CloudTrail context. When the signal fires, the investigation message includes pre-built queries for cross-service correlation:

source:cloudtrail @userIdentity.arn:*<username>*     → IAM actions
source:guardduty                                      → GuardDuty findings
source:vpc @network.client.ip:<suspicious-ip>         → Network flows

The rule (FSxN: Suspicious Deletion After IAM Role Assumption) fires on >30 deletions in 10 minutes and embeds a correlation checklist:

Was there a recent AssumeRole or ConsoleLogin from an unusual IP?
Did the user's permissions change in the last 24 hours?
Is the client IP in the corporate VPN range?
Are there GuardDuty findings for this account?

Note: This rule activates fully once CloudTrail logs are flowing to Datadog via the AWS Integration. The FSxN detection works immediately; CloudTrail queries return results after integration setup.

SOC Triage Runbook

A 7-step Notebook guides analysts from signal to resolution:

Step	Action	Output
1. Signal Context	Verify user, IP, time, volume	Timeseries widget
2. User Verification	Check AD status, role changes, maintenance windows	Manual checklist
3. Impact Assessment	Identify affected files and directories	Top List (paths)
4. Cross-Service Correlation	CloudTrail, GuardDuty, VPC Flow Logs	Query templates
5. Decision Matrix	Map conditions to response actions	Decision table
6. Response Actions	Disable account, snapshot, notify, ticket	Checklist
7. Post-Incident	Update thresholds, document, lessons learned	Checklist

Accessible at: Notebooks → "FSxN Security Signal Triage Runbook"

Case Management

Cases provide structured investigation tracking across the SOC team:

Project: FSXN — All FSx for ONTAP security investigations
Case creation: Manual from Security Signals, or auto-create via Workflow
Priority levels: P1 (critical mass deletion) through P4 (informational)
Lifecycle: Open → In Progress → Resolved → Closed

When a critical signal fires, the workflow creates a case automatically with:

Signal link and context
Affected user and IP
Investigation notebook link
Response checklist

Threat Intelligence (GeoIP Enrichment)

The Log Pipeline's GeoIP Processor automatically enriches every FSxN log with geographic data from the client_ip field:

@client_ip: 10.0.5.99 → @network.client.geoip: { country: "JP", city: "Tokyo", ... }

A detection rule flags access from unexpected countries:

# Rule: "FSxN: File Access from Unusual Geography"
# Query: source:fsxn -@network.client.geoip.country:JP -@network.client.geoip.country:US
# Threshold: >5 events/15min from non-JP/US IPs

This catches compromised credentials being used from abroad without needing an external threat intelligence feed — Datadog's built-in GeoIP database handles the enrichment at ingest time.

GuardDuty Correlation

A detection rule correlates FSxN deletion events with GuardDuty findings from the same source IP:

FSxN: >10 deletions/30min from IP X
  + GuardDuty finding for IP X (UnauthorizedAccess, Recon, Trojan)
  = Critical Security Signal with full context

The rule's investigation message includes pre-built GuardDuty queries:

source:guardduty @detail.resource.instanceDetails.networkInterfaces.privateIpAddress:{{@client_ip}}

GuardDuty findings flow to Datadog via the same Forwarder Lambda that handles CloudTrail — no additional setup needed.

Automated Snapshot Remediation

When mass deletion is confirmed (Critical signal reviewed by analyst), the Workflow invokes fsxn-snapshot-remediation Lambda:

Security Signal (Critical) → Analyst confirms → Workflow → Lambda → ONTAP REST API → Snapshot

The Lambda:

Receives volume name, SVM, and reason from the Workflow
Authenticates to ONTAP via Secrets Manager credentials
Creates a timestamped snapshot: remediation_20260614_215530_mass_deletion
Returns snapshot name and status for the Case record

# Lambda invocation payload (from Datadog Workflow)
{
    "volume_name": "finance_share",
    "svm_name": "ProductionSVM",
    "reason": "Mass deletion detected",
    "user": "CORP\\suspicious-user"
}

Configuration: Set ONTAP_MGMT_IP and ONTAP_CREDENTIALS_SECRET_ARN environment variables on the Lambda to point to your FSx for ONTAP management endpoint.

TLS note: The Lambda uses cert_reqs="CERT_NONE" for ONTAP REST API calls because FSx for ONTAP uses self-signed certificates by default. In production, upload the ONTAP CA certificate to the Lambda layer (/opt/certs/ontap-ca.pem) and configure urllib3.PoolManager(ca_certs="/opt/certs/ontap-ca.pem") to validate the connection.

Snapshot Storm Prevention (Cooldown)

To prevent runaway Snapshot creation during sustained mass-deletion events:

# Before creating snapshot, check for recent remediation snapshots
existing = http.request("GET",
    f"https://{mgmt_ip}/api/storage/volumes/{vol_uuid}/snapshots"
    f"?name=remediation_*&order_by=create_time desc&max_records=1",
    headers=base_headers)

if existing.status == 200:
    snaps = json.loads(existing.data).get("records", [])
    if snaps:
        last_snap_time = snaps[0].get("create_time", "")
        # Skip if a remediation snapshot was created in the last 15 minutes
        if is_within_cooldown(last_snap_time, minutes=15):
            return {"statusCode": 200, "body": "Skipped — cooldown active"}

ONTAP limits: Each volume supports up to 1023 Snapshots. The cooldown prevents hitting this limit during sustained attack scenarios.

Remediation Audit Trail

The Lambda invocation itself must be auditable — proving that the Snapshot was created by an authorized pipeline, not a rogue actor:

CloudTrail: Lambda invocation recorded with InvokedBy: workflow.datadoghq.com
ONTAP audit log: Snapshot creation appears as an administrative event with the API user
Datadog Case: Snapshot name and status recorded in the Case timeline
Lambda CloudWatch Logs: Full request/response logged with correlation ID

What's Next

This article covered the complete detection-to-response lifecycle. Future enhancements:

Datadog Threat Intel feeds — When Datadog makes the Threat Intel Indicators API available on AP1, feed internal IP reputation lists for richer enrichment
Multi-SVM snapshot orchestration — Extend the remediation Lambda to snapshot across multiple SVMs in parallel
Automated account lockout — Chain the Workflow to invoke AD lockout via Systems Manager Run Command

GitHub Repository

Shipping FSx for ONTAP Audit Logs to CrowdStrike Falcon LogScale via HEC — Parser v1.1.0

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 14 Jun 2026 16:28:49 +0000

Scope note: This article targets CrowdStrike Falcon LogScale HEC ingestion via Amazon FSx for ONTAP S3 Access Points. For on-premises ONTAP or Cloud Volumes ONTAP, the audit log retrieval path differs (e.g., NFS/SMB export or FPolicy), while the parser and HEC delivery model can still be reused. Falcon Next-Gen SIEM connector-based ingestion (via the Falcon console Data Connectors UI) may require a different setup and should be validated separately.

Which path applies to you? If your tenant exposes LogScale repositories and ingest tokens, use the HEC path in this article. If your tenant uses Falcon Next-Gen SIEM Data Connectors, implement this as a connector/parser workflow and validate separately.

TL;DR
Why CrowdStrike LogScale?
Architecture
The HEC Protocol: One Format, Multiple Destinations
Parser v1.1.0: Designed for Speed and Maintainability
FIELD_MAPPING: Zero-Code Adaptation to ONTAP Changes
S3 Access Point Permission Model
Compatibility Verification (Splunk HEC)
LogScale CQL Query Examples
Performance Benchmarks
Deployment
Cost
Data Classification and Privacy
Token and Egress Governance
Production Readiness Checklist
HEC Path vs OTel / Alloy Path
Pipeline Observability
Production Checkpoint Design
Splunk HEC Compatibility Notes
Lessons Learned
What's Next
Partner Discovery Checklist

Validated / Not Yet Validated

Area	Status
Parser unit tests	✅ Verified (108 tests)
Splunk HEC compatibility	✅ Verified
LogScale HEC live ingest	⏳ Pending required LogScale entitlement
LogScale CQL queries	📝 Syntax draft, pending live validation
CrowdStrike UI screenshots	⏳ Pending required LogScale entitlement
Deployment parameterization	✅ Verified — endpoint, HEC path, source, sourcetype, index, memory, timeout are configurable

Live LogScale validation plan (when tenant is available):

Ingest token permission and authentication
HEC ingest success and HTTP response
Repository assignment and data visibility
Parser assignment and field extraction
Timestamp correctness (@timestamp = event time)
CQL query validation against live data
Alert trigger and notification
Dashboard screenshot capture
Fusion SOAR workflow trigger test

TL;DR

For decision makers: CrowdStrike Falcon LogScale receives FSx for ONTAP audit logs via the same HEC (HTTP Event Collector) protocol as Splunk. This means reduced switching cost — you can move between LogScale, Splunk, or any HEC-compatible SIEM without changing your Lambda code. Note: while the HEC wire format is shared, repository/index semantics, parser configuration, query language (CQL vs SPL), alerting rules, and retention policies differ between platforms and require per-platform configuration. Parser v1.1.0 processes 178,000 events/second in parser-only benchmarks. AWS cost: validation estimate ~$1/month under low-volume assumptions.

CrowdStrike pricing note: CrowdStrike licensing and third-party ingestion entitlement vary by contract and product edition. Confirm availability, daily ingest quota, retention, and pricing with your CrowdStrike account team before production planning.

For NetApp teams: This pattern keeps audit data on ONTAP and adds a serverless security analytics path through S3 Access Points, without changing existing SMB/NFS client access or requiring data copies to S3.

For engineers: template.yaml deploys the full stack (Lambda + Scheduler + DLQ + Alarm) in one aws cloudformation deploy command. The parser uses a FIELD_MAPPING table — new ONTAP field names require zero code changes.

FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda
    → CrowdStrike Falcon LogScale (/api/v1/ingest/hec)

This is Part 15 of Serverless Observability for FSx for ONTAP.

Why CrowdStrike LogScale?

Three reasons this combination makes sense:

Unified XDR + File Audit: Falcon EDR endpoint telemetry and FSx file access logs in the same platform. A contractor accessing sensitive files on the NAS? LogScale correlates that with their endpoint behavior.
Index-free architecture: LogScale's index-free architecture reduces index management overhead and is designed for high-scale search workloads. Validate search performance against your own retention, query patterns, and ingest volume.
HEC compatibility: LogScale accepts a Splunk HEC-compatible JSON envelope at /api/v1/ingest/hec. If you're already using Splunk HEC somewhere, the HEC-style JSON envelope is largely reusable, while parser, repository, and query behavior should still be validated per destination.

Architecture

┌─────────────────────────────────────────────────────────────────┐
│ FSx for ONTAP                                                   │
│   vserver audit create -format xml                              │
│   → Audit log files written to audit volume                     │
└──────────────┬──────────────────────────────────────────────────┘
               │ S3 Access Point (read-only)
               ▼
┌─────────────────────────────────────────────────────────────────┐
│ EventBridge Scheduler (every 1–5 minutes)                       │
│   → Invokes Lambda                                              │
└──────────────┬──────────────────────────────────────────────────┘
               │
               ▼
┌─────────────────────────────────────────────────────────────────┐
│ Lambda (Python 3.12, ARM64, 256MB)                              │
│   1. Read new XML files from S3 AP                              │
│      - PoC: checkpoint in SSM Parameter Store                   │
│      - Production: DynamoDB conditional checkpoint              │
│   2. Parse XML → normalize → HEC format                         │
│   3. POST to LogScale /api/v1/ingest/hec                        │
│      Authorization: Bearer <ingest-token>                       │
└──────────────┬──────────────────────────────────────────────────┘
               │ HTTPS (gzip optional)
               ▼
┌─────────────────────────────────────────────────────────────────┐
│ CrowdStrike Falcon LogScale                                     │
│   Repository: fsxn_audit                                        │
│   → Search, dashboards, alerts, correlation with EDR data       │
└─────────────────────────────────────────────────────────────────┘

Real-time path (FPolicy)

For sub-second latency (e.g., ransomware detection), use the FPolicy path:

ONTAP → FPolicy TCP:9898 → ECS Fargate → SQS → Lambda → LogScale

FPolicy sends file operation notifications in real-time — no polling required.

Ransomware response: Treat audit-log polling as an investigation and evidence pipeline. For prevention or near-real-time response, evaluate ONTAP FPolicy external mode and Autonomous Ransomware Protection (ARP) alongside this LogScale integration.

ONTAP Audit Log Lifecycle

Validate ONTAP audit log rotation behavior before production deployment:

How frequently audit logs are rotated (rotate-size / rotate-schedule-minute)
Whether Lambda reads only closed (rotated) files or also active files
Whether existing audit tools require EVTX and whether parallel XML output is feasible
How S3 Access Point LastModified maps to ONTAP file close time
Expected audit volume size per rotation and FSx throughput impact

Format choice: This pipeline uses XML (-format xml) because it can be parsed with Python standard library in Lambda without additional dependencies. EVTX is the FSx for ONTAP default and works well with Windows Event Viewer, but requires EVTX-specific parsing libraries in the Lambda package or Layer.

Existing audit tools: If you already use a batch-based audit log tool that expects EVTX, verify format compatibility before changing ONTAP audit settings. This serverless pipeline is designed for detection and investigation; it is not a drop-in replacement for tools that provide audit summarization, compression, retention workflows, or compliance report templates. Consider a complementary deployment where the existing tool handles audit reporting and this pipeline handles SOC detection and Falcon correlation.

The HEC Protocol: One Format, Multiple Destinations

CrowdStrike LogScale's HEC endpoint (/api/v1/ingest/hec) accepts a Splunk HEC-compatible JSON envelope. The only differences:

Aspect	CrowdStrike LogScale	Splunk
Endpoint	`/api/v1/ingest/hec`	`/services/collector/event`
Auth header	`Authorization: Bearer <token>`	`Authorization: Splunk <token>`
Payload body	Same HEC-style JSON envelope; validate parser/repository behavior per platform	Same HEC-style JSON envelope

This means:

The same Lambda handler can be reused for both by changing the URL, auth header, and destination-specific configuration
You can switch between LogScale and Splunk without modifying event formatting
Testing against a local Splunk Docker validates the HEC payload shape and provides a strong compatibility signal for LogScale; live LogScale ingest still requires tenant validation

HEC Event Format

{
  "time": 1780710900.0,
  "event": {
    "timestamp": "2026-06-06T01:55:00.000000Z",
    "event_type": "4663",
    "source": "fsxn-ontap",
    "svm": "ProductionSVM",
    "user": "CORP\\user-finance-01",
    "client_ip": "10.0.1.50",
    "operation": "File",
    "path": "/share/finance/quarterly-reports/Q2-2026-revenue.xlsx",
    "result": "Audit Success",
    "s3_key": "audit/2026-06-06/events.xml",
    "log_format": "xml"
  },
  "source": "fsxn-ontap",
  "sourcetype": "fsxn:audit:xml",
  "index": "fsxn_audit"
}

time field: Per LogScale HEC docs: "time: Time in seconds since January 1, 1970 in UTC. Is translated to @timestamp on ingestion." Always include this top-level field to ensure correct event timestamp assignment in LogScale. If omitted, LogScale uses ingest time instead of event time.

Multi-tenant note: For MSSP or multi-account deployments, add bounded tenant metadata such as aws_account_id, environment, and customer_alias inside the event object. Avoid putting high-cardinality values into metric dimensions.

Identity enrichment: For stronger Falcon Identity / ITDR correlation, consider splitting user into user_domain and user_name fields, and enriching with SID / UPN / email where possible. IP-only correlation (client_ip) can be unreliable in DHCP, VPN, NAT, and VDI environments.

Portability note: For maximum portability across HEC-compatible destinations, this example keeps searchable attributes inside the event JSON object rather than relying on Splunk-specific fields behavior. LogScale auto-parses JSON event objects into searchable fields. Validate parser and field extraction behavior in your LogScale tenant before depending on any destination-specific metadata fields.

Parser v1.1.0: Designed for Speed and Maintainability

The shared parser (fsxn_log_parser) handles EVTX, XML, and JSON formats. Key design decisions in v1.1.0:

Universal Entry Point

from fsxn_log_parser import parse

result = parse(data, key="audit-2026-06-06.xml", metrics=my_callback)
# result.events  → list of AuditEvent
# result.format  → "xml"
# result.parse_duration_ms → 2.8
# result.event_count → 500

One function handles format detection, parsing, normalization, and metrics — all automatically.

Format Detection (Strategy Pattern)

from fsxn_log_parser import detect_format, register_format

# Built-in: evtx (magic bytes), xml (< prefix), json ([/{ prefix)
fmt = detect_format(data, key="file.xml")  # → "xml"

# Extensible: register custom formats without modifying core code
register_format("custom", lambda data, key: key.endswith(".custom"))

Detection inspects only the first 8–64 bytes — O(1) regardless of file size.

Performance Optimizations

Technique	Impact
`rpartition("}")` for namespace stripping	~15% faster than `split("}")[-1]`
`child.get("Name")` direct access	Avoids `.attrib` dict creation
`BytesIO` for streaming iterparse	Better buffer management
Local `get = event.get` binding	Reduces attribute lookups
Char-count heuristic for streaming threshold	Avoids O(n) `.encode()`
`struct.unpack_from` on buffer	Zero-copy EVTX parsing

XML Parsing and XXE Hardening

try:
    import defusedxml.ElementTree as SafeET
    _parse_xml_string = SafeET.fromstring
except ImportError:
    # Fallback: stdlib ET.fromstring (no XXE protection —
    # for production with untrusted XML, add defusedxml to your Lambda Layer)
    _parse_xml_string = ET.fromstring

FIELD_MAPPING: Zero-Code Adaptation to ONTAP Changes

The biggest maintainability win in v1.1.0:

FIELD_MAPPING = {
    "timestamp": ["TimeCreated_SystemTime", "timestamp"],
    "event_type": ["EventID", "event_type"],
    "svm":       ["Computer", "SVMName", "svm"],
    "user":      ["SubjectUserName", "UserName", "user"],
    "client_ip": ["IpAddress", "ClientIP", "client_ip"],
    "operation": ["ObjectType", "Operation", "operation"],
    "path":      ["ObjectName", "path"],
    "result":    ["Keywords", "Result", "result"],
}

When ONTAP introduces a new field name (e.g., in a version upgrade), you update this table — no code changes required:

# Support new ONTAP field name
FIELD_MAPPING["user"].insert(0, "NewOntapUserField")

The normalize_event function resolves fields by iterating candidates left-to-right, returning the first non-empty value.

S3 Access Point Permission Model

FSx for ONTAP S3 Access Points use a dual-layer authorization model. Per AWS documentation, both the access point policy AND the underlying file system identity permissions must permit the request.

Layer 1: IAM Policy (Lambda execution role)

The Lambda execution role must have s3:GetObject and s3:ListBucket permissions on the access point ARN:

Resource:
  - arn:aws:s3:<region>:<account>:accesspoint/<name>/object/*   # GetObject
  - arn:aws:s3:<region>:<account>:accesspoint/<name>            # ListBucket

Layer 2: S3 Access Point Resource Policy

The access point itself must have a resource policy granting access to the Lambda execution role. Use s3control put-access-point-policy to configure.

Layer 3: File System Identity

FSx for ONTAP maps S3 API calls to NFS/SMB identity. The NFS export policy or NTFS ACLs on the audit volume must allow read access for the mapped identity.

Key implication: If your Lambda gets AccessDenied despite correct IAM, check the access point resource policy and the file system export policy. All three layers must allow the request.

Troubleshooting ListObjectsV2 AccessDenied: Verify both the access point ARN resource (for s3:ListBucket) and the object ARN resource (for s3:GetObject) are present in the Lambda role policy, and confirm the access point resource policy explicitly allows the Lambda execution role.

For production, enforce read-only access consistently across the Lambda IAM policy, the S3 Access Point resource policy, and the mapped FSx file-system identity.

In multiprotocol environments, validate the volume or qtree security style and effective permissions for the S3 Access Point file-system identity. Mixed security-style volumes can produce unexpected access results if effective permissions differ by path.

Compatibility Verification (Splunk HEC)

Note: Live LogScale ingest validation is pending the required CrowdStrike LogScale entitlement.

Since CrowdStrike LogScale's free trial does not include HEC ingest capability (Data Connectors require a paid Next-Gen SIEM license), we validated using Splunk Enterprise Docker — which accepts the same HEC-style JSON envelope used by this implementation.

Step	Result
XML audit log parsing (5 events)	✅ EventID 4663/4656/4660
HEC delivery	✅ HTTP 200 `{"text":"Success","code":0}`
Indexing	✅ `fsxn_audit` index, all events searchable
Field extraction	✅ user, path, client_ip, event_type, result, svm
Splunk Search UI	✅ All fields parsed and filterable

The HEC-style JSON envelope used in this implementation is compatible with Splunk HEC and provides a strong compatibility signal for LogScale HEC. A successful Splunk HEC test validates payload shape, timestamp metadata, and basic delivery behavior. It does not replace live LogScale tenant validation.

LogScale CQL Query Examples

Note: The CQL examples below are intended as starting points. Validate in your LogScale repository after parser assignment and field extraction are confirmed.

Correlation Use Cases

Once audit events are in LogScale, investigate alongside Falcon endpoint telemetry:

Did the same user trigger Falcon endpoint detections near the file access spike?
Did the client host run compression, staging, or cloud upload tools before high-volume reads?
Did identity risk increase before first-seen access to sensitive shares?

Audit Event Count vs Client I/O Count

For SMB, audit event counts are not identical to raw client I/O counts. ONTAP may suppress repeated read/write events on the same object to avoid excessive logging (e.g., EventID 4663 records only the first SMB read and first SMB write per handle). Treat detection thresholds as audit-event baselines, not storage I/O baselines.

SOC Detection Examples

The detection queries below are starter patterns. Validate CQL syntax, field extraction, threshold values, and business context in your LogScale tenant before enabling alerts. Do not enable these as production alerts without baseline tuning — thresholds should be adjusted per share, user population, and normal business activity. The time-bucketing syntax below is illustrative; adjust to your LogScale CQL version and repository parser behavior.

Mass file deletion detection

#repo=fsxn_audit event_type="4660"
| bucket(span=5m)
| groupBy([_bucket, user, client_ip], function=count())
| _count > 50
| sort(_count, order=desc)

First-seen access to sensitive share

#repo=fsxn_audit path=/share/finance/* OR path=/share/hr/* OR path=/share/legal/*
| groupBy([user], function=[min(@timestamp, as=first_seen), count()])
| first_seen > now() - 24h
| sort(first_seen, order=desc)

This detects first-seen access within the repository retention window, not necessarily the user's first access ever.

High-volume file access (possible exfiltration)

High-volume reads may also be caused by backup, indexing, migration, or legitimate batch processing. Correlate with known job schedules before escalating.

#repo=fsxn_audit result="Audit Success"
| bucket(span=1h)
| groupBy([_bucket, user, client_ip], function=count())
| _count > 1000
| sort(_count, order=desc)

Response Automation with Fusion SOAR

After a detection is validated and tuned, connect LogScale alerts to Falcon Fusion SOAR workflows for enrichment and response orchestration. A typical workflow:

Enrich the user and client host context
Check Falcon detections or incidents in the same time window
Check known backup, indexing, or migration job schedules
Notify the SOC channel and storage owner
Escalate to containment or identity action only after human approval

NFS-Friendly Detection Note

For NFS-heavy environments, prefer normalized fields such as operation, path, user, and client_ip rather than Windows-specific event_type values. NFS operations use operation names (e.g., REMOVE, RENAME, READ) that map differently from SMB EventIDs.

Find all file access by a specific user

#repo=fsxn_audit user="CORP\\user-finance-01"
| table([@timestamp, event_type, path, result, client_ip])

Detect unusual access patterns (after-hours access)

#repo=fsxn_audit
| parseTimestamp(field=timestamp, format="yyyy-MM-dd'T'HH:mm:ss")
| hour := formatTime(field=@timestamp, format="HH")
| hour > "19" OR hour < "07"
| groupBy([user, path], function=count())
| sort(_count, order=desc)

Files not accessed in 30 days

#repo=fsxn_audit
| groupBy(path, function=[max(@timestamp, as=last_access)])
| last_access < now() - 30d
| sort(last_access, order=asc)

Failed access attempts (permission issues)

#repo=fsxn_audit result="Audit Failure"
| groupBy([user, path], function=count())
| _count > 3
| sort(_count, order=desc)

Performance Benchmarks

Caveat: The 178K events/sec result is a parser-only microbenchmark. End-to-end throughput depends on S3 Access Point read performance (tied to FSx provisioned throughput), Lambda memory, HEC batching, network latency, LogScale ingest quota, and retry behavior.

Environment: Lambda ARM64 (Graviton) 256MB, Python 3.12
Parser: v1.1.0 (FIELD_MAPPING, iterparse, XXE hardening when defusedxml is packaged)

┌─────────────────────┬──────────┬─────────────────────┐
│ Input               │ Time     │ Throughput          │
├─────────────────────┼──────────┼─────────────────────┤
│ 5 events (2KB)      │ 0.045ms  │ 111,000 events/sec  │
│ 500 events (135KB)  │ 2.8ms    │ 178,000 events/sec  │
│ 5,000 events (1.3MB)│ ~28ms    │ ~178,000 events/sec │
├─────────────────────┼──────────┼─────────────────────┤
│ 1M events/day       │ ~6 sec   │ total daily compute │
└─────────────────────┴──────────┴─────────────────────┘

At 178K events/sec, a 5-minute Lambda invocation can process ~53 million events — far exceeding any realistic FSx for ONTAP audit log volume.

Deployment

# 1. Store ingest token in Secrets Manager
aws secretsmanager create-secret \
  --name "crowdstrike/fsxn-ingest-token" \
  --secret-string '{"ingest_token":"<your-token>"}'

# 2. Deploy stack
aws cloudformation deploy \
  --template-file integrations/crowdstrike/template.yaml \
  --stack-name fsxn-crowdstrike-integration \
  --parameter-overrides \
    FsxS3AccessPointArn=arn:aws:s3:<region>:<account>:accesspoint/<name> \
    LogScaleIngestTokenSecretArn=<secret-arn> \
    LogScaleUrl=https://cloud.us.humio.com \
    HecPath=/api/v1/ingest/hec \
  --capabilities CAPABILITY_NAMED_IAM

# 3. Upload handler code (template creates a placeholder; upload the actual handler)
cd integrations/crowdstrike/lambda && zip function.zip handler.py
aws lambda update-function-code \
  --function-name fsxn-crowdstrike-integration-shipper \
  --zip-file fileb://function.zip

S3 Access Point design: When creating the access point, document the chosen file access type (read-only recommended — Lambda only needs to list and read converted audit logs), file-system user identity (UNIX for UNIX-style volumes, Windows for NTFS), and network configuration (VPC-restricted for production).

ONTAP pre-flight checks (run before Lambda deployment):

vserver audit show -instance and vserver audit show -fields destination,format,events — confirm audit is enabled with XML format

volume show -fields security-style,junction-path — verify audit volume

vserver security file-directory show-effective-permissions — verify read access for the S3 AP identity

Verify the S3 Access Point file-system identity can read the audit log path

Splunk HEC compatibility testing: For local validation without the required LogScale entitlement, set LogScaleUrl to your Splunk Docker endpoint (e.g., https://localhost:8088) and HecPath=/services/collector/event. The payload format is validated against the same HEC envelope.

defusedxml for production: For production XML parsing hardening, include defusedxml in the Lambda deployment package or Lambda Layer.

Recommended CI checks: cfn-lint for template, pytest for parser/handler, dependency vulnerability scan, secret scanning, malicious XML / XXE regression test, and local Splunk HEC integration test.

Time from zero to first HEC-compatible test event in Splunk Docker: ~30 minutes. Live LogScale first-event timing is pending tenant validation.

Production deployment: For production, version the Lambda artifact and promote it across environments through CI/CD. Keep the CloudFormation template, Lambda package, parser version, and dashboard definitions tied to the same release tag. Use SAM build/deploy or a CI pipeline instead of manual update-function-code.

Cost

Component	Monthly
Lambda (5-min schedule, ~1s avg execution)	~$0.50
Secrets Manager	~$0.40
EventBridge Scheduler	~$0.00
S3 AP reads	~$0.05
AWS total	~$1/month

Cost caveats: The AWS estimate assumes Lambda is not in a VPC requiring NAT Gateway, low CloudWatch Logs retention, no custom KMS key, and low retry volume. CrowdStrike licensing cost is excluded. S3 Access Point read cost and performance depend on request volume, object size, and the underlying FSx for ONTAP file system throughput configuration. Additional CloudWatch custom metrics, dashboards, alarms, and log retention may increase AWS cost depending on metric cardinality and retention settings.

LogScale cost, third-party ingestion entitlement, daily ingest quota, and retention depend on your CrowdStrike product edition and contract. Confirm these details with your CrowdStrike account team before production planning.

Daily ingest estimate: daily_gb = events_per_day * avg_event_size_bytes / (1024^3)

Cost optimization options:

Filter low-value event types only after compliance and incident-response sign-off (e.g., handle-close events may be high volume but useful in forensic timelines)
Truncate or hash high-cardinality fields to reduce event size
Compress HEC request payloads (gzip)
Tune ONTAP audit policy scope to relevant shares only

Data Classification and Privacy

FSx for ONTAP audit logs contain information that may be business-sensitive or subject to privacy regulations:

Field	Sensitivity	Example
`user`	PII — identifies individuals	`CORP\user-finance-01`
`client_ip`	Network topology exposure	`10.0.1.50`
`path`	Business-sensitive file paths	`/share/finance/quarterly-reports/Q2-2026-revenue.xlsx`
`svm`	Infrastructure naming	`ProductionSVM`
`timestamp`	Access pattern analysis	Working hours / after-hours

Considerations:

File paths may reveal business operations, project codenames, or M&A activity
Username + path + timestamp combinations enable detailed behavior profiling
IP addresses expose internal network topology
Sending this data to an external platform (CrowdStrike) requires appropriate data processing agreements
Evaluate whether PII redaction (e.g., via OTel Collector transform processor) is required before external transmission
Confirm your organization's data classification policy covers audit log content shipped to third-party SIEMs
Treat the SIEM repository itself as a sensitive data store — file paths, usernames, and timestamps can reveal business activity and user behavior

Future consideration for regulated environments: For healthcare, public sector, or financial services, consider a minimization mode that hashes or drops selected fields (user, client_ip, or full path) before external transmission, while preserving enough metadata for investigation. The OTel Collector transform processor or a pre-processing Lambda can implement field-level redaction.

Example minimization policy:

Keep: event_type, operation, result, svm, timestamp

Hash: user, client_ip

Truncate: path to directory level (replace file names with hashes)

FIELD_MAPPING Masking Strategy (v1.2.0 proposal)

Extend the existing FIELD_MAPPING table with a per-field action parameter:

FIELD_MAPPING = {
    "timestamp": {"keys": ["TimeCreated_SystemTime", "timestamp"], "action": "keep"},
    "user":      {"keys": ["SubjectUserName", "UserName"],         "action": "hash"},
    "client_ip": {"keys": ["IpAddress", "ClientIP"],               "action": "mask_subnet"},
    "path":      {"keys": ["ObjectName", "path"],                  "action": "truncate_dir"},
    "event_type": {"keys": ["EventID", "event_type"],              "action": "keep"},
    "result":    {"keys": ["Keywords", "Result"],                   "action": "keep"},
}

Actions:

keep: Pass through unchanged
hash: hashlib.sha256(salt + value).hexdigest()[:16] — preserves correlation without exposing identity
mask_subnet: 10.0.1.50 → 10.0.1.0/24 — hides host, preserves network segment
truncate_dir: /share/finance/Q2-revenue.xlsx → /share/finance/[REDACTED]

Operational requirement: Maintain a secure lookup table (separate from SIEM) mapping hashed values to originals. This enables authorized investigators to de-anonymize during incident response with dual-approval.

Salt management: The hash salt MUST NOT be hardcoded in Lambda code or environment variables. Store it in AWS Secrets Manager (or derive from KMS GenerateDataKey). Lambda retrieves and caches the salt at cold start via the same auth_cache pattern used for HEC tokens. Rotate the salt quarterly; maintain a salt history table to support lookups against older hashes.

Audit Log Immutability (S3 Object Lock)

For compliance environments requiring tamper-evident audit trails:

S3 Object Lock (COMPLIANCE mode): Once set, even root cannot delete objects before the retention period expires. Suitable for SEC 17a-4, FINRA, FISC.
Governance mode: Allows privileged users to override (suitable for internal policy enforcement without regulatory mandate).

# Enable Object Lock on the audit log bucket (must be set at bucket creation)
aws s3api create-bucket --bucket fsxn-audit-immutable \
  --object-lock-enabled-for-object-lock true

# Set default retention (COMPLIANCE mode, 7 years)
aws s3api put-object-lock-configuration --bucket fsxn-audit-immutable \
  --object-lock-configuration '{
    "ObjectLockEnabled": "Enabled",
    "Rule": {"DefaultRetention": {"Mode": "COMPLIANCE", "Years": 7}}
  }'

Token and Egress Governance

Token Management

Store in Secrets Manager: Never embed HEC ingest tokens in Lambda environment variables or code. Use secretsmanager:GetSecretValue at runtime with caching.
Rotate periodically: Generate new ingest tokens in LogScale and update the Secrets Manager secret. The Lambda's auth_cache module handles reload-on-401/403 automatically.
Least-privilege token scope: Create a dedicated ingest token scoped to the fsxn_audit repository only — not a global token.

Egress Control

Restrict Lambda egress where feasible: If the Lambda runs in a VPC, route outbound traffic through an approved egress path (proxy, firewall, or controlled NAT) and restrict destinations to approved CrowdStrike LogScale endpoints according to your network policy.
Alarm on authentication failures: Set a CloudWatch Metric Filter + Alarm for HTTP 401/403 responses from the HEC endpoint. This detects token expiration, revocation, or misconfiguration early.

Monitoring

# Example: CloudWatch Metric Filter pattern for HEC auth failures
# { $.status_code = 401 || $.status_code = 403 }

Configure a CloudWatch Alarm that triggers on >0 auth failures in a 5-minute window to alert on token issues before data loss occurs.

Production Readiness Checklist

Before promoting this integration from PoC to production:

Validation

[ ] Validate live LogScale ingest — Confirm events appear in the fsxn_audit repository with correct field extraction (pending paid tenant)
[ ] Confirm LogScale repository and ingest token design — Create a dedicated repository, assign the expected parser to the ingest token, document token ownership, rotation, and emergency revocation
[ ] Confirm repository/parser/retention — Verify LogScale repository settings: assigned parser, retention period, ingest quota allocation
[ ] Set top-level HEC time — Ensure Lambda sets epoch-seconds time field on every HEC event (see HEC Event Format section)

Reliability

[ ] Define retry/DLQ/replay strategy — Configure SQS DLQ for failed HEC deliveries; document the replay procedure (see docs/en/runbooks/dlq-replay.md)
[ ] Validate IAM least privilege — Audit Lambda execution role: only s3:GetObject, s3:ListBucket on the AP, secretsmanager:GetSecretValue on the token secret. For checkpoint: PoC uses ssm:GetParameter/PutParameter; production uses DynamoDB (dynamodb:GetItem, PutItem, UpdateItem with condition expressions)
[ ] Confirm token rotation procedure — Test Secrets Manager secret rotation and verify Lambda handles token refresh without data loss
[ ] Estimate daily volume — Calculate expected daily ingest volume (events × avg event size) against the ingest quota and retention terms confirmed for your CrowdStrike tenant
[ ] Enable CloudWatch Alarms — DLQ depth > 0, Lambda error rate > 1%, checkpoint staleness > 15 minutes

Security and Governance

[ ] Confirm tenant isolation model — For MSSP or multi-account: define repository/index separation, token scope, IAM boundary, and customer-specific retention
[ ] Define audit evidence handling — Confirm retention, export procedure, chain-of-custody requirements, and how delivery gaps are detected and explained
[ ] Protect the audit log volume — Confirm audit log volume placement, backup/replication policy, retention, and recovery procedure. Do not place audit log destination on the SVM root volume (root volume content is not replicated in SVM DR). If audit logs are part of regulated evidence, align ONTAP audit log volume protection with the SIEM retention and evidence export policy
[ ] Data classification sign-off — Confirm audit log content classification and external transmission approval (see Data Classification section)
[ ] Egress governance — Implement token storage, rotation, egress restriction, and 401/403 alarming (see Token and Egress Governance section)
[ ] Harden XML parsing — Package defusedxml in the Lambda Layer or fail closed if unavailable (see XML Parsing and XXE Hardening section)
[ ] Define SIEM access governance — Confirm who can search FSx audit logs, who can export results, and how incident evidence is retained
[ ] Define audit reporting workflow — Identify recurring audit reports (monthly access summary, deletion tracking, sensitive folder access), export format, report owner, approval process, and retention period. See also: Existing Audit Tool Coexistence Guide
[ ] Validate audit format coexistence — If an existing audit tool requires EVTX, confirm whether a separate SVM or format migration strategy is needed (ONTAP does not support simultaneous EVTX and XML output per SVM). See coexistence guide

Observability

[ ] Validate delivery SLO measurement — Confirm LogFileAgeSeconds is a suitable proxy for audit log close-to-delivery lag in your environment
[ ] Emit pipeline health metrics — Use CloudWatch EMF or Lambda Powertools Metrics to emit files processed, events parsed, events sent, HEC success/failure, delivery latency, and log file age. Use SQS native metrics for DLQ depth

HEC Path vs OTel / Alloy Path

This article uses the HEC path because Falcon LogScale provides a Splunk HEC-compatible endpoint. For teams that standardize on OpenTelemetry, the same normalized audit events can also be routed through an OTel Collector or Grafana Alloy pipeline.

Use HEC when:

The target SIEM natively supports HEC (LogScale, Splunk)
You want a lightweight Lambda-to-SIEM delivery path
You are validating Splunk / LogScale compatibility

Use OTel / Alloy when:

You need multi-backend routing (e.g., Grafana + Splunk + custom)
You want field transformation, redaction, or sampling before export
You want a common telemetry contract across multiple backends

For OTel / Alloy-based routing, apply minimization before export: hash user and client_ip, truncate path to directory level, and use only bounded low-cardinality attributes such as event_type, operation, and result as metric dimensions when deriving metrics.

High-cardinality warning: Do not promote user, full path, or client_ip to metric labels. Keep them as log fields / event attributes. Use only bounded, low-cardinality dimensions such as environment, region, fsx_file_system_id, event_type, operation, and result. Use svm only if the number of SVMs is bounded and operationally meaningful.

Decision Matrix: Which Path to Choose?

Factor	→ HEC Direct	→ OTel/Alloy
Single SIEM destination (LogScale or Splunk)	✅	—
Minimize infrastructure components	✅	—
Multi-backend fanout needed	—	✅
Field redaction/hashing before external transmission	—	✅
Need trace_id correlation with application traces	—	✅
Already running OTel Collector in environment	—	✅
PoC / validation phase (simplicity first)	✅	—
Regulatory minimization required (hash PII at edge)	—	✅

Start with HEC, evolve to OTel: Most teams start with the HEC direct path (this article) for initial validation, then add an OTel Collector when multi-backend routing or pre-processing needs emerge. The normalized field schema is identical in both paths — switching is a routing change, not a data model change.

Pipeline Observability

Suggested Delivery SLO

99% of audit log files are delivered to the target SIEM within 10 minutes
of being closed by ONTAP. No checkpoint staleness over 15 minutes during
expected audit activity windows.

To measure this SLO, track LogFileAgeSeconds (current_time minus the audit
log object's last modified time from S3 AP). DeliveryLatencyMs only measures
Lambda-side processing and HEC response latency, not end-to-end lag.

Note: LogFileAgeSeconds is a practical proxy for end-to-end lag. Validate
how S3 Access Point LastModified maps to ONTAP audit log rotation / close
timing in your environment.

Pipeline Health Metrics

For production observability, emit pipeline health metrics using CloudWatch Embedded Metric Format (EMF) or AWS Lambda Powertools Metrics.

EMF dimensions: Avoid high-cardinality dimensions such as user, path, client_ip, and s3_key. Use bounded dimensions such as FunctionName, environment, and target.

Metric	Unit	Purpose
`FilesScanned`	Count	Files listed per invocation
`FilesProcessed`	Count	Files successfully parsed and shipped
`EventsParsed`	Count	Total audit events extracted
`EventsSent`	Count	Events successfully delivered to HEC
`HecSuccess`	Count	HTTP 2xx responses
`HecFailure`	Count	HTTP 4xx/5xx responses
`DeliveryLatencyMs`	Milliseconds	Time from S3 read to HEC response
`CheckpointAgeSeconds`	Seconds	Time since last checkpoint advance (derived from checkpoint state; recommended for production)
`DlqMessages`	Count	Messages in dead letter queue (from SQS native metric)
`LogFileAgeSeconds`	Seconds	Time since audit log file was last modified (measures SLO)

Gray Failure Detection

Watch for gray failures, not only hard failures:

HEC returns intermittent 429 / 5xx but Lambda doesn't error
Checkpoint advances slowly but does not stop
DLQ remains empty but delivery latency increases
Event count drops unexpectedly compared to historical baseline

Handle HEC 429 / 5xx with exponential backoff and bounded retries. Do not advance the checkpoint until delivery succeeds or the failed object is safely moved to DLQ for replay.

Observability as Code

For production, manage not only the Lambda pipeline but also dashboards, alarms, metric filters, and runbook links as code. The CloudFormation template should include:

CloudWatch Dashboard for pipeline health
Alarms for DLQ depth, Lambda errors, HEC 401/403, HEC 5xx, checkpoint staleness
Metric filters for authentication failures
Runbook links in alarm descriptions

Splunk HEC Compatibility Notes

For teams using this integration with Splunk (via HecPath=/services/collector/event):

Verification Coverage

Aspect	Status
HEC timestamp (`time` metadata)	✅ Reflected as event time
sourcetype assignment	✅ `fsxn:audit:xml` assigned
source assignment	✅ `fsxn-ontap` assigned
JSON field extraction	✅ All `event` fields searchable
Duplicate replay tolerance	📝 Pending production replay test

HEC Acknowledgement Gap

Splunk HEC provides Indexer Acknowledgement (/services/collector/ack) — a mechanism to confirm events are committed to disk, not just accepted into the ingestion pipeline.

Aspect	Splunk HEC	LogScale HEC
Delivery guarantee	`ackId` confirms disk write	HTTP 200 = accepted (best-effort)
Retry safety	Replay until ack received	No built-in replay mechanism
Data loss window	Zero (if ack used correctly)	Between acceptance and disk flush

Recommendation for production: Use a write-ahead pattern — persist events to S3 (or DynamoDB) before HEC delivery. Treat HEC as best-effort delivery. On failure, replay from the durable store. This pattern works identically for both Splunk and LogScale.

DLQ Replay Strategy

When HEC is unavailable and events accumulate in the DLQ (SQS):

Approach	Pros	Cons	Recommended
Replay DLQ JSON directly to HEC	Fast, no re-parsing	Requires DLQ messages to be complete HEC payloads	✅
Reset checkpoint and re-parse from S3	Guaranteed consistency	Slower, re-reads S3, may re-process already-delivered events	—

Recommended: Store the complete HEC JSON payload in DLQ messages. On recovery, drain the DLQ and POST each message directly to the HEC endpoint. This avoids re-parsing and ensures idempotent delivery (same time + event content).

See docs/en/runbooks/dlq-replay.md for the step-by-step procedure.

SPL vs CQL Query Comparison

For SOC analysts working across both platforms:

Operation	Splunk SPL	LogScale CQL
Time bucket (5 min)	`\	bin _time span=5m`
Top 10 users	`\	top limit=10 user`
Count by user	`\	stats count by user`
Filter + aggregate	`source="fsxn" event_type=4660 \	stats count by user`
Time range	`earliest=-1h latest=now`	Query time picker or `@timestamp > now() - 1h`
String match	`path="finance"`	`path = /share/finance/*` (glob)

Key difference: SPL uses a pipe-forward streaming model where each command transforms the event set sequentially. CQL uses a similar pipe model but with different function names and grouping semantics. The normalized field names (user, path, client_ip, event_type, result) are intentionally aligned so queries translate 1:1 in structure.

Production Checkpoint Design

For production, checkpoint should be advanced only after successful HEC delivery. Use DynamoDB conditional writes to avoid concurrent Lambda invocations advancing the same checkpoint.

DynamoDB Schema

Attribute	Type	Purpose
`file_path`	String (PK)	S3 key of the audit log file
`etag`	String	Object ETag for idempotency
`status`	String	PENDING → PROCESSING → COMPLETED / FAILED
`lease_expiry`	Number (epoch)	Auto-release time for ghost lock prevention
`event_count`	Number	Events successfully delivered
`updated_at`	String (ISO)	For staleness detection
`ttl`	Number (epoch)	Auto-delete COMPLETED records after 7 days

Lease-Based Concurrency Control

Lambda invocation
    │
    ├── DynamoDB ConditionExpression:
    │   "attribute_not_exists(file_path)
    │    OR #s = :failed
    │    OR lease_expiry < :now"
    │
    ├── [Success] → Set status=PROCESSING, lease_expiry=now+15m
    │                → Read S3 → Parse → Ship to HEC
    │                → Set status=COMPLETED
    │
    └── [ConditionalCheckFailed] → Skip (another instance owns it)

Ghost Lock Prevention

If Lambda crashes or times out, the lease_expiry attribute ensures another invocation can reclaim the file after 15 minutes:

# Acquire lease with ghost-lock prevention
table.update_item(
    Key={"file_path": s3_key},
    UpdateExpression="SET #s = :processing, lease_expiry = :expiry, updated_at = :now",
    ConditionExpression=(
        "attribute_not_exists(file_path) OR #s = :failed OR lease_expiry < :now"
    ),
    ExpressionAttributeNames={"#s": "status"},
    ExpressionAttributeValues={
        ":processing": "PROCESSING",
        ":failed": "FAILED",
        ":expiry": int(time.time()) + 900,  # 15 min lease
        ":now": datetime.utcnow().isoformat(),
    },
)

Alarm: Set a CloudWatch alarm when any item has status=PROCESSING for longer than 2× the lease period. This detects systematic Lambda failures that bypass the normal retry path.

Use updated_at to derive CheckpointAgeSeconds for dashboarding and checkpoint staleness alarms.

Multi-SVM / Multi-FileSystem Key Design

For large-scale environments with multiple FSx file systems or SVMs, use a composite partition key to avoid DynamoDB hot partitions:

PK: {fsx_file_system_id}#{svm_name}#{s3_key}

Example: fs-0123456789abcdef0#svm-prod#audit/2026-06-14/events-001.xml

This ensures DynamoDB distributes writes across partitions even at scale (100+ SVMs), avoiding RCU/WCU burst throttling on a single partition.

Catch-up Storm Prevention

After a Lambda outage (hours of accumulated unprocessed files), recovery can cause a burst of S3 AP read requests that impacts FSx ONTAP production workloads.

Mitigations:

Reserved Concurrency = 1: During catch-up, limit Lambda to a single concurrent execution to serialize file processing
Max files per invocation: Cap at 50 files per Lambda run; let the next scheduled invocation handle the rest
Backoff on queue depth: If DynamoDB shows >100 PENDING items, add a 5-second sleep between file reads
Runbook: Document the recovery procedure — temporarily increase ScheduleRate to rate(1 minute) while keeping concurrency at 1

Production Readiness Checklist item: Validate recovery behavior by simulating a 4-hour Lambda outage and measuring S3 AP read throughput during catch-up against the FSx provisioned throughput capacity.

Lessons Learned

1. HEC compatibility is a hidden superpower

By targeting the HEC protocol, we reduced the amount of destination-specific Lambda code needed for LogScale support. The payload envelope remains largely reusable, while repository, parser, query language, alerting, and retention settings still require LogScale-specific validation.

2. The trial doesn't include ingest

CrowdStrike's Falcon EDR trial includes the Next-Gen SIEM UI (read-only search, dashboards) but does NOT include Data Connectors / HEC ingest. The "Add data connector" page returns 404. CrowdStrike trial and licensing behavior may vary by product edition and contract. Confirm Falcon LogScale or Next-Gen SIEM ingest entitlement, daily quota, retention, and pricing with your CrowdStrike account team before production planning.

3. FIELD_MAPPING > hardcoded `.get()` chains

The v1.0.0 parser had 10 lines of chained .get() calls in normalize_event. The v1.1.0 FIELD_MAPPING table:

Makes field resolution self-documenting
Supports new ONTAP versions without code changes
Centralizes all field name knowledge in one place
Is still fast (inner loop uses local get binding)

4. Test with Docker, deploy to Cloud

Splunk Enterprise Docker (splunk/splunk:latest --platform linux/amd64) provides a fully functional HEC in 2 minutes. We used it to validate the HEC-style payload shape before live LogScale tenant validation — at $0 cost.

What's Next

CrowdStrike UI verification: When the required CrowdStrike LogScale or Next-Gen SIEM entitlement is available, capture search screenshots showing FSx audit events with full field extraction
Charlotte AI integration: Using natural language to query audit logs ("show me all file deletions by contractors this week")
Falcon Identity correlation: Cross-referencing file access with AD authentication events for insider threat detection
OTel / Alloy routing option: Add an optional pipeline that routes normalized audit events through OpenTelemetry Collector or Grafana Alloy for multi-backend delivery and field minimization
Existing audit tool coexistence: Explore complementary deployment where existing batch-based tools handle audit reporting and this pipeline handles SOC detection and Falcon correlation
ARP / FPolicy correlation: Correlate LogScale audit events with ONTAP Autonomous Ransomware Protection alerts and FPolicy notifications for ransomware investigation
Falcon content package: Package starter LogScale detections for mass delete, abnormal access volume, after-hours access, and repeated access failures — with metadata (name, required fields, CQL, thresholds, false positives, MITRE mapping, response, tuning notes), dashboards, lookup tables, and Fusion SOAR workflow templates for repeatable deployment
Identity-aware file access graph: Correlate FSx audit logs with AD / IdP authentication events, EDR telemetry, and CloudTrail to build an investigation graph for insider threat and ransomware analysis

Partner Discovery Checklist

Before deployment, validate the following with the customer:

FSx audit configuration (format, rotation, volume location)
S3 Access Point identity and security style
Expected daily event volume and file sizes
SIEM entitlement and daily ingest quota
Retention and compliance requirements
Privacy / PII redaction requirements
Network egress policy and approved endpoints
Audit log format decision (XML for serverless parsing, EVTX for Windows Event Viewer)
Audit destination volume and protection policy (backup, replication, retention)
Volume/qtree security style and effective permissions for S3 AP identity
SMB/NFS audit policy coverage and scope
Existing audit tool deployment: scope, format expectations, reporting use cases, and complementary vs replacement positioning
ARP / FPolicy coexistence with this pipeline

CrowdStrike discovery:

Is the customer using Falcon LogScale repositories or NG-SIEM connectors?
Is HEC ingestion enabled and contractually allowed?
What is the daily ingest entitlement and retention?
Which repository/parser/token will own FSx audit logs?
Who owns detection tuning and alert routing?
Will Fusion SOAR / Charlotte AI be used for investigation workflows?
LogScale repository/view strategy per customer (MSSP)
Ingest token per customer and rotation owner (MSSP)
Parser package ownership and change control

GitHub Repository

Governance & Cross-Platform Access: Lake Formation, PII Anonymization, and Multi-Engine Reality for S3 Tables

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Mon, 08 Jun 2026 16:49:37 +0000

Previously...

In Part 1, we built the metadata catalog. In Part 2, we added AI classification and vector search. Now we need to answer the hard questions:

Who can see what? (governance)
What about PII? (anonymization)
Can Databricks/Snowflake access this? (cross-platform)

Lake Formation: Governance on Unstructured Data

The Problem

Unstructured data on NAS storage may be well protected at the file-system layer, but it is often not consistently classified, searchable, or governed from analytics and AI workflows:

No unified classification → you may not know what's sensitive across the entire corpus
File-system permissions exist, but analytics/AI tools can't leverage them for discovery
Audit trails may exist at the file-system layer, but they are often not unified with analytics and AI query activity

The Solution

With metadata in S3 Tables (Iceberg), Lake Formation provides:

┌───────────────────────────────────────────────────┐
│  Lake Formation                                   │
│                                                   │
│  Table-level:  SELECT, DESCRIBE                   │
│  Column exposure: controlled via Athena Views     │
│                   (hide embedding_vector, paths)  │
│  Row filtering: WHERE sensitivity_level = 'public'│
│  Audit:        CloudTrail logs metadata queries   │
└───────────────────────────────────────────────────┘

Verified: Access Control in Action

Step 1: Authorized user queries metadata
  → ✅ SUCCEEDED (3 rows returned)

Step 2: Revoke SELECT permission
  → 🔒 BLOCKED: "Column 'file_name' cannot be resolved
     or requester is not authorized"

Step 3: Restore permission
  → ✅ SUCCEEDED (access restored)

Step 4: CloudTrail audit
  → All queries logged with user identity and timestamp

Every query against the metadata table is governed and audited. This gives you 100% metadata query governance coverage in this PoC. Raw file access remains governed separately by FSx for ONTAP file-system permissions, S3 Access Point policies, and application-specific access paths.

Lake Formation Governance Status

Capability	Status	Notes
Table-level SELECT / DESCRIBE	✅ Verified	Grant/revoke works correctly
Athena query governance	✅ Verified	Unauthorized access blocked
CloudTrail audit logging	✅ Verified	All queries logged with user identity
Column-level exclusion (ColumnWildcard)	⚠️ Failed	On tested S3 Tables federated catalog path
Row-level filtering / LF-Tags	📋 Design pattern	Taxonomy defined, needs validation
Column exposure via Athena Views	✅ Workaround	Recommended alternative to column-level grants

Observed Limitation: Column-Level Grants on This S3 Tables Federated Catalog Path

In this PoC, table-level Lake Formation SELECT grants worked as expected. However, column exclusion grants using ColumnWildcard with ExcludedColumnNames returned InvalidInputException: Permissions modification is invalid against the s3tablescatalog/... federated catalog path we tested.

AWS documentation describes table, column, and row-level permissions for S3 Tables integrated with Lake Formation. Therefore, treat this as an observed limitation in our specific validation path (CLI command, region, catalog ID, engine version), not a confirmed general product limitation. The exact error and test conditions are recorded in the verification evidence.

Workaround: Create Athena Views that expose only permitted columns:

-- View for general users (no embeddings, no PII paths)
CREATE VIEW metadata.public_files AS
SELECT file_id, file_name, file_type, classification, confidence_score
FROM "s3tablescatalog/fsxn-metadata-catalog"."metadata"."unstructured_files"
WHERE is_deleted = false AND sensitivity_level = 'public';

-- Apply Lake Formation on the view
-- Users query the view, not the base table

Governance model choice: For simple use cases, table/column-level permissions suffice. For dynamic, attribute-based access (e.g., "only files classified as 'public'"), use LF-Tags. For enterprise SSO integration, combine with IAM Identity Center. For enterprise governance, map sensitivity_level, path_classification, tenant_id, and pii_status to LF-Tags. See governance/lf-tag-taxonomy.yaml.

Untested alternative: Registering the S3 Tables table in a standard (non-federated) Glue Catalog may enable column-level permissions. This requires manual Iceberg metadata location configuration and has not been verified.

PII Detection: English + Japanese

The Challenge

Amazon Comprehend's detect_pii_entities API supports only English and Spanish. For Japanese PII (names, addresses, My Number), we need a different approach.

Dual-Engine Architecture

Language	Engine	Detectable PII	Latency	Cost
English	Amazon Comprehend	NAME, EMAIL, PHONE, ADDRESS, SSN, CREDIT_CARD, DATE_TIME	~200ms	$0.0001/100 chars
Japanese	Bedrock Claude	氏名, メール, 電話, 住所, マイナンバー, クレジットカード, 生年月日	~2-5s	~$0.003/request

Data privacy note: When using Bedrock Claude for PII detection, document text is sent to the Bedrock API. Per AWS's data privacy policy, Bedrock does not store or use your inputs/outputs to train models. For highly sensitive workloads, consider VPC endpoints and AWS PrivateLink for Bedrock access.

Japanese PII Detection (Verified)

# Bedrock Claude detects Japanese PII via prompt
response = bedrock.invoke_model(
    modelId="anthropic.claude-3-haiku-20240307-v1:0",
    body=json.dumps({
        "messages": [{"role": "user", "content":
            f"Detect all PII in this text. Return JSON array: "
            f'[{{"type":"...","value":"...","begin":N,"end":N}}]\n\n'
            f"Text:\n{japanese_text}"}]
    })
)

Results on a controlled synthetic sample (not real personal data):

PII Type	Detected Value
NAME	山田太郎
EMAIL	taro.yamada@example.co.jp
PHONE	090-1234-5678
ADDRESS	〒150-0002 東京都渋谷区渋谷1-2-3
MY_NUMBER	1234 5678 9012
CREDIT_CARD	4111-1111-1111-1111
DATE_OF_BIRTH	1985年3月15日

Anonymization Pipeline

Original document
       │
       ▼
PII Detection (Comprehend or Bedrock)
       │
       ├─ No PII → has_pii = false (no action needed)
       │
       └─ PII found → has_pii = true
                          │
                          ▼
              Redaction: all PII → [REDACTED]
                          │
                          ▼
              Store anonymized version
              anonymization_status = "completed"

Before:

Name: Taro Yamada
Email: taro.yamada@example.com
Phone: 090-1234-5678
SSN: 123-45-6789

After:

Name: [REDACTED]
Email: [REDACTED]
Phone: [REDACTED]
SSN: [REDACTED]

Data Clean Room Pattern

┌─────────────────────────────────────────┐
│  Restricted Table (full metadata)       │
│  • has_pii, anonymized_path, raw paths  │
│  • Access: Security team only           │
│  • Lake Formation: strict SELECT grant  │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│  Public Table (anonymized metadata)     │
│  • classification, summary (redacted)   │
│  • No PII, no raw file paths            │
│  • Access: All analysts                 │
│  • Lake Formation: broad SELECT grant   │
└─────────────────────────────────────────┘

Encryption and Data Residency

At rest: S3 Tables uses SSE-S3 encryption by default. All metadata is encrypted.
In transit: All API calls use TLS 1.2+.
Data residency: Both metadata (S3 Tables) and raw files (FSx for ONTAP) remain in the same AWS region. No cross-border data transfer occurs in the default architecture.

For detailed data sovereignty analysis, see the Architecture Document — Data Sovereignty section.

Audit Log Retention

CloudTrail: Default 90-day event history. For long-term retention, create a Trail delivering to S3 (recommended: 1+ year for regulated industries)
Lake Formation: Data access audit logs are recorded via CloudTrail
OpenSearch: Access logs can be delivered to CloudWatch Logs
Analysis: Use CloudTrail Lake (SQL queries) or Athena + S3 (cost-efficient) for audit analysis

For detailed operational monitoring setup, see the Operational Monitoring section in the architecture document.

Path Sensitivity Model

File paths can reveal sensitive context even when file contents are not exposed (e.g., /hr/layoffs/2026/ or /legal/mna/target-company/).

Recommended controls:

Store raw_path only in the restricted metadata table
Expose hashed_path or anonymized_path to general users
Use path_classification: public / internal / restricted / confidential
Apply Lake Formation grants to curated views, not the base table

Raw Data Access Boundary

This architecture governs metadata access through S3 Tables and Lake Formation. It does not automatically replace:

ONTAP/NFS/SMB file-system permissions
S3 Access Point resource policies
IAM permissions for raw file reads
Application-level authorization
Downstream use of presigned URLs or copied files

Treat metadata governance and raw data governance as two linked but separate control planes. Both must be configured for end-to-end security.

S3 Access Point Identity Boundary

Each FSx for ONTAP S3 Access Point has an associated file-system identity (OntapFileSystemIdentity — UNIX UID/GID or Windows domain user). All file access through that AP is authorized as that identity.

For each access point, document:

IAM principals allowed to use the access point
Access point policy (allowed S3 actions)
Associated UNIX or Windows file-system identity
Allowed volume / prefix scope
Whether the identity can access files beyond what metadata governance intends
Audit evidence location

If the AI enrichment access point uses a broad UNIX identity (e.g., root or a service account with wide read access), metadata-level Lake Formation controls do not prevent raw file reads through that AP. Scope the AP identity to minimum required access.

See security/s3-access-point-identity-matrix.yaml for the template.

Permission Identity Strategy

For multiprotocol environments (NFS + SMB + S3 AP):

Record discovery_protocol: nfs / smb / s3ap
Record access_point_identity_type: unix / windows
Record effective_reader_identity
Record permission_source: nfs_mode / ntfs_acl / mixed
Do not assume metadata visibility implies raw file readability

Retention and Deletion Semantics

This PoC uses metadata records to represent file discovery and enrichment state. For regulated workloads, define:

Metadata retention period (how long to keep catalog records)
Raw file retention period (governed by storage policy, not this catalog)
Anonymized metadata retention period
Deletion request workflow (who can request, who approves, how it's executed)
Snapshot expiration impact on deletion (Iceberg time travel may expose deleted metadata until snapshots expire)
Audit evidence retention (keep deletion evidence longer than the data itself)

Important: Iceberg time travel is useful for recovery, but it means deleted metadata may still be queryable during the snapshot retention window. Align snapshot expiration with your data deletion SLA.

Snowflake-side retention: If redacted metadata is synced into Snowflake-managed tables, define Snowflake-side retention, Time Travel (default 1 day, up to 90 days), and Fail-safe (7 days, non-configurable) separately from Iceberg snapshot retention. Deletion from the Snowflake copy does not delete from the Iceberg source, and vice versa.

Approval Evidence Template (for Regulated Industries)

For organizations requiring formal access approval documentation:

Approval ID: <unique-id>
Data owner: <name/group>
Security owner: <name/group>
Platform owner: <name/group>
Allowed metadata columns: <columns>
Allowed raw file prefixes: <prefixes>
Allowed operations: metadata query only / raw file read / anonymized export
Review date: <date>
Expiration date: <date>
Evidence location: verification-evidence/<path>

Regulated Workload Readiness

For public sector, healthcare, financial services, and other regulated industries, validate the following before production deployment:

Area	Requirement	Status in this PoC
Data residency	Metadata and raw files in same AWS Region	✅ Single region (ap-northeast-1)
Encryption at rest	S3 Tables: SSE-S3; FSx: at-rest encryption	✅ Default encryption
Encryption in transit	TLS 1.2+ for all API calls	✅ AWS default
Raw data access boundary	File reads governed by S3 AP policy + ONTAP permissions	✅ Documented
Metadata access boundary	Lake Formation table-level + CloudTrail audit	✅ Verified
AI processing data flow	Content sent to Bedrock API, not stored by provider	✅ Per AWS data protection policy
PII detection limitations	English (Comprehend) + Japanese (Claude) only	⚠️ Other languages not covered
Human review workflow	Low-confidence queue defined	✅ Design documented
Audit log retention	CloudTrail 90-day default; configure Trail for longer	⚠️ Requires Trail setup
Deletion SLA	Define separately for metadata, raw files, and snapshots	⚠️ Requires policy definition
Legal/compliance sign-off	Not in scope for this PoC	❌ Required before production

AI governance note: AI enrichment in this pattern is assistive metadata generation. It does not constitute authoritative regulatory classification. Final classification decisions, data handling approvals, and compliance certifications must be confirmed by data owners, security teams, legal counsel, and compliance officers.

Cross-Platform Access: The Current Reality

Fully Verified ✅

Platform	Access Method	Status
Athena	Direct query via Glue federated catalog	✅ Fully verified
Lambda/Python	PyIceberg SDK	✅ Fully verified
EMR Spark	Glue Iceberg REST (EMR 7.13.0+)	✅ Fully verified (SELECT, COUNT, time travel)
Snowflake	Glue Iceberg REST + VENDED_CREDENTIALS	✅ Fully verified (CREATE TABLE, SELECT, COUNT, DESCRIBE, AUTO_REFRESH)
Snowflake	External Stage (FSx S3 AP) + TO_FILE + Cortex AI	✅ Fully verified

Expected / Requires Validation ⚠️

Platform	Access Method	Status
EMR Trino	Glue Iceberg REST (EMR 7.13.0+)	⚠️ Expected (same EMR SigV4 handling as Spark)
Redshift Spectrum	Same as Athena (Glue catalog)	⚠️ Expected, not fully validated

What Doesn't Work (Yet) ⚠️

Platform	Tested method	Result	Tested	Status
Databricks SQL Warehouse	`CREATE CONNECTION TYPE iceberg_rest` to S3 Tables REST	`CONNECTION_TYPE_NOT_SUPPORTED`	2026-05-31	Observed limitation in this path
Databricks Spark cluster	Iceberg REST + SigV4 via spark.conf.set / cluster config	`NO_SUCH_CATALOG_EXCEPTION` (UC blocks external catalog registration)	2026-06-01	Confirmed: UC Foreign Catalog required
Databricks Delta Sharing	Delta Sharing server accessing S3 AP-backed storage	Sharing server uses same UC storage credentials; cannot bypass session policy	2026-06-01	Confirmed limitation (not a workaround for S3 AP)
Databricks NFS → UC Volume	NFS mount path as UC External Volume	Cloud storage URIs only (s3://, abfss://, gs://); NFS/FUSE paths not supported	2026-06-01	Confirmed limitation; internal feature request exists
Snowflake	External Iceberg Table with S3 Tables direct REST endpoint	Not a supported catalog type (use Glue REST instead)	2026-05-31	Use Glue REST + VENDED_CREDENTIALS (✅ verified)
Snowflake	CATALOG INTEGRATION with default ACCESS_DELEGATION_MODE	Defaults to EXTERNAL_VOLUME_CREDENTIALS which triggers ListObjectsV2 (rejected by S3 Tables)	2026-06-02	✅ Resolved: set explicit `ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS`
Snowflake	Lake Formation column-level via VENDED_CREDENTIALS	AllowFullTableExternalDataAccess=false blocks all VENDED_CREDENTIALS access	2026-06-08	Use Snowflake Horizon (Row Access Policy / Dynamic Masking) for column governance
Snowflake Open Catalog	Polaris as Iceberg catalog	Not tested	TBD	Strategic alternative

Databricks: Three Integration Paths to Validate

Update note (2026-06-09): We revalidated the S3 Tables path after Databricks announced GA for Foreign Iceberg and credential vending (May 28, 2026). Glue Connection creation and credential configuration succeeded, but Unity Catalog External Location validation still failed because S3 Tables internal buckets reject standard S3 API validation (HeadBucket/ListBucket). The S3 Tables path remains blocked in this tested Databricks UC configuration. A new Databricks support case has been submitted.

For Databricks business users: The value is not only table access. The value is turning previously invisible NAS files into governed metadata assets that can be searched, explained, lineage-tracked, and consumed from Databricks SQL, AI/BI, and dashboards.

In this PoC, CREATE CONNECTION TYPE iceberg_rest to the S3 Tables REST endpoint returned CONNECTION_TYPE_NOT_SUPPORTED on Databricks SQL Warehouse (tested 2026-05-31). This does not mean Databricks lacks Iceberg REST support — Databricks provides Unity Catalog Iceberg REST endpoints and Foreign Iceberg capabilities that evolve rapidly.

Confirmed Limitations (2026-06-01)

Path	Result	Confirmed by
Spark cluster + Iceberg REST (spark.conf.set / cluster config)	❌ UC blocks external catalog registration	Databricks support + our testing
Delta Sharing via S3 Access Point	❌ Sharing server uses same UC storage credentials	Databricks support
NFS mount path as UC External Volume	❌ Cloud storage URIs only (s3://, abfss://, gs://)	Databricks support
DataSync → S3 → UC External Delta Table → Delta Sharing	✅ Works (Delta format required)	Databricks support

Delta Sharing note: Delta Sharing is not a workaround for the FSx S3 Access Point session policy limitation in our tested path. The sharing server uses the same UC storage credentials and cannot bypass the session policy that blocks S3 AP ARNs. Note that Databricks has announced first-class Iceberg format support in Delta Sharing (Jan 2026), enabling providers to share Iceberg tables via the Iceberg REST Catalog API. This broader capability is not contradicted by our finding — our limitation is specific to S3 AP-backed storage access through UC credentials, not Delta Sharing's format support in general.

NFS Volume note: UC External Volumes require cloud storage URIs. An internal feature request (AHA) exists for EFS/NFS access via UC. Until this is implemented, DataSync → S3 → UC External Location remains the only supported path.

📢 Databricks users: If S3 Tables access from Databricks is important for your workflow, the UC Foreign Catalog for S3 Tables feature is being tracked internally by Databricks (request DB-I-15824). Contact your Databricks account team to express interest and increase prioritization. Snowflake achieved full S3 Tables access via VENDED_CREDENTIALS in June 2026 — the same architectural pattern should be feasible for UC.

Immediate workaround for Databricks: Use DataSync → S3 → UC External Table to sync metadata into a standard S3 location accessible by Unity Catalog. This is not zero-copy for the synced metadata, but raw files remain on FSx for ONTAP.

Path 1: Spark cluster + Iceberg REST (SigV4)

Best for technical validation and batch processing. Two endpoint options:

Tested 2026-06-01: On Databricks with Unity Catalog enabled, external Iceberg catalogs cannot be registered via spark.conf.set or cluster Spark config. Unity Catalog controls catalog registration exclusively. Both Serverless (CONFIG_NOT_AVAILABLE) and All-Purpose clusters (NO_SUCH_CATALOG_EXCEPTION) fail. Unity Catalog Foreign Catalog (Path 2) is the required approach.

# Path 1a: Direct S3 Tables REST endpoint (used in this PoC)
spark.sql.catalog.s3tables.uri=https://s3tables.ap-northeast-1.amazonaws.com/iceberg
spark.sql.catalog.s3tables.rest.signing-name=s3tables

# Path 1b: AWS Glue Iceberg REST endpoint (recommended for production + Lake Formation)
spark.sql.catalog.s3tables.uri=https://glue.ap-northeast-1.amazonaws.com/iceberg
spark.sql.catalog.s3tables.rest.signing-name=glue

Common config for both:

spark.sql.catalog.s3tables=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.s3tables.catalog-impl=org.apache.iceberg.rest.RESTCatalog
spark.sql.catalog.s3tables.warehouse=arn:aws:s3tables:ap-northeast-1:<ACCOUNT>:bucket/fsxn-metadata-catalog
spark.sql.catalog.s3tables.rest.sigv4-enabled=true
spark.sql.catalog.s3tables.rest.signing-region=ap-northeast-1

Path 2: Unity Catalog Foreign Iceberg

Register external Iceberg tables into Unity Catalog if supported for the target catalog/storage path. Best for Databricks governance, lineage, and discovery. Verify refresh semantics and read/write limitations. Retested for S3 Tables on 2026-06-09: Glue Connection and credentials succeeded, but UC External Location validation failed because S3 Tables internal buckets reject standard S3 API validation. This path remains blocked for the tested S3 Tables configuration.

Documentation/version note: Databricks Iceberg capabilities are evolving rapidly. Earlier documentation and our initial validation showed limitations around Foreign Iceberg credential vending and automatic refresh behavior. After the May 2026 GA announcement, we revalidated the S3 Tables path on 2026-06-09. Credential configuration progressed further, but the tested path still failed at UC External Location validation against S3 Tables internal storage. Additionally, Databricks supports catalog federation with AWS Glue (Hive Metastore type), which can expose Glue-registered tables in UC. Whether a future Iceberg REST catalog federation path could bypass the S3 Tables internal bucket constraint is an open question.

Refresh semantics: If UC Foreign Iceberg works for S3 Tables via Glue REST, define refresh semantics explicitly. Our metadata catalog is append-only (new records added on file events). Analysts should know whether Databricks reads the latest Iceberg snapshot automatically or only after REFRESH FOREIGN TABLE. Without auto-refresh, Athena and Databricks may show temporarily different results until the next refresh cycle. Plan for a scheduled refresh job or event-driven trigger.

AWS reference for this path: AWS has published guidance on accessing S3 Iceberg tables from Databricks using the Glue Iceberg REST Catalog. This validates the architectural direction of B-4/B-5, though S3 Tables-specific compatibility requires separate validation.

Path 3: AWS Glue Catalog Federation with Databricks

AWS Glue can federate metadata from Databricks Unity Catalog for Iceberg tables. This is the reverse direction but useful for cross-platform governance patterns.

Federation Directionality

Pattern	Direction	Primary governance	Best for
UC Foreign Catalog / Catalog Federation to Glue	Databricks reads AWS-managed metadata	Unity Catalog	Databricks users querying AWS Iceberg (S3 Tables)
AWS Glue federation to UC	AWS reads Databricks-managed metadata	Lake Formation / Glue	Athena/EMR/Redshift reading UC Iceberg/UniForm

AWS reference: AWS has published guidance on accessing S3 Iceberg tables from Databricks using AWS Glue Iceberg REST Catalog, and on federating Databricks Unity Catalog data into AWS Glue Data Catalog. Both directions are documented.

Why Iceberg here (not Delta Lake)? This architecture uses Iceberg because S3 Tables is Iceberg-native, and the Iceberg REST endpoint enables multi-engine access (Athena, EMR, Snowflake). For Databricks-only environments, Delta Lake on S3 remains the natural choice. This pattern targets multi-platform scenarios.

Databricks UC Audit Logging for External Engines (Confirmed 2026-06-01)

External engine access via the UC Iceberg REST Catalog endpoint is fully auditable:

Audit aspect	Confirmed behavior
Metadata requests (listNamespaces, listTables, loadTable)	✅ Logged in `system.access.audit` under `uniformIcebergRestCatalog`
Vended credential issuance	✅ Logged as `loadTableCredentials` / `generateTemporaryTableCredential`
Audit fields	user_identity, source_ip_address, user_agent, event_time, action_name, request_params
Distinguish external vs internal	✅ `service_name = 'uniformIcebergRestCatalog'` (external) vs `'unityCatalog'` (internal)

Note: Databricks audit logs record credential issuance, not individual S3 file reads after credentials are vended. Complement with AWS CloudTrail + S3 access logging for file-level audit.

Databricks integration documentation:

For UC Foreign Catalog validation steps, see databricks/uc-foreign-iceberg-validation.md

For coexistence planning, see databricks/coexistence-roadmap.md

For audit investigation, see databricks/audit-correlation-guide.md

Snowflake: S3 Tables via Glue REST + VENDED_CREDENTIALS ✅

Working Configuration (Verified 2026-06-05)

Snowflake can directly query S3 Tables Iceberg tables via the Glue Iceberg REST endpoint with VENDED_CREDENTIALS. Here's the complete working setup:

-- 1. Catalog Integration (CRITICAL: explicit ACCESS_DELEGATION_MODE)
CREATE OR REPLACE CATALOG INTEGRATION s3tables_glue_rest_int
  CATALOG_SOURCE = ICEBERG_REST
  TABLE_FORMAT = ICEBERG
  CATALOG_NAMESPACE = 'metadata'
  REST_CONFIG = (
    CATALOG_URI = 'https://glue.ap-northeast-1.amazonaws.com/iceberg'
    CATALOG_API_TYPE = AWS_GLUE
    CATALOG_NAME = '<ACCOUNT_ID>:s3tablescatalog/fsxn-metadata-catalog'
    ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS  -- MUST be explicit
  )
  REST_AUTHENTICATION = (
    TYPE = SIGV4
    SIGV4_IAM_ROLE = 'arn:aws:iam::<ACCOUNT_ID>:role/fsxn-snowflake-verification-role'
    SIGV4_SIGNING_REGION = 'ap-northeast-1'
  )
  ENABLED = TRUE;

-- 2. Schema WITHOUT default EXTERNAL_VOLUME (critical)
CREATE SCHEMA FSXN_LAKEHOUSE.S3TABLES_VENDED;
USE SCHEMA FSXN_LAKEHOUSE.S3TABLES_VENDED;

-- 3. Table WITHOUT EXTERNAL_VOLUME parameter (critical)
CREATE ICEBERG TABLE s3tables_vended_creds_test
  CATALOG = 's3tables_glue_rest_int'
  CATALOG_TABLE_NAME = 'unstructured_files';

AWS prerequisites (must be completed before Snowflake configuration):

# Register S3 Tables resource with Lake Formation (--with-federation is REQUIRED)
aws lakeformation register-resource \
  --resource-arn "arn:aws:s3tables:ap-northeast-1:<ACCOUNT_ID>:bucket/fsxn-metadata-catalog" \
  --role-arn "arn:aws:iam::<ACCOUNT_ID>:role/S3TablesRoleForLakeFormation" \
  --with-federation \
  --region ap-northeast-1

# Grant SELECT + DESCRIBE to Snowflake's IAM role (table-level)
aws lakeformation grant-permissions \
  --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<ACCOUNT_ID>:role/fsxn-snowflake-verification-role"}' \
  --resource '{"Table":{"CatalogId":"<ACCOUNT_ID>:s3tablescatalog/fsxn-metadata-catalog","DatabaseName":"metadata","Name":"unstructured_files"}}' \
  --permissions SELECT DESCRIBE \
  --region ap-northeast-1

IAM role policy must include: glue:GetTable, glue:GetDatabase, glue:GetCatalog, lakeformation:GetDataAccess, s3tables:GetTableBucket, s3tables:GetTable, s3tables:GetNamespace, s3tables:GetTableData, s3tables:GetTableMetadataLocation.

Verify integration (expected DESCRIBE output after creation):

DESCRIBE CATALOG INTEGRATION s3tables_glue_rest_int;

Property	Value
ENABLED	true
CATALOG_SOURCE	ICEBERG_REST
TABLE_FORMAT	ICEBERG
CATALOG_NAMESPACE	metadata
REST_CONFIG	{CATALOG_URI=https://glue.ap-northeast-1.amazonaws.com/iceberg, ...}
REST_AUTHENTICATION	{TYPE=SIGV4, SIGV4_IAM_ROLE=arn:aws:iam::<ACCOUNT_ID>:role/..., ...}
API_AWS_IAM_USER_ARN	arn:aws:iam::465774455528:user/<snowflake-user-id>
API_AWS_EXTERNAL_ID	<external-id-for-trust-policy>
REFRESH_INTERVAL_SECONDS	30

Setup step: Copy API_AWS_IAM_USER_ARN and API_AWS_EXTERNAL_ID from this output into your IAM role's trust policy to allow Snowflake to assume the role.

Verified Capabilities (2026-06-08)

Operation	Result	Performance
CREATE ICEBERG TABLE	✅	6.5s
SELECT * LIMIT 5	✅ (5 rows)	1.9s
COUNT(*)	✅ (170 rows)	66ms
DESCRIBE TABLE	✅ (23 columns)	69ms
ALTER ... SET AUTO_REFRESH = TRUE	✅	131ms
SHOW ICEBERG TABLES	✅ (UNMANAGED type)	567ms
Time Travel	✅ (available, snapshot-dependent)	—

Screenshots (URL bar excluded, S3 Tables internal bucket masked):

COUNT() returns 170 rows in 66ms*

DESCRIBE TABLE shows all 23 Iceberg columns

SHOW ICEBERG TABLES confirms UNMANAGED type with S3TABLES_GLUE_REST catalog

SELECT * LIMIT 5 returns actual file metadata from S3 Tables

AUTO_REFRESH + Time Travel verification:

AUTO_REFRESH verified: PyIceberg appended 1 record → Snowflake COUNT() automatically updated from 170 to 171 within 30 seconds*

-1200) = 170" width="800" height="457">
Time Travel: querying 20 minutes ago returns 170 (before the append), confirming snapshot history is accessible

About FILE_PATH: The FILE_PATH column shows the S3 path used during metadata ingestion (via FSx for ONTAP S3 Access Point). This is the path recorded in the Iceberg metadata catalog — it does not mean the files were copied to S3. The actual files remain on FSx for ONTAP and are accessible via NFS, SMB, or S3 Access Point depending on your application's protocol.

Key Insight: Why Previous Attempts Failed

ACCESS_DELEGATION_MODE defaults to EXTERNAL_VOLUME_CREDENTIALS when not explicitly specified. In this default mode, Snowflake validates storage access through the External Volume path, which triggers ListObjectsV2 against S3 Tables internal buckets — an operation that returns MethodNotAllowed.

With VENDED_CREDENTIALS explicit:

Snowflake calls Glue REST loadTable
Lake Formation (via GetTemporaryGlueTableCredentials) returns temporary storage credentials in the loadTable response config map
Snowflake uses these credentials to access data files directly by exact path
No ListObjectsV2 is required — Snowflake reads files by exact path from Iceberg metadata

Note: The Glue REST endpoint does not implement the standard Iceberg REST /credentials endpoint. Credential vending works through Lake Formation's proprietary mechanism embedded in the loadTable response. This is transparent to Snowflake when configured correctly.

Governance Limitation: Lake Formation Column-Level (2026-06-08)

Lake Formation column-level filtering is NOT enforced via the VENDED_CREDENTIALS path:

When AllowFullTableExternalDataAccess = false, the entire VENDED_CREDENTIALS path is blocked
Explicit column/table-level grants + ExternalDataFilteringAllowList do not resolve this
AllowFullTableExternalDataAccess = true is required for VENDED_CREDENTIALS to function

Technical context: AllowFullTableExternalDataAccess controls whether external engines (those using Lake Formation credential vending) can access table data without per-table SELECT grants. When set to false, fine-grained column/row filtering is the intended enforcement mechanism — but for S3 Tables accessed via VENDED_CREDENTIALS, this currently results in complete access denial rather than filtered access. This may be a service-specific constraint of the S3 Tables federated catalog path, or it may require additional AllowExternalDataFiltering + ExternalDataFilteringAllowList configuration that was not effective in our testing. A feature request has been submitted to AWS.

Workaround: Use Snowflake Horizon for column-level governance:

-- Row Access Policy: restrict by sensitivity_level
CREATE OR REPLACE ROW ACCESS POLICY metadata_sensitivity_filter AS
  (sensitivity_level VARCHAR) RETURNS BOOLEAN ->
    CASE
      WHEN IS_ROLE_IN_SESSION('SECURITY_ADMIN') THEN TRUE
      WHEN sensitivity_level IN ('public', 'internal') THEN TRUE
      ELSE FALSE
    END;

ALTER TABLE s3tables_vended_creds_test ADD ROW ACCESS POLICY
  metadata_sensitivity_filter ON (sensitivity_level);

-- Dynamic Data Masking: hide embedding vectors from non-ML roles
CREATE OR REPLACE MASKING POLICY mask_embedding AS
  (val BINARY) RETURNS BINARY ->
    CASE
      WHEN IS_ROLE_IN_SESSION('ML_ENGINEER') THEN val
      ELSE NULL
    END;

ALTER TABLE s3tables_vended_creds_test MODIFY COLUMN
  embedding_vector SET MASKING POLICY mask_embedding;

Snowflake Iceberg Access Modes (Summary)

Access mode	Best for	Status
Glue REST + VENDED_CREDENTIALS	S3 Tables direct query	✅ VERIFIED
External Stage (FSx S3 AP) + TO_FILE	File AI analysis (Cortex COMPLETE)	✅ VERIFIED
Metadata sync to Snowflake table	BI / Cortex Search / governance	Available
Object Store Catalog	Direct metadata file read	❌ Blocked (S3 Tables internal bucket)
Snowflake Open Catalog (Polaris)	Alternative Iceberg catalog	Not tested

📖 Investigation History (2026-06-01 to 2026-06-05) — click to expand

2026-05-31: Tested S3 Tables direct REST endpoint as External Iceberg catalog → not a supported catalog type.

2026-06-01: Created CATALOG INTEGRATION using ICEBERG_REST + AWS_GLUE + VENDED_CREDENTIALS. DESCRIBE succeeded but CREATE ICEBERG TABLE failed with "Failed to retrieve credentials from the Catalog". Root cause identified: Glue REST does not implement /credentials endpoint (UnknownOperationException).

2026-06-02: AWS Support confirmed Lake Formation uses proprietary mechanism (GetTemporaryGlueTableCredentials) for credential vending, not standard Iceberg REST /credentials. Snowflake Support confirmed Error 004174 occurs when s3.access-key-id/secret/token absent from loadTable response.

2026-06-02: Tested Object Store catalog and EXTERNAL_VOLUME_CREDENTIALS mode — both blocked by S3 Tables internal bucket rejecting ListObjectsV2.

2026-06-03: Discovered register-resource --with-federation was missing. After setup, loadTable response included credentials. However, CREATE TABLE still failed at storage validation (ListObjectsV2).

2026-06-05: Snowflake Support identified the critical distinction: ACCESS_DELEGATION_MODE defaults to EXTERNAL_VOLUME_CREDENTIALS. Explicitly setting VENDED_CREDENTIALS + schema without External Volume + CREATE TABLE without External Volume parameter → SUCCESS. CREATE TABLE + SELECT both working.

2026-06-08: Additional testing confirmed COUNT(*), DESCRIBE, AUTO_REFRESH, SHOW ICEBERG TABLES all working. Lake Formation column-level filtering NOT enforced via this path (AllowFullTableExternalDataAccess=false blocks all access).

External Stage note: Snowflake External Stage against the FSx S3 Access Point alias was verified in this PoC (2026-05-31, ap-northeast-1). Update (2026-06-02): TO_FILE (Cortex COMPLETE multimodal) also verified working — Claude Sonnet 4.5 can directly read files from FSx for ONTAP via S3 AP-backed External Stage. See snowflake/external-stage-fsx-s3ap-validation.md for exact DDL and verified operations.

Snowflake Metadata Activation Pattern

If you sync only the metadata into Snowflake (not raw files), you preserve the zero-copy principle for actual data while enabling Snowflake-native use cases:

Governed metadata analytics and executive dashboards
File inventory and PII coverage reporting
Cortex Search over redacted summaries (RAG on metadata)
Snowflake Intelligence / Cortex Analyst style business Q&A
Row Access Policies and Dynamic Masking on synced metadata

Horizon Catalog note: When metadata reaches Snowflake, Snowflake governance features such as Row Access Policies and Dynamic Masking can be applied to Snowflake-managed access paths. For external engine access via Iceberg REST, validate the exact Open Catalog / Horizon behavior for your target engine and security model.

Metadata sync best practice: Sync curated latest-record metadata, not the append-only base table, unless analysts explicitly need history. Preserve scan_run_id, change_type, and is_deleted for audit and reconciliation. Use MERGE INTO keyed by file_id or path_hash to make metadata activation idempotent. See snowflake/metadata-sync-example.sql for the full pattern.

Governance policy mapping: When syncing metadata into Snowflake, map AWS-side fields such as sensitivity_level, tenant_id, pii_status, and path_classification to Snowflake tags, masking policies, and row access policies. Track policy drift between Lake Formation and Snowflake governance. See snowflake/path-decision-guide.md for the full policy mapping.

Snowflake Cortex Search Activation Pattern

If redacted metadata and summaries are synced into Snowflake, Cortex Search can provide Snowflake-native enterprise search and RAG over metadata — without managing embeddings, infrastructure, or search quality tuning.

Why Cortex Search here:

Business users can search approved metadata without operating a separate vector database
RAG and enterprise search can run over redacted summaries already governed in Snowflake
Search quality, embedding management, and index refresh are delegated to Snowflake-managed services
This is best suited for Snowflake-first organizations that want business-facing discovery inside the AI Data Cloud

Use Cortex Search for:

Executive metadata search (natural language queries over file inventory)
File inventory Q&A (powered by LLM + retrieval)
PII coverage reporting and compliance dashboards
Governed search over redacted summaries

OpenSearch Serverless NextGen remains the AWS-native serving index for this PoC. Cortex Search is an optional Snowflake-native alternative for organizations that standardize on Snowflake for business discovery.

Role separation: S3 Tables / Iceberg remains the metadata source of truth. OpenSearch (AWS path) or Cortex Search (Snowflake path) are serving indexes for search UX. Choose based on your primary platform. Cortex Search operates over redacted summaries and metadata synced into Snowflake, not raw files, unless the customer explicitly chooses to copy/extract document content into Snowflake (which would break the zero-copy raw data principle).

Cortex Search scope: Cortex Search should operate on redacted metadata and summaries by default. If raw document content is extracted or copied into Snowflake for Cortex use cases, treat that as a separate data movement decision with its own governance, retention, and cost model.

Snowflake activation cost drivers: Snowflake activation introduces separate cost drivers from the AWS-native catalog: warehouse compute for metadata sync tasks and dashboards, Cortex Search service usage (based on corpus size and query volume), task/stream orchestration for refresh, and small metadata storage. These costs should be modeled separately from the AWS-native catalog cost ($114/month estimate in Part 1 does not include Snowflake-side compute).

Retention alignment: Confirm Snowflake account edition, table type, and retention settings before promising deletion SLAs. Snowflake Time Travel (1–90 days) and Fail-safe (7 days) operate independently from Iceberg snapshot expiration. Snowflake-side deletion evidence should be retained separately from Iceberg snapshot expiration evidence.

Snowflake Metadata Product Contract

When activating metadata in Snowflake, expose a curated subset as the governed metadata product:

Recommended curated columns:

Column	Purpose	Governance
file_id	Unique identifier	—
business_domain	Organizational grouping	Row access policy
file_type	File format	—
classification	AI-generated classification	—
sensitivity_level	Data sensitivity tier	Snowflake tag + masking policy
pii_status	PII detection result	Access policy / dashboard filter
redacted_summary	AI-generated (PII-free) summary	Cortex Search source column
owner_team	Business ownership	Business glossary / stewardship
last_seen_at	Last scan timestamp	—
data_quality_status	Enrichment quality flag	—

Snowflake governance mapping:

sensitivity_level → Snowflake tag + masking policy
tenant_id / business_domain → row access policy
pii_status → access policy / dashboard filter
redacted_summary → Cortex Search source column
owner_team → business glossary / stewardship workflow

Databricks Metadata Activation Pattern

If UC Foreign Catalog is not yet validated for your S3 Tables path, sync only the redacted metadata into a UC-managed Delta table. This preserves the zero-copy principle for raw files while enabling Databricks-native use cases:

Databricks SQL dashboards and executive reporting
AI/BI Genie over curated metadata (natural language queries)
UC lineage and audit on metadata usage
ML feature generation from file metadata
Operational reporting on PII coverage and enrichment backlog

Raw files remain on FSx for ONTAP. Only the small metadata table (~MB scale for 100K files) is synced.

This is analogous to the Snowflake metadata activation pattern: it copies only curated metadata, not the original unstructured files. Both patterns preserve the zero-copy principle for raw data.

Databricks Raw File Access Decision:

Requirement Recommended path

Governed metadata analytics only UC Foreign Catalog (if validated) or sync metadata to UC Delta

Raw file processing in Databricks DataSync → S3 → UC External Volume

Zero-copy raw file access from Databricks Not supported in validated paths (NFS mount works but without UC governance)

Business discovery / BI Sync redacted metadata to UC Delta table

If metadata is synced into Databricks for BI, include Databricks SQL / Jobs compute cost in the activation model. This does not affect raw-file zero-copy storage, but it is part of the business-facing analytics cost.

Other Lakehouse Engines to Validate

Beyond Databricks and Snowflake, the most natural validation targets for this metadata catalog are:

Engine	Access path	Likely fit	Validation priority
Trino / Starburst	Glue Iceberg REST or S3 Tables REST	Federated SQL, ad hoc query	High
EMR Spark	Glue Iceberg REST (native since EMR 7.5.0+)	Bulk backfill, batch enrichment	High
Redshift Spectrum	Glue catalog (external schema)	DWH integration, BI	Medium
Dremio	Glue catalog or Iceberg REST	Query acceleration, BI	Medium
StarRocks / Doris	Glue Iceberg REST	Low-latency serving queries	Medium
Apache Flink	Glue Iceberg REST	Streaming metadata updates	Low
dbt (via Athena)	dbt-athena + Iceberg materialization	Analytics engineering, governed marts	Medium
Apache NiFi	Iceberg REST or Polaris	Event-driven ingestion	Low

These engines should be validated against:

S3 Tables direct REST vs AWS Glue Iceberg REST
Read vs write capability
Lake Formation behavior (credential vending, column/row filtering)
Snapshot freshness after external writes
Latest-record view compatibility
Case-sensitivity and lowercase naming requirements

Key finding from validation (2026-06-08): AWS Glue Iceberg REST supports SigV4-authenticated catalog access. Lake Formation credential vending works through a proprietary mechanism (GetTemporaryGlueTableCredentials). Snowflake requires explicit ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS — the default mode fails. Engines that can sign requests with their own IAM credentials (EMR Spark ✅ verified, Trino on EMR expected, PyIceberg ✅ verified) work out of the box. Snowflake also works when configured correctly (✅ verified 2026-06-05). EMR requirement: 7.13.0+ (7.5.0 has a credential resolution bug). Governance note: Lake Formation column-level filtering is NOT enforced via the VENDED_CREDENTIALS path for Snowflake.

Trino note: AWS has published guidance on querying S3 Tables from Trino using the Iceberg REST endpoint. Trino's Iceberg connector supports REST catalogs natively, making it one of the most straightforward third-party validation targets.

EMR Spark note: For large-scale backfill or re-enrichment (100K+ files), Spark on EMR Serverless or EMR on EC2 can be used as an alternative to Lambda/Fargate. Use Glue Iceberg REST for centralized metadata access with Lake Formation governance. Verified (2026-06-02): EMR Serverless Spark 7.13.0 successfully reads S3 Tables metadata via Glue Iceberg REST — SHOW NAMESPACES, SHOW TABLES, SELECT, COUNT, and snapshot history all work. Requires EMR 7.13.0+ (7.5.0 has a credential resolution bug for S3 Tables warehouse format).

Redshift note: Validate separately from Athena — external schema setup, Glue statistics, Lake Formation permissions, and query latency against latest-record views may differ.

For the full compatibility matrix, see lakehouse-tools/tool-compatibility-matrix.yaml.

Catalog Authority Rule

For each Iceberg table, define exactly one authoritative catalog for metadata pointer and commit coordination. Do not operate S3 Tables, Polaris, Gravitino, Nessie, and Glue as independent writable catalogs for the same table unless the integration explicitly supports federation without dual writes.

                    ┌─────────────────────┐
                    │ Authoritative Catalog│
                    │ (ONE per table)      │
                    │ • S3 Tables + Glue   │
                    │   (this PoC)         │
                    └──────────┬──────────┘
                               │
              ┌────────────────┼────────────────┐
              │                │                │
              ▼                ▼                ▼
         Read-only        Read-only        Read-only
         consumers        consumers        consumers
         (Trino,          (Databricks,     (Snowflake,
          Dremio,          UC Foreign       Cortex,
          StarRocks)       Catalog)         Open Catalog)

Split-brain warning: If two catalogs independently write to the same Iceberg table, snapshot pointers can diverge, causing data loss or corruption. Federation (one writer, many readers) is safe. Dual-write is not.

The Bigger Picture

                    S3 Tables (Iceberg)
                           │
              ┌────────────┼────────────┐
              │            │            │
              ▼            ▼            ▼
         Athena ✅    Databricks ❌  Snowflake ✅
         EMR Spark ✅  (UC Foreign    (Glue REST +
         PyIceberg ✅   path still    VENDED_CREDENTIALS
                      blocked in     verified 2026-06-05)
                      tested config)

Databricks integration summary (confirmed 2026-06-01):

Direct S3 AP access: ❌ (UC session policy)
NFS mount → UC Volume: ❌ (cloud URI only)
Delta Sharing via S3 AP: ❌ (same credentials)
DataSync → S3 → UC: ✅ (supported workaround, not zero-copy for synced data)
UC Foreign Catalog / Foreign Iceberg via Glue: ❌ Retested 2026-06-09; Glue Connection and credentials succeeded, but UC External Location validation failed against S3 Tables internal storage. Support case submitted.

For capability-level details such as read, write, time travel, metadata tables, and governance behavior, see verification-evidence/cross-platform-compatibility.yaml.

This is a temporary gap. S3 Tables is relatively new (GA Dec 2024), and cross-platform federation is actively being developed. Feature requests have been filed with both platforms. Timeline for native S3 Tables support is unknown, but the Iceberg ecosystem is converging rapidly — Unity Catalog 2.0's native Iceberg support and Snowflake's Open Catalog (Polaris) both point toward broader interoperability.

Catalog Decision Guide

In the Iceberg world, the catalog is the system of record for table metadata pointers and atomic operations. Choose based on your primary platform:

Primary platform	Recommended catalog	Notes
AWS-first / Athena-first	S3 Tables + Glue/Lake Formation	Used in this PoC
Databricks-first	Unity Catalog Managed/Foreign Iceberg	Best for UC governance, lineage, discovery
Snowflake-first	Snowflake Open Catalog (Polaris)	Best for Snowflake-governed Iceberg interoperability; validate external engine governance behavior
Neutral / OSS-first	Apache Polaris or other REST catalog	Maximum portability

Dual catalog warning: Avoid running two authoritative catalogs for the same Iceberg table. Use Snowflake Open Catalog / Polaris when Snowflake or a neutral REST catalog should be authoritative. Use S3 Tables when AWS-native Athena / Lake Formation / Glue governance is authoritative. If both platforms need access, use federation (one authoritative catalog, others read via REST).

When to Consider Snowflake Open Catalog / Polaris

Use S3 Tables + Glue/Lake Formation when AWS-native governance is authoritative (this PoC).

Consider Snowflake Open Catalog / Polaris when:

Snowflake should be the primary governance and interoperability plane
Multiple engines need Iceberg REST access through a neutral catalog
Snowflake-managed Iceberg or Snowflake-first AI/Data Cloud workflows are the center of gravity
You want managed Polaris instead of operating your own REST catalog

This would be a different authoritative-catalog design from the current PoC and should not be mixed as a second writer for the same table.

Databricks-first note: For organizations standardizing on Databricks, consider whether the metadata catalog itself should be managed in Unity Catalog as Managed Iceberg or Delta + UniForm, then exposed to AWS engines through Glue federation to UC or the UC Iceberg REST endpoint. Use S3 Tables when AWS-native Athena/Lake Formation is the primary governance path. The choice depends on which governance plane (UC or Lake Formation) is authoritative for your organization.

Format Decision for Databricks Environments

Option	Best for	Tradeoff
S3 Tables Iceberg	AWS-first Athena/LF governance	UC integration pending (Foreign Catalog validation)
UC Managed / Foreign Iceberg	Databricks-first open format governance	Validate current feature availability, region support, and limitations
Delta + UniForm	Databricks-native pipelines + Iceberg read compatibility	Iceberg metadata generated asynchronously; non-Databricks writes constrained
Metadata sync to Delta	BI activation in Databricks SQL	Metadata copy, but raw files remain zero-copy

Summary: What We Built

Layer	Technology	Status
Storage	FSx for ONTAP (files) + S3 Tables (metadata)	✅ Verified
AI	Bedrock Claude Vision + Titan Embeddings V2	✅ Verified
Search	OpenSearch Serverless NextGen (scale-to-zero)	✅ Verified
Governance	Lake Formation (table-level) + CloudTrail	✅ Verified
PII	Comprehend (EN) + Bedrock Claude (JA)	✅ Verified
Cross-platform	Athena ✅, EMR Spark ✅, PyIceberg ✅, Snowflake ✅, Databricks ⚠️	Mostly verified

The Numbers

42 seconds: Full demo execution time
$0.07: Total demo cost
Near $0 idle compute/search cost: Persistent metadata, logs, and audit trails may still incur small charges
$114/month: Projected cost at 100K files, 1000 changes/day
95%: Storage cost reduction vs S3 full copy
0.95: AI classification confidence (invoice detection)
7/7: PII entities detected and redacted

For regulated workloads, align Iceberg snapshot retention with deletion SLAs and audit evidence retention.

What's Next for This Project

Monitor support cases: Databricks UC Foreign Catalog for S3 Tables — timeline unknown
Production hardening: SQS batching, DLQ alerting, reconciliation jobs
Multi-language PII: Extend beyond EN/JA to other languages
Cost optimization: Provisioned Throughput for high-volume Bedrock usage
Production semantics: File identity, latest-record views, index reconciliation, and snapshot retention alignment
ONTAP production hardening: S3 Access Point identity matrix, FPolicy event filtering, SnapMirror catalog rebinding, and FSx performance dashboard
Snowflake governance: Implement Horizon Row Access Policies and Dynamic Data Masking for column-level protection (since Lake Formation column-level is not enforced via VENDED_CREDENTIALS)

Get Involved

⭐ Star the repo if this was useful
🐛 Open an Issue for questions or suggestions
🍴 Fork and adapt for your own unstructured data catalog

This concludes the 3-part series. All code is at github.com/Yoshiki0705/fsxn-lakehouse-integrations. Questions? Open a GitHub Issue.

Governance disclaimer: This article provides governance guidance and architectural patterns. It does not substitute for legal or compliance judgment. Final regulatory determinations should be confirmed with legal and compliance teams.

AI Enrichment Pipeline: From Sample Classification to 100K-File Metadata Search with Bedrock and OpenSearch NextGen

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Mon, 08 Jun 2026 16:37:44 +0000

Quick Recap: What We Built in Part 1

In Part 1, we built a metadata catalog on Apache Iceberg (S3 Tables) that makes unstructured files on FSx for ONTAP instantly searchable via Athena SQL — in under 2 seconds, at $5-15/month for 100K files, without bulk-copying raw files.

But basic metadata (file name, size, type) only gets you so far. What if you could ask: "Find all invoices from Q4" or "Show me files similar to this contract"?

That requires AI enrichment: automatically classifying files and generating vector embeddings for similarity search.

What We're Building

File on FSx for ONTAP
       │
       │ S3 Access Point (read)
       ▼
┌─────────────────────────────────────────┐
│  Bedrock Claude Vision                  │
│  "What is this file?"                   │
│  → classification: "invoice"            │
│  → confidence: 0.95                     │
│  → summary: "Invoice #INV-2024-..."     │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  Titan Embeddings V2                    │
│  "Represent this file as a vector"      │
│  → 1024-dimensional embedding           │
│  → normalized for cosine similarity     │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  S3 Tables (Iceberg)                    │
│  classification, confidence_score,      │
│  summary, embedding_vector              │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  OpenSearch Serverless NextGen          │
│  kNN vector search                      │
│  "Find files similar to X"              │
│  Scale-to-zero: $0 when idle            │
└─────────────────────────────────────────┘

AI Classification: Bedrock Claude Vision

How It Works

For image files (PNG, JPEG, TIFF), we send the file to Claude Vision with a simple prompt:

response = bedrock.invoke_model(
    modelId="anthropic.claude-3-haiku-20240307-v1:0",
    body=json.dumps({
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": 512,
        "messages": [{"role": "user", "content": [
            {"type": "image", "source": {
                "type": "base64",
                "media_type": "image/png",
                "data": image_b64
            }},
            {"type": "text", "text": 
                'Classify this image. Respond JSON only: '
                '{"classification":"...","confidence_score":0.X,"summary":"..."}'}
        ]}]
    })
)

Results (Measured)

File	Classification	Confidence	Latency	Cost
invoice_sample.png	Invoice	0.95	~3s	$0.01
product_inspection.png	Pie Chart	1.0	~2s	$0.01
sensor_dashboard.png	IoT Sensor Dashboard	0.9	~3s	$0.01

Key insight: In this demo, Claude 3 Haiku classified sample images in ~2-3 seconds at roughly $0.01/image. Production accuracy and cost depend on image size, prompt length, model version, and document type.

Model version note: Model ID anthropic.claude-3-haiku-20240307-v1:0 was used at time of testing. Check Bedrock model IDs for the latest available version.

For Non-Image Files

File Type	Enrichment Strategy	Cost
PDF	Extract text → summarize with Claude	$0.01-0.05
CSV/Parquet	Schema extraction + row count	~$0 (metadata only)
Audio	Transcribe → summarize	$0.02-0.10
Video	Frame sampling → Vision	$0.05-0.50
CAD/3D	Metadata extraction only	~$0

Vector Embeddings: Titan Embeddings V2

Every file gets a 1024-dimensional vector embedding based on its content or AI-generated description:

response = bedrock.invoke_model(
    modelId="amazon.titan-embed-text-v2:0",
    body=json.dumps({
        "inputText": summary_text,  # AI-generated description
        "dimensions": 1024,
        "normalize": True
    })
)
embedding = json.loads(response["body"].read())["embedding"]
# → [0.023, -0.041, 0.089, ...] (1024 floats)

Why 1024 Dimensions?

Dimensions	Cost	Accuracy	Storage	Use Case
256	Lowest	Good	1KB/file	High-volume, cost-sensitive
512	Low	Better	2KB/file	General purpose
1024	Medium	High	4KB/file	Recommended balance
1536	Higher	Highest	6KB/file	Maximum precision

1024 dimensions was a practical default for this PoC. Validate 256/512/1024/1536 dimensions against your own top-k relevance and storage/cost targets (~4KB per file × 100K files = 400MB total at 1024-dim).

Pricing note: Titan Embeddings V2 charges per 1K input tokens ($0.00002). The cost is the same whether you request 256, 512, or 1024 dimensions — so there's no cost penalty for choosing higher dimensions.

Embedding Storage in Iceberg

Embeddings are stored as binary type in the Iceberg table:

import struct

# Convert float list to binary for Iceberg storage
embedding_bytes = struct.pack(f"{len(embedding)}f", *embedding)

# Write to Iceberg table
arrow_table = pa.table({
    "file_id": [file_id],
    "embedding_vector": [embedding_bytes],
    "enrichment_status": ["completed"],
    ...
})
table.append(arrow_table)

Important: Append-Only Writes and Deduplication

Iceberg on S3 Tables uses append-only writes. If you enrich the same file twice (e.g., after a retry), you'll get duplicate records. Use this dedup pattern in queries:

WITH ranked AS (
  SELECT *, ROW_NUMBER() OVER (PARTITION BY file_id ORDER BY modified_at DESC) as rn
  FROM "s3tablescatalog/fsxn-metadata-catalog"."metadata"."unstructured_files"
)
SELECT * FROM ranked WHERE rn = 1 AND is_deleted = false;

S3 Tables auto-compaction handles the storage overhead of duplicates over time.

Vector Search: OpenSearch Serverless NextGen

The Scale-to-Zero Revolution

Before May 2026, OpenSearch Serverless had a minimum cost of ~$350/month (2 OCUs always running). This made it impractical for PoC and dev environments.

OpenSearch Serverless NextGen (GA May 2026) introduces scale-to-zero:

State	Cost	Latency
Idle (no queries)	$0/month	—
Cold start (first query)	$0.24/OCU-hour	10-30 seconds
Warm (subsequent queries)	$0.24/OCU-hour	~54ms

This changes the economics completely: you can keep vector search compute cost near zero until you actually need it.

kNN Search Implementation

from opensearchpy import OpenSearch
from requests_aws4auth import AWS4Auth

# Generate query embedding
query_embedding = get_embedding("find invoice or payment documents")

# kNN search
results = client.search(index="fsxn-metadata-embeddings", body={
    "size": 5,
    "query": {
        "knn": {
            "embedding_vector": {
                "vector": query_embedding,
                "k": 5
            }
        }
    }
})

Note: Vector search requires OpenSearch — you cannot perform kNN queries directly on the embedding_vector binary column in Athena. The Iceberg table stores embeddings for durability; OpenSearch provides the search interface.

Search Results (Measured)

Query: "find invoice or payment documents"

Results:
  1. invoice_sample.png (score: 0.6749)
     Classification: Invoice
     Summary: "Invoice #INV-2024-..."

  2. (other similar files ranked by cosine similarity)

Score interpretation:

0.9+: Near-identical content
0.7-0.9: Highly similar
0.5-0.7: Related topic
< 0.5: Weak or no relation

Our score of 0.67 for "invoice or payment documents" → invoice_sample.png is reasonable — the query is broad, and the match is correct.

Improving search scores: Use more specific queries ("Q4 2024 invoice from vendor ABC" vs "find invoices"), enrich files with more detailed summaries, or increase embedding dimensions to 1536 for higher precision at ~50% more storage cost.

These score bands are demo heuristics, not universal thresholds. Calibrate thresholds with labeled examples for each document type and business workflow.

The Complete Pipeline

Processing Flow

New file detected (FPolicy event or batch scan)
       │
       ▼
┌─ Is it an image? ──────────────────────────┐
│  YES → Claude Vision classification        │
│  NO  → Metadata-only (file type, size)     │
└────────────────────────────────────────────┘
       │
       ▼
┌─ Generate embedding ──────────────────────┐
│  Input: classification + summary text     │
│  Output: 1024-dim normalized vector       │
└───────────────────────────────────────────┘
       │
       ▼
┌─ Write to S3 Tables (Iceberg) ────────────┐
│  classification, confidence_score,        │
│  summary, embedding_vector,               │
│  enrichment_status = "completed"          │
│  index_status = "pending"                 │
└───────────────────────────────────────────┘
       │
       ▼
┌─ Index in OpenSearch ─────────────────────┐
│  file_id, embedding_vector, metadata      │
│  (for kNN similarity search)              │
│  index_status = "indexed" / "stale" / "failed" │
└───────────────────────────────────────────┘

Error Handling

Error	Strategy	Result
Bedrock ThrottlingException	Exponential backoff (1s, 2s, 4s)	Retry up to 3 times
Bedrock ModelNotReadyException	Wait 5s, retry	Model warming up (first invocation)
File read failure (S3 AP)	Mark as `failed`, retry next cycle	No data loss
Low confidence (< 0.3)	Mark as `low_confidence`	Human review queue
Lambda timeout (large files)	Fallback to ECS Fargate	No timeout limit

Monitoring the Pipeline

How do you know when something goes wrong? Set up these CloudWatch alarms:

Metric	Source	Alert Condition	Action
DLQ message count	CloudWatch (SQS)	> 0	Inspect DLQ messages, redrive
Lambda error rate	CloudWatch (Lambda)	> 5% for 5 min	Check logs, Iceberg commit conflict?
Bedrock throttling	CloudWatch (Bedrock)	> 10/min	Reduce request rate, adjust backoff
Enrichment backlog	Athena query (pending count)	> 1000	Increase Lambda concurrency or batch size
OpenSearch index size	OpenSearch metrics	> 80% capacity	Add shards or rotate index

# Quick health check: DLQ + Lambda errors in one command
aws cloudwatch get-metric-data --metric-data-queries '[
  {"Id":"dlq","MetricStat":{"Metric":{"Namespace":"AWS/SQS","MetricName":"ApproximateNumberOfMessagesVisible","Dimensions":[{"Name":"QueueName","Value":"fsxn-metadata-sync-dlq"}]},"Period":300,"Stat":"Sum"}},
  {"Id":"errors","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Errors","Dimensions":[{"Name":"FunctionName","Value":"fsxn-metadata-sync"}]},"Period":300,"Stat":"Sum"}}
]' --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%S) --end-time $(date -u +%Y-%m-%dT%H:%M:%S)

For detailed operational monitoring guidance, see the Operational Monitoring section in the architecture document.

Cost at Scale

Volume	AI Cost	Embedding Cost	OpenSearch	Total
100 files/day	$1/day	$0.002/day	$0 (idle)	~$30/month
1,000 files/day	$10/day	$0.02/day	~$42/month	~$342/month
10,000 files/day	$100/day	$0.20/day	~$84/month	~$3,084/month

At 10K files/day, consider batch processing during off-hours and Provisioned Throughput for Bedrock to reduce per-request cost.

Cost optimization tip: Not all files need AI enrichment. A common pattern: images → Vision classification, documents → text extraction + embedding, data files (CSV/Parquet) → metadata only (no AI cost). This can reduce AI costs by 60-80% depending on your file mix.

Batch Inference: For initial bulk enrichment (10K+ files), Bedrock Batch Inference can reduce costs by ~50% compared to real-time invocations. Use real-time for incremental new files, batch for backfill.

# Batch Inference example — submit a batch job for bulk classification
import boto3, json

bedrock = boto3.client("bedrock", region_name="ap-northeast-1")

# 1. Prepare input JSONL file in S3 (one request per line)
# Each line: {"recordId":"file-001","modelInput":{"anthropic_version":"bedrock-2023-05-31",...}}

# 2. Create batch inference job
response = bedrock.create_model_invocation_job(
    jobName="metadata-enrichment-backfill",
    modelId="anthropic.claude-3-haiku-20240307-v1:0",
    roleArn="arn:aws:iam::<ACCOUNT>:role/BedrockBatchRole",
    inputDataConfig={
        "s3InputDataConfig": {
            "s3Uri": "s3://my-bucket/batch-input/enrichment-requests.jsonl"
        }
    },
    outputDataConfig={
        "s3OutputDataConfig": {
            "s3Uri": "s3://my-bucket/batch-output/"
        }
    }
)
# Job runs asynchronously — results written to S3 when complete
# Typical processing: 10K files in ~30 minutes at ~50% cost reduction

The batch input JSONL contains prompts, file references, or extracted/redacted text depending on your design. It does not require copying the original raw files from FSx for ONTAP to S3. If images are included as base64, treat the JSONL as temporary processing data.

Batch job monitoring: Use EventBridge rules to detect Bedrock batch job state changes (COMPLETED, FAILED). Route to SNS → Lambda to automatically write results back to S3 Tables.

Prompt Caching: If using the same system prompt across all classifications (recommended), Bedrock's Prompt Caching feature can reduce input token costs by up to 90% for repeated prompts.

EMR Spark for large-scale backfill: For initial backfill or re-enrichment of 100K+ files, Spark on EMR Serverless or EMR on EC2 can be used as an alternative to Lambda/Fargate. EMR 7.13.0+ supports Glue as an Iceberg REST catalog, enabling distributed metadata writes with Lake Formation governance. Verified 2026-06-02: SELECT, COUNT, and time travel all work on EMR Serverless 7.13.0. Use Lambda for incremental (event-driven) processing and Spark for bulk operations.

Search Index Consistency

OpenSearch is a derived index, not the system of record. S3 Tables / Iceberg remains the metadata source of truth.

Recommended controls:

Store iceberg_snapshot_id in OpenSearch documents for traceability
Store embedding_model_id and prompt_version in both Iceberg and OpenSearch
Reconcile OpenSearch index against latest Iceberg view periodically
Mark index_status: pending / indexed / stale / failed
If search returns a stale result, fall back to Athena query on the base table

FPolicy Event Design

For incremental metadata sync via ONTAP FPolicy:

Use batch scan for initial backfill (not FPolicy)
Use FPolicy only for incremental changes after initial catalog population
Prefer create / close-with-modification / rename / delete events
Avoid read / open events (excessive volume, no catalog value)
Apply path and extension filters to reduce event noise
Add backpressure via SQS batching (not fan-out Lambda per event)

FPolicy can significantly impact file system throughput if configured too broadly. Filter to only the operations and paths that matter for catalog updates.

Hybrid Search Pattern

For production discovery, vector search should be combined with lexical filters and keyword search:

Lexical search: file_name, path, classification, summary, tags
Vector search: embedding similarity (kNN)
Filters: tenant_id, sensitivity_level, file_type, path_classification, last_modified

OpenSearch supports both search and vector collection types. Use a single index with both text fields and vector fields for hybrid queries. S3 Tables / Iceberg remains the metadata source of truth; OpenSearch is the serving index.

For sensitive workloads, use VPC interface endpoints for Bedrock Runtime and S3 VPC endpoints for batch input/output. See genai/bedrock-private-connectivity.md.

Storage Tier Impact During Backfill

Initial AI enrichment may read cold files from capacity pool storage, causing higher latency and throughput consumption.

Recommended controls:

Run backfill during off-hours (minimize impact on production NFS/SMB)
Limit Lambda concurrency during backfill
Enrich only selected file types first (images → documents → data files)
Monitor FSx capacity pool read activity via CloudWatch
Separate backfill cost from steady-state cost in planning

Backfill vs Incremental Cost Model

Separate cost planning for:

Phase	Scope	Cost driver	Optimization
Initial backfill	All existing files (e.g., 100K)	Bedrock AI at scale	Batch Inference (~50% savings)
Daily incremental	New/modified files (e.g., 1000/day)	Real-time Lambda + Bedrock	Selective enrichment by file type
Re-enrichment	After prompt/model change	Full re-scan of enriched files	Batch + compare confidence delta
OpenSearch reindex	After schema/embedding change	Index rebuild	Off-hours, parallel shards

The largest cost spike is typically the initial backfill, not steady-state. Plan Bedrock Batch Inference and off-peak scheduling for the first catalog population.

For adjustable assumptions, see verification-evidence/cost-assumptions.yaml.

Try It Yourself

# Enrich pending files with AI
python3 demo/scripts/demo-enrich.py \
  --table-bucket-arn <TABLE_BUCKET_ARN> \
  --ap-alias <AP_ALIAS> \
  --max-files 10

# Search by natural language
python3 demo/scripts/demo-search.py \
  --query "find documents about contracts or agreements"

AI Safety and Human Review Boundary

AI enrichment should not be treated as authoritative classification for regulated data without human review.

For regulated industries: AI enrichment is assistive metadata generation, not authoritative classification. Final regulatory classification must be confirmed by data owners, security, legal, and compliance teams. This system provides AI-generated signals to accelerate human review — it does not replace it.

Deterministic vs AI boundary: AI generates classifications and summaries, but pipeline state transitions, retry logic, deduplication, access controls, and audit evidence are deterministic and version-controlled. The deterministic pipeline guarantees reproducibility; AI provides enrichment quality.

Recommended controls:

Human review queue for low-confidence classifications (< 0.7)
Sampling review for high-confidence results (periodic spot-check)
False negative testing for PII detection
Model/prompt version recorded in metadata (enriched_at + model ID)
Re-enrichment policy when model or prompt changes

Recommended metadata columns for AI lineage:

classification_model_id — which model produced the classification
embedding_model_id — which model produced the embedding
prompt_version — version of the classification prompt
enrichment_code_version — version of the enrichment Lambda/script
enriched_at — timestamp of enrichment
human_review_status — pending / approved / rejected
human_reviewed_by — reviewer identity (if applicable)
human_reviewed_at — review timestamp

Evaluation Plan

For production use, do not rely only on model-reported confidence. Create a labeled validation set and measure:

Classification accuracy (overall and per document type)
Precision / recall per category
False positive rate for PII detection
False negative rate for PII detection
Embedding search top-k relevance (nDCG@5, MRR)
Human review acceptance rate
Cost per accepted classification

Business acceptance metrics (beyond model accuracy):

Time saved per analyst for file discovery
Dataset discovery lead-time reduction (days → hours target)
Business owner approval rate for AI classifications
Cost per useful search result
False negative risk by document category (which misses matter most?)
Governance coverage (% of assets searchable in BI/AI tools)

The 7/7 PII detection result was measured on a controlled synthetic sample. Production use requires evaluation with domain-specific documents, false-positive/false-negative review, human approval workflow, and legal/compliance sign-off.

Snowflake users: Snowflake can now directly query S3 Tables Iceberg metadata via Glue REST + VENDED_CREDENTIALS (verified 2026-06-05). Additionally, you can sync redacted metadata into Snowflake-managed tables for Cortex Search / Snowflake Intelligence business-facing discovery. In this PoC, OpenSearch remains the AWS-native vector search component.

What's Next

In Part 3, we'll cover:

Lake Formation governance: Column-level access control on metadata
PII detection and anonymization: Comprehend (English) + Bedrock Claude (Japanese)
Cross-platform access: What works and what doesn't with Databricks and Snowflake
Data Clean Room pattern: Separate tables for sensitive vs. anonymized metadata

Full code: github.com/Yoshiki0705/fsxn-lakehouse-integrations

From Hours to Seconds: An AI-Powered Metadata Catalog for Unstructured Data on FSx for ONTAP

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Mon, 08 Jun 2026 16:26:50 +0000

What Works Now vs What Requires Validation

This article separates verified AWS-native capabilities from cross-platform paths that still require validation. The core pattern — keeping raw files on FSx for ONTAP and cataloging only metadata in S3 Tables — is verified. Databricks paths are still evolving. Snowflake Glue REST + VENDED_CREDENTIALS and External Stage paths are verified in this PoC, with governance limitations noted below. Validate all cross-platform paths in your own environment before production use.

Component	Status	Notes
AWS Native PoC (Athena + S3 Tables + Bedrock + OpenSearch + Lake Formation)	✅ Verified	Full end-to-end in 42 seconds
Glue Iceberg REST endpoint access	✅ Verified	Both S3 Tables REST and Glue REST confirmed
Lake Formation table-level governance	✅ Verified	Grant/revoke/audit working
Lake Formation column-level exclusion	⚠️ Observed limitation	Failed on tested federated catalog path
Databricks SQL Warehouse direct	⚠️ Observed limitation	`iceberg_rest` connection type not supported
Databricks Spark + Iceberg REST	❌ Blocked by UC	spark.conf.set and cluster config both fail; UC Foreign Catalog required
Databricks UC Foreign Catalog	❌ Still blocked	Retested post-Foreign Iceberg GA (2026-06-09): Glue Connection ✅, Credentials ✅, but External Location fails — S3 Tables internal bucket rejects standard S3 API validation. No bypass available.
Databricks Delta Sharing via S3 AP	❌ Confirmed	Sharing server uses same UC credentials; not a workaround for S3 AP session policy
Databricks NFS → UC Volume	❌ Confirmed	Cloud storage URIs only; internal feature request exists
Databricks UC audit logging	✅ Confirmed	External engine access fully logged
Snowflake via Glue REST (VENDED_CREDENTIALS)	✅ Verified	Explicit `ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS`; CREATE TABLE + SELECT + COUNT + AUTO_REFRESH all working (2026-06-05)
Snowflake External Stage (FSx S3 AP)	✅ Verified	LIST, SELECT/COPY, and TO_FILE + Cortex AI all verified

Important distinction: This pattern does not use FSx for ONTAP S3 Access Points as an Iceberg warehouse. Raw files stay on FSx for ONTAP, while only the metadata catalog is written to S3 Tables. Direct Iceberg table writes to FSx for ONTAP S3 Access Points are tracked separately as a known limitation because Iceberg commit behavior and S3FileIO compatibility require additional validation.

This is an Iceberg Adoption Pattern, Not a Raw-Data Migration

This pattern does not convert the original unstructured files into Iceberg table data. Instead, it adopts Iceberg for the metadata layer only.

Scope	What happens
Data files	Not migrated. Raw files remain on FSx for ONTAP.
Metadata table	Newly created as an Iceberg table on S3 Tables.
Processing jobs	Metadata scan and AI enrichment jobs write append-only metadata.
Consumers	Athena, EMR, Snowflake, Databricks, and BI/search tools consume curated metadata views.

Storage Boundary: What Moves and What Doesn't

FSx for ONTAP S3 Access Point:
  ✅ Raw file READ path only (AI enrichment input)
  ❌ NOT an Iceberg warehouse
  ❌ NOT a table commit target
  ❌ NOT bulk-copied to S3

S3 Tables:
  ✅ Iceberg METADATA table (file catalog)
  ✅ Metadata source of truth
  ✅ Query and governance target

Data movement disclosure (for regulated environments): Raw files are NOT bulk-copied to S3. However, during AI enrichment, selected file content is temporarily read via the S3 Access Point and sent to Amazon Bedrock APIs for classification/embedding. Per AWS Bedrock data protection policy, model providers have no access to customer prompts or completions. Extracted/redacted metadata and embeddings are written to S3 Tables, OpenSearch, and optionally to Snowflake or Databricks depending on the activation path. Define your data flow boundary documentation before regulated-workload deployment.

The Problem: Most Enterprise Unstructured Data is Difficult to Discover and Govern

Most organizations store terabytes of unstructured data — PDFs, images, CAD files, sensor logs — on network-attached storage. This data is:

Undiscoverable: "Where is that invoice from last quarter?" requires manual searching or asking colleagues
Governed at the file-system layer, but not classified or searchable from analytics and AI workflows
Audit trails may exist at the file-system layer, but they are often not unified with analytics and AI query activity

Think of this as unstructured-data modernization: inventory first, classify selectively, govern metadata, and activate only what is needed — without bulk-copying the raw files.

Business Outcomes (Beyond Technical Metrics)

This pattern is not only about faster file search. It is about:

Reducing dataset discovery lead time for AI projects (days → hours)
Improving PII visibility across the organization (unknown → 95%+ coverage target)
Lowering duplicate storage cost ($230-256/month eliminated for 10TB)
Creating governed metadata products for analytics and AI teams
Enabling AI-readiness without raw-data copy or migration
Activating governed metadata in Snowflake AI Data Cloud for Cortex Search, semantic Q&A, executive dashboards, and business-facing file discovery

The traditional solution? Copy everything to S3 and build a catalog. But at 10TB, that's ~$230-256/month just for the copy — plus sync pipelines, duplicate governance, and data drift.

The Solution: Hot Metadata × Cold Data

What if we could catalog every file without moving it?

┌─────────────────────────────────────────────────────────┐
│  HOT: Metadata (Apache Iceberg on S3 Tables)            │
│  • File path, type, size, timestamps                    │
│  • AI classification + confidence score                 │
│  • Vector embedding (1024-dim, similarity search)       │
│  • PII detection flag                                   │
│  • Cost: ~$5-15/month for 100K files                    │
└────────────────────────┬────────────────────────────────┘
                         │ file_path reference
┌────────────────────────▼────────────────────────────────┐
│  COLD: Actual Files (FSx for ONTAP)                     │
│  • PDF, images, CAD, video, audio, logs                 │
│  • Deduplication (50-70% storage savings typical*)      │
│  • NFS/SMB (existing workflows) + S3 AP (AI/analytics)  │
│  • No bulk raw-data copy required                       │
└─────────────────────────────────────────────────────────┘

Key insight: Keep the data where it is. Move only the metadata into a queryable format.

Architecture

FSx for ONTAP ──S3 Access Point──→ AI Enrichment (Bedrock)
       │                                    │
       │                                    ▼
       │                          S3 Tables (Iceberg)
       │                                    │
       │                                    ▼
       │                          ┌──────────────────┐
       │                          │ Query Engines    │
       │                          │ • Athena (SQL)   │
       │                          │ • OpenSearch     │
       │                          │   (vector kNN)   │
       │                          │ • Lake Formation │
       │                          │   (governance)   │
       │                          └──────────────────┘
       │
       └──NFS/SMB──→ Existing applications (unchanged)

Observability (production add-on):

       ┌──────────────────────────────────────┐
       │  • CloudWatch Metrics + Alarms       │
       │  • CloudWatch Logs (Lambda/SQS)      │
       │  • CloudTrail (governance audit)     │
       │  • OpenSearch Dashboards (search UX) │
       │  • FSx metrics (throughput, IOPS,    │
       │    latency, capacity pool reads)     │
       └──────────────────────────────────────┘

Components:

Component	Role	Cost
FSx for ONTAP S3 Access Point	Read files for AI processing (no copy)	Included with FSx
S3 Tables	AWS managed Apache Iceberg table service (auto-compaction, REST endpoint)	~$5/month metadata
Bedrock Claude Vision	Image classification	~$0.01/file in this demo
Titan Embeddings V2	1024-dim vectors for similarity search	$0.00002/1K input tokens
OpenSearch Serverless NextGen	kNN vector search (scale-to-zero)	$0 idle compute when inactive
Lake Formation	Metadata access governance	No additional Lake Formation charge

S3 Tables Iceberg REST endpoint: https://s3tables.<region>.amazonaws.com/iceberg
Check S3 Tables availability for regional support before deployment.

Deduplication ratio is a general ONTAP range. Actual savings depend on data characteristics and were not measured in this PoC.

PoC Results (Verified 2026-05-31)

We built and verified this end-to-end in a single day. Here's what we measured:

S3 Tables Access Paths: Which Endpoint Should You Use?

Access path	Best for	Governance path	Verified
S3 Tables Iceberg REST (`s3tables.<region>.amazonaws.com/iceberg`)	Direct Iceberg client / simple PoC	IAM + S3 Tables permissions	✅
AWS Glue Iceberg REST (`glue.<region>.amazonaws.com/iceberg`)	Production analytics integration	IAM + Lake Formation	✅
Athena via Glue federated catalog	SQL analytics	Lake Formation + Athena	✅
PyIceberg local client	Lightweight validation	IAM/LF depending on endpoint	✅

For production workloads with centralized governance, the AWS Glue Iceberg REST endpoint is recommended over the S3 Tables direct endpoint. See AWS docs.

Catalog authority rule: S3 Tables + Glue is the authoritative catalog for this metadata table in this PoC. Other engines should consume the table through the authoritative catalog or a controlled metadata activation path. Do not configure multiple writable catalogs for the same Iceberg table — dual-write causes split-brain and potential data corruption.

Athena Iceberg behavior depends on Athena engine version, Iceberg version, Glue/Lake Formation integration, and table maintenance state. Validate DDL/DML requirements separately before using this as a write-heavy production catalog.

Verification details are recorded in evidence-record.yaml and cross-platform-compatibility.yaml.

Before vs After

Metric	Before	After	Improvement
File discovery time	Minutes-hours	< 2 seconds	100x+ at scale
AI classification	Manual	Automatic (6 sec/file)	Fully automated
Storage cost (10TB)	~$250/month (S3 copy)	$5-15/month (metadata only)	95% reduction
Metadata query governance	Not applicable	100% in this PoC	Complete for metadata queries
Idle compute/search cost	N/A	Near $0 when inactive	Persistent metadata/logs may still incur small charges

Search Time Scaling (Measured + Projected)

Files	ListObjectsV2	Athena SQL	Speedup
40	892 ms	3.0 sec	0.3x
1,000	22.3 sec	1.8 sec	12x
10,000	3.7 min	1.8 sec	124x
100,000	37.2 min	1.8 sec	1,239x
1,000,000	371.7 min	1.8 sec	12,389x

At 40 files, ListObjectsV2 is faster — Athena has cold start overhead. Athena query time does not scale linearly with the number of files on FSx because it queries the Iceberg metadata table instead of listing the raw file namespace. In this controlled demo, the query stayed around ~1.8 seconds for projected file counts, but production latency depends on Iceberg metadata size, manifest count, predicate selectivity, Athena cold start, and table maintenance state.

Projection method: ListObjectsV2 latency was extrapolated linearly from the measured 40-file scan. This is intentionally conservative for demonstrating namespace-scan behavior, but it is not a service benchmark.

The 42-Second Demo

Our complete demo runs all 8 steps in 42 seconds:

Step 1: Before/After search comparison     ✅ (ListObjectsV2 vs Athena)
Step 2: Infrastructure deploy              ✅ (CloudFormation, skippable)
Step 3: Metadata scan (40 files)           ✅ (3 seconds)
Step 4: AI enrichment (Bedrock Vision)     ✅ (invoice → 0.95 confidence)
Step 5: Athena query + Time Travel         ✅ (< 2 seconds)
Step 6: Vector similarity search           ✅ (kNN score 0.67)
Step 7: PII detection + anonymization      ✅ (7/7 entities, all redacted)
Step 8: Cost & ROI analysis                ✅ ($0.07 total demo cost)

Total demo cost: $0.07. After the demo, the compute/search components can scale to zero. If you retain S3 Tables metadata, logs, or audit trails, small storage/logging charges may still apply.

AI Classification Results

File	Classification	Confidence
invoice_sample.png	Invoice	0.95
product_inspection.png	Pie Chart	1.0
sensor_dashboard.png	IoT Sensor Dashboard	0.9

In this demo, Bedrock Claude Vision classified sample images at roughly $0.01/file with sub-10-second latency. Production cost and latency depend on image size, prompt length, model version, and retry behavior.

Vector Similarity Search

Query: "find invoice or payment documents"
→ invoice_sample.png (score: 0.6749)

OpenSearch Serverless with scale-to-zero capability (GA May 2026) provides kNN search — no minimum cost when idle. Cold start is ~10-30 seconds, warm queries are ~54ms.

Verified in this PoC environment on 2026-05-31. Check the latest OpenSearch Serverless documentation and regional availability before deployment.

Governance: Lake Formation Access Control

Step 1: Authorized query    → ✅ SUCCEEDED (3 rows)
Step 2: Revoke SELECT       → 🔒 BLOCKED (access denied)
Step 3: Restore SELECT      → ✅ SUCCEEDED
Step 4: CloudTrail audit    → All queries logged with user identity

Metadata queries are governed and audited. Raw file access remains governed separately by FSx file-system permissions, S3 Access Point policies, and application access paths.

Cost Analysis

This Demo

Component	Cost
Bedrock AI (5 files)	$0.05
OpenSearch (~6 min)	$0.024
Lambda + Athena	$0.001
Total	$0.07

Projected Monthly (10TB, 100K files, 1000 changes/day)

Component	Monthly
S3 Tables (metadata)	$5
Lambda (sync + AI)	$36
Bedrock (AI enrichment)	$30
OpenSearch (business hours)	$42
SQS + misc	$1
Total	$114/month
S3 copy eliminated	-$230-256/month

Net effect: The AI-powered catalog costs less than the S3 copy it eliminates.

Without AI enrichment (metadata scan + Athena only): ~$42/month. AI processing is optional and can be enabled per-file-type.

S3 Standard pricing: us-east-1 $0.023/GB, ap-northeast-1 $0.025/GB. Verified 2026-06-01 via AWS Pricing API.

For reproducibility, see: evidence-record.yaml, cost-assumptions.yaml, comprehensive-test-results.yaml

Known Limitations (Honest Assessment)

Limitation	Impact	Workaround
Databricks SQL Warehouse `CREATE CONNECTION TYPE iceberg_rest` to S3 Tables REST failed in this validation (2026-05-31)	SQL Warehouse direct path unavailable in tested method	Retested 2026-06-09; still blocked in tested UC path. Use curated metadata sync to UC Delta as practical workaround; support case submitted.
Databricks Spark cluster: UC blocks external catalog registration (2026-06-01)	Cannot use spark.conf.set or cluster config for external Iceberg catalogs	UC Foreign Catalog tested 2026-06-09 — External Location validation fails against S3 Tables internal bucket. Sync metadata to UC Delta table instead.
Databricks Delta Sharing: cannot bypass S3 AP session policy (2026-06-01)	Sharing server uses same UC credentials	DataSync → S3 → UC → Delta Sharing works for copied data; validate target table format and catalog support separately
Databricks NFS mount: cannot register as UC External Volume (2026-06-01)	NFS/FUSE paths not supported for UC Volumes	DataSync → S3 → UC External Location; internal feature request exists
Snowflake External Iceberg Table with S3 Tables REST endpoint was not a supported catalog type in this validation (2026-05-31)	Direct S3 Tables REST path unavailable in tested method	✅ Resolved (2026-06-05): Use Glue REST + explicit `ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS`. Schema must have no default External Volume. AWS prerequisite: `register-resource --with-federation`. Lake Formation column-level filtering NOT enforced via this path.
LF column exclusion grant failed in tested S3 Tables federated catalog path	Can't hide specific columns via tested grant pattern	Athena Views; track AWS support status
At 40 files, ListObjectsV2 is faster than Athena	Architecture value is at scale (100K+)	Expected — Athena has cold start overhead

Naming note: Use lowercase table, namespace, and column names for S3 Tables integrated with AWS analytics services. Mixed-case names may not be visible to Athena / Glue / Lake Formation. See S3 Tables naming rules.

Performance Boundaries Not Yet Validated

This PoC validates the architecture shape, not production scale limits. The following require separate testing:

FSx throughput impact under concurrent NFS/SMB/S3 access
S3 Access Point metadata operation impact under large namespace scans
S3 API request concurrency vs FSx provisioned throughput capacity
Impact of scan jobs on production SMB/NFS latency
ListObjectsV2 pagination behavior at 1M+ files
Lambda concurrency and S3 AP request throttling
Iceberg manifest growth and compaction behavior
Athena query latency with high snapshot counts
OpenSearch indexing throughput during bulk backfill
File size distribution and small-file amplification effects
Cold vs warm namespace access behavior (capacity pool reads during backfill)

ONTAP Object Model Mapping

ONTAP / FSx object	Role in this pattern
FSx file system	Performance / HA boundary
SVM	Protocol and administrative boundary
Volume	Catalog scope and S3 Access Point attachment target
Junction path / SMB share	Existing application namespace
S3 Access Point	S3 API boundary for AI/analytics (with associated file-system identity)
Iceberg table	Metadata catalog, not raw data store

Each S3 Access Point has an associated OntapFileSystemIdentity (UNIX UID/GID or Windows domain user) that authorizes all file access through that AP. IAM policy is evaluated first, then ONTAP file-system permissions. See security/s3-access-point-identity-matrix.yaml.

Iceberg Table Maintenance Plan

For production, define:

Snapshot retention period and table maintenance behavior — verify S3 Tables service-managed policies and any configurable retention settings
Manifest rewrite cadence (if metadata table grows large)
Orphan file cleanup policy
Deduplication view or materialized latest-record table
Time travel retention policy
Athena engine version and Iceberg version compatibility
Append-only dedup query as default named query for analysts

For operational steps, see ops/iceberg-maintenance-runbook.md. For details on Iceberg spec vs S3 Tables service behavior, see docs/standards-vs-service-behavior.md.

Iceberg does not enforce primary-key uniqueness in this PoC. Consumers should query curated latest-record views instead of the append-only base table. See ops/athena-named-queries/latest_records.sql in the repo.

Apache Iceberg is the open table format. Amazon S3 Tables is an AWS managed table bucket service that uses Apache Iceberg. Some operational behavior, endpoint support, and governance integration are AWS service-specific and should be validated separately from the Iceberg specification itself.

File Identity Strategy

file_id method	Best for	Tradeoff
`hash(volume_id + normalized_path)`	General purpose	Rename = new file_id
`hash(volume_id + file_handle/inode)`	Rename tracking	Requires inode access
Content hash (SHA-256)	Immutable documents	Expensive for large files
`path + last_modified + size`	Lightweight PoC only	Fragile under overwrites

Production should define how rename, overwrite, delete, and permission changes are represented in the metadata table.

Recommended production columns: source_system_id, volume_id, normalized_path, path_hash, content_hash, scan_run_id, change_type (created / modified / deleted / renamed / permission_changed).

For FlexClone-based dev/test datasets, decide whether cloned files should retain lineage to source files. If lineage matters, store clone_parent_volume_id, clone_parent_snapshot_id, and catalog_environment (prod / dev / test / dr). See dr/snapmirror-catalog-rebinding.md for DR failover considerations.

For manufacturing and engineering workloads, see schema/extensions/manufacturing_metadata.yaml for domain-specific metadata fields such as part number, revision, plant, machine, and inspection lot.

Multi-Tenant Deployment Considerations

If this pattern is provided by a partner or platform team to multiple business units or customers, define the isolation boundary explicitly.

Isolation model	Recommended when	Tradeoff
Table bucket per tenant	Strong isolation required	Higher operational overhead
Namespace per tenant	Balanced isolation and operations	Shared table bucket governance required
tenant_id column in one table	Internal multi-BU catalog	Requires strict LF-Tags / row filters
OpenSearch index per tenant	Search isolation required	More index management
Shared OpenSearch index + tenant filter	Lower cost	Must enforce filter in every query path

For partner-led deployments, document tenant onboarding automation, offboarding deletion/retention policy, per-tenant cost allocation tags, and audit evidence location.

Business KPI Mapping

Business problem	Baseline metric	Target metric	How this PoC measures it
Employees cannot find documents	Average search time	< 10 sec	Search latency + result relevance
Manual classification is slow	Files classified/day/person	10x improvement	AI enrichment throughput
Sensitive files are unknown	% files classified for PII	95%+ coverage target	PII scan completion rate
Duplicate S3 copy is costly	Monthly duplicate storage cost	Reduce by 50%+	Metadata-only architecture cost
AI projects lack data inventory	Dataset discovery lead time	Days → hours	Catalog completeness
Business users need governed discovery	% searchable assets in BI/AI tools	80%+ of approved metadata visible	Expose curated metadata views to Athena, Databricks, Snowflake, or BI tools

Try It Yourself

FSx for ONTAP prerequisites:

SVM and volume selected as catalog scope
S3 Access Point attached to the target volume
Associated UNIX or Windows identity documented
NFS/SMB production workload impact reviewed
CloudWatch metrics dashboard enabled

# Clone the repo
git clone https://github.com/Yoshiki0705/fsxn-lakehouse-integrations.git
cd fsxn-lakehouse-integrations/integrations/iceberg-metadata-catalog

# Install dependencies
pip install -r requirements.txt

# Run the demo (requires FSx for ONTAP with S3 Access Point)
cd demo/scripts
./run-demo.sh --ap-alias <your-ap-alias-ext-s3alias>

Don't have FSx for ONTAP? You can still explore the architecture:

What's Next

This is Part 1 of a 3-part series:

Part 1 (this article): Architecture & PoC Results
Part 2: AI Enrichment Pipeline — Bedrock Vision + Titan Embeddings + OpenSearch NextGen
Part 3: Governance & Cross-Platform Access — Lake Formation, PII Anonymization, Databricks/Snowflake Integration

Key Takeaways

Don't copy data to make it searchable — catalog the metadata instead. Apache Iceberg + S3 Tables gives you a managed metadata layer with time travel.
Selective AI enrichment plus scale-to-zero search can keep PoC and low-traffic environments cost-efficient — compute/search components idle near $0; persistent metadata and logs may incur small charges.
42 seconds, $0.07 — that's the barrier to entry for an AI-powered data catalog on your existing NAS storage.
Start small, grow incrementally — from metadata-only scan (Level 1) to full business workflow integration (Level 5). See the Production Maturity Model for the progression path.

All code and documentation is available at github.com/Yoshiki0705/fsxn-lakehouse-integrations. Feedback welcome via GitHub Issues.

28 Industry Reference Patterns with FSx for ONTAP S3 Access Points — Phase 15

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 07 Jun 2026 03:00:23 +0000

TL;DR

Phase 15 expands the pattern library from 17 to 28 industry-specific use cases, providing reference implementations across major AWS Industry verticals where FSx for ONTAP file processing is relevant. Each new pattern includes a CloudFormation template, Step Functions workflow, Python Lambda functions, 8-language documentation, and property-based tests. Combined with 6 FlexCache/FlexClone patterns and 1 SAP/ERP pattern, the repository now offers 35 deployable reference patterns for enterprise file processing on FSx for ONTAP.

The SAP/ERP pattern focuses on controlled document/report processing around ERP-adjacent file exports (IDoc, spool), not direct transactional SAP data manipulation.

Important: These are reference implementations with production-readiness guidance, not fully certified production systems. Customers must validate against their own regulatory, security, and operational requirements before production use.

For S3 standard bucket users: This library is not a replacement for S3 data lake patterns. It is a file-data integration pattern for customers who want to process FSx ONTAP-resident data through S3-compatible APIs while preserving NAS access paths. See docs/s3-bucket-user-guide.md for a detailed comparison.

Serverless boundary: Compute (Lambda), orchestration (Step Functions), eventing (EventBridge), and AI services (Bedrock, Textract, Rekognition) are serverless/managed. FSx for ONTAP is a fully managed file system with provisioned capacity and operational considerations — it is not scale-to-zero storage. This is a serverless processing pattern over existing enterprise file data, not a pure serverless storage pattern.

When NOT to use this: If your workload is already object-native, does not require NFS/SMB coexistence, and can use standard S3 data lake patterns — prefer S3-native serverless architecture.

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

Why 28 Use Cases?

AWS organizes customers into 22 industry verticals. When we mapped our existing 17 patterns against these verticals, several gaps stood out:

Telecommunications — No CDR/network log processing pattern
Advertising & Marketing — No creative asset management
Travel & Hospitality — No document processing for reservations
Agriculture & Food — No traceability or crop monitoring
Sustainability/ESG — No ESG metrics extraction
Nonprofit — No grant management automation
Utilities — No drone/SCADA-based asset inspection
Real Estate — No portfolio analysis
HR — No resume screening (with PII protection)
Chemicals — No SDS/lab notebook processing
Transportation (railway) — No deterioration detection

Phase 15 fills all of these, covering 19 of 22 AWS Industry verticals (remaining 3 — Consumer Packaged Goods, Mining, Software/Internet — have limited file-processing relevance for this pattern type). Combined with 11 Japan-market focus areas (all covered), the repository addresses the vast majority of enterprise file processing scenarios.

The 11 New Patterns

P0: Foundation Patterns

UC	Industry	Key AWS Services	Differentiator
UC18	Telecom	Athena, Bedrock	CDR/syslog anomaly detection with 7-day baseline
UC19	AdTech	Rekognition, Textract, Bedrock	Brand compliance scoring + moderation

P1: Document Intelligence

UC	Industry	Key AWS Services	Differentiator
UC20	Travel	Textract, Comprehend, Rekognition	Multilingual reservation extraction + facility inspection
UC21	Agriculture	Rekognition, Textract, Bedrock	GeoTIFF crop analysis + lot traceability
UC22	Transportation	Rekognition, Textract, Bedrock	Safety-critical escalation trigger + deterioration trends

P2: Specialized Processing

UC	Industry	Key AWS Services	Differentiator
UC23	Sustainability	Textract, Bedrock	ESG metric extraction + GRI/TCFD/ISSB mapping
UC24	Nonprofit	Textract, Comprehend, Bedrock	Grant application + outcome matching
UC25	Utilities	Rekognition, Bedrock, Athena	Drone + SCADA + thermal tri-modal inspection
UC26	Real Estate	Rekognition, Textract, Bedrock	Property analysis + lease extraction + PII flagging
UC27	HR	Textract, Comprehend, Bedrock	Recruiting document triage with PII protection
UC28	Chemicals	Textract, Rekognition, Bedrock	SDS hazard extraction + GHS compliance + lab notebook

Architecture: One Pattern, Many Industries

Architecture Classification

Layer	Classification
Workflow orchestration	Serverless (Step Functions)
Compute	Serverless (Lambda)
Eventing / scheduling	Serverless (EventBridge)
AI/ML services	Managed service consumption (Bedrock, Textract, Rekognition, Comprehend)
File storage	Managed/provisioned (FSx for ONTAP)
Operations model	Hybrid: serverless processing + managed file storage

Lambda concurrency must be bounded by FSx ONTAP S3 AP throughput behavior. Do not treat Lambda concurrency as the only scaling control.

Common Workflow Pattern

Every pattern follows the same proven architecture:

EventBridge Scheduler
       │
       ▼
Step Functions State Machine
       │
       ├── Discovery Lambda (VPC-internal, ONTAP API)
       │        │
       │        ▼
       │   S3 Access Point (list + classify files)
       │
       ├── Processing Map (parallel, Retry + Catch)
       │        │
       │        ▼
       │   [Rekognition | Textract | Comprehend | Bedrock | Athena]
       │
       └── Report Lambda
                │
                ├── Output → S3 AP (FSx ONTAP volume)
                └── SNS Notification

What changes per industry:

File prefixes and extensions (Discovery Lambda configuration)
AI/ML service selection (Rekognition for images, Textract for documents, Bedrock for reasoning)
Domain-specific schemas (ESG metrics, GHS sections, CDR fields)
Review thresholds (60% escalation trigger for safety-critical defects, 80% standard detection, 90% auto-approve threshold)
Compliance requirements (PII filtering for HR, data classification labels, audit trails)

For production deployments, validate how S3 AP-generated output files appear from existing NFS/SMB clients, including ownership, permissions, naming convention, and Snapshot/SnapMirror policy impact. See ONTAP Integration Notes.

Shared Modules: The Productivity Multiplier

The 11 new patterns reuse the same shared/ modules that power the original 17:

Module	Purpose	Used By
`s3ap_helper.py`	S3 Access Point abstraction (alias + ARN)	All 28 UCs
`exceptions.py`	Domain exceptions + error handler decorator	All 28 UCs
`observability.py`	EMF metrics + structured logging	All 28 UCs
`human_review.py`	Confidence-based review decisions	UC22, UC25, UC27
`data_classification.py`	Output data labeling (INTERNAL/CUI/etc.)	UC23, UC24, UC27, UC28
`schemas/events.py`	TypedDict event/response schemas	All 28 UCs

Adding a new industry pattern takes 2-3 hours (not days) because the infrastructure is already solved. A new pattern is considered field-shareable only after DemoMode execution, cfn-lint validation, unit/property tests, success metrics, data classification, and human review thresholds are documented.

Key Design Decisions for New Patterns

1. Safety-Critical Thresholds (UC22)

Railway infrastructure inspection cannot accept false negatives. We use a dual-threshold approach:

STANDARD_THRESHOLD = 80       # General defect detection trigger
SAFETY_CRITICAL_THRESHOLD = 60  # Bridges, signaling, rail joints — lower to reduce false negatives
HUMAN_REVIEW_THRESHOLD = 90    # Auto-approve only above this

Critical design intent: 60% is NOT an auto-approval threshold. It is an escalation trigger — any signal above 60% for safety-critical categories triggers mandatory human review. The system is designed to surface potential defects for expert evaluation, not to automate safety decisions. All detections below 90% confidence require human review regardless of category.

2. PII-First Design (UC27)

Recruiting document triage handles personal data. The pattern enforces:

No PII in logs — structured logging strips personal identifiers
Protected characteristic exclusion — Bedrock prompt explicitly excludes age, gender, ethnicity
Encrypted output — all results written with data classification labels
Audit trail — every scoring decision is logged with justification (not content)

Regulatory notice: UC27 is a document triage and summarization workflow, not an automated hiring decision system. Final hiring decisions must remain with qualified human reviewers. Customers must validate against local labor law, privacy regulations (GDPR, APPI, CCPA), and anti-discrimination requirements before any use in recruitment processes. Output must not include ranking by protected attributes, and explanation fields must cite only job-relevant qualifications.

3. Tri-Modal Inspection (UC25)

Utilities asset inspection combines three data modalities in a single workflow:

Visual (drone images) → Rekognition defect detection
Temporal (SCADA logs) → Athena time-series anomaly detection
Thermal (FLIR images) → Hot-spot classification (≥10°C differential)

The Step Functions workflow processes all three in parallel Map states, then merges results for a unified maintenance priority report.

4. ESG Framework Mapping (UC23)

Sustainability reporting requires mapping extracted metrics to multiple frameworks simultaneously:

GRI (Global Reporting Initiative)
TCFD (Task Force on Climate-related Financial Disclosures)
ISSB (International Sustainability Standards Board)

Bedrock performs the mapping using structured prompts with framework-specific indicator definitions.

Testing: 1,499+ Tests Across 28 Patterns

Each new pattern includes:

Unit tests with moto for AWS service mocking
Property-based tests (Hypothesis) for invariant verification
cfn-lint validation for all CloudFormation templates
ruff linting for Python code quality

Notable property tests:

UC22: severity_level ∈ {critical, major, minor, observation} for all inputs
UC25: SCADA thresholds within physical bounds (voltage ±5%, frequency ±0.5 Hz)
UC27: No protected characteristics appear in any output field
UC28: All GHS mandatory sections validated for completeness

Responsible AI and Human Review

These patterns are reference workflows, not fully automated decision systems. For regulated or safety-critical domains (healthcare, finance, transportation, HR, public sector), customers must define:

Human review thresholds — what confidence level requires expert validation
Appeal/escalation process — how incorrect classifications are corrected
Audit trail requirements — what decisions need immutable logging
Data retention policy — how long intermediate results are kept
Model evaluation criteria — accuracy, hallucination rate, bias testing on domain data
Local regulatory review — jurisdiction-specific compliance (FISC, HIPAA, GDPR, NARA, labor law)

The shared/human_review.py module provides a framework for confidence-based routing, but threshold values and escalation procedures must be defined by domain experts, not by template defaults.

Customers are responsible for validating these workflows against their own policies, risk classification, and regulatory obligations before production use.

Pattern Selection Guide

Customer Situation	Recommended Starting Pattern
FSx ONTAP already used for shared files	UC by industry + DemoMode=false
No FSx ONTAP yet, wants to evaluate workflow	Any UC + DemoMode=true
Document-heavy workload (PDF, contracts, reports)	UC20 / UC23 / UC24 / UC26 / UC27 / UC28
Image-heavy inspection workload	UC19 / UC21 / UC22 / UC25
Logs / time-series / analytics workload	UC18 / UC25 (SCADA)
Safety-critical review required	UC22 / UC25 with human_review module
PII-sensitive workflow	UC27 / UC26 with data_classification module
ESG / sustainability reporting	UC23 with framework mapping
Greenfield object-native workload (no NAS)	Prefer standard S3 + serverless-native architecture

DemoMode to Production Path

Area	DemoMode (evaluation)	Production (FSx ONTAP)
Input source	Regular S3 bucket	FSx ONTAP S3 Access Point
Permissions	S3 IAM only	IAM + S3 AP policy + ONTAP file identity
Network	Public AWS service path	Internet-origin or VPC-origin design decision (NetworkOrigin is immutable after creation)
Data	Sample/synthetic data	Customer-controlled NAS data
Governance	Demo labels only	Data classification + lineage + retention
Cost	~$0.10/execution	+ FSx ONTAP infrastructure (~$194/month base)
Code compatibility	Standard S3 bucket semantics	Validate the FSx ONTAP S3 AP API subset and unsupported S3 bucket features before production
Access point lifecycle	N/A	NetworkOrigin changes require creating a new S3 AP

Cost varies by region, deployment type, SSD capacity, throughput capacity, backups, and data transfer; the figure above is a baseline estimate for Single-AZ / 128 MBps / 1 TB SSD. This cost model is not scale-to-zero storage. Use this pattern when the value of processing existing NAS data in place outweighs the baseline FSx ONTAP infrastructure cost.

Deployment: 30 Minutes to First Result

Every pattern includes a samconfig.toml.example and step-by-step deployment:

# 1. Copy and configure
cp samconfig.toml.example samconfig.toml
# Edit: S3AccessPointAlias, VpcId, SubnetIds, etc.

# 2. Deploy
sam build && sam deploy --guided

# 3. Execute
aws stepfunctions start-execution \
  --state-machine-arn <ARN from outputs>

# 4. Verify
aws stepfunctions describe-execution --execution-arn <ARN>
# Status: SUCCEEDED

For patterns without FSx for ONTAP, DemoMode=true uses a regular S3 bucket — ideal for evaluation without infrastructure commitment.

Benchmark Insight: Small Files Don't Need More Throughput

During Phase 15 deployment verification, we ran benchmarks at 128/256/512 MBps throughput capacity with a 202-byte JSON manifest:

Throughput	P50 @ conc=1	P50 @ conc=25	P50 @ conc=50
256 MBps	56.9 ms	60.3 ms	257.9 ms
512 MBps	59.8 ms	59.9 ms	246.1 ms

Conclusion: For metadata-heavy workloads (JSON manifests, small config files, document headers), throughput capacity increase has zero effect on latency. The bottleneck is connection overhead (TLS + S3 AP routing), not bandwidth. Save costs by staying at 128 MBps for these workloads.

Sizing reference from a specific test environment, not a service limit.

Documentation: 8 Languages × 28 Patterns

Every pattern includes documentation in:
🇯🇵 Japanese (primary) · 🇺🇸 English · 🇰🇷 Korean · 🇨🇳 Chinese (Simplified) · 🇹🇼 Chinese (Traditional) · 🇫🇷 French · 🇩🇪 German · 🇪🇸 Spanish

Each language includes:

README.md — Overview, deployment, success metrics
docs/architecture.md — Mermaid data flow diagram
docs/demo-guide.md — Step-by-step demo with verification checklist

Each UC README includes Success Metrics with Business Outcome, Technical KPI, Quality KPI, Cost KPI, and Go/No-Go criteria. This article summarizes the portfolio; detailed success criteria live with each pattern.

What Changed Since Phase 14

Metric	Phase 14	Phase 15	Delta
Use cases	17	28	+11
Total patterns	24	35	+11
Test count	~800	1,499+	+699
Industries covered	14/22	19/22	+5
Languages	8	8	—
Shared modules	8	11	+3
Documentation files	~400	~700	+300

Who Should Use Each New Pattern?

Recommended Starting Patterns

Start here if...	Pattern	Why
You want document intelligence	UC20 or UC26	Multilingual extraction + property/lease analysis
You want log analytics	UC18	CDR/syslog anomaly detection with baseline
You need PII-safe document triage	UC27	Protected characteristic exclusion built-in
You need inspection workflows	UC22 or UC25	Safety-critical escalation + tri-modal
You want ESG extraction	UC23	Multi-framework mapping (GRI/TCFD/ISSB)

Full Pattern List

If you are...	Start with...	Why
Telecom operator with CDR data	UC18	Anomaly detection across network logs
Ad agency managing creative assets	UC19	Automated brand compliance scoring
Hotel chain with inspection photos	UC20	Facility condition monitoring at scale
Agricultural cooperative	UC21	Crop health + traceability in one workflow
Railway/transit operator	UC22	Safety-critical deterioration detection
ESG reporting team	UC23	Multi-framework metric extraction
Grant-making foundation	UC24	Application processing + outcome matching
Power utility with drone programs	UC25	Tri-modal inspection (visual + SCADA + thermal)
Real estate portfolio manager	UC26	Property analysis + lease extraction
Recruiting team (APAC/EMEA)	UC27	PII-compliant recruiting document triage
Chemical manufacturer	UC28	SDS compliance + lab notebook digitization

What's Next

VPC-internal Lambda benchmark — True VPC path performance (eliminates Internet latency)
FPolicy TCP-level Replay Storm — Real ONTAP event replay (requires ECS rebuild)
Cross-repository integration — Link patterns to fsxn-lakehouse-integrations for analytics pipelines
Glue Data Catalog integration — Schema versioning and data quality checks for output datasets
Community contributions — Pattern template for community-submitted industry use cases

Resolved from Phase 14: FlexCache × S3 AP integration confirmed as not currently supported by AWS — tracked in Field Feedback Log. FC1 Recovery Metrics depend on this feature. Both remain pending AWS feature availability.

Ownership Model

Layer	Recommended Owner
Shared modules (`shared/`)	Platform / DevOps team
UC business logic (`functions/`)	Application / data team
FSx ONTAP and S3 AP infrastructure	Storage / platform team
IAM, data classification, encryption	Security team
Success metrics and Go/No-Go	Business owner
Regulatory compliance mapping	GRC / legal team

Compliance Positioning

These templates do not certify compliance with any specific regulation. They provide implementation hooks for audit logging, retention, classification, and human review that customers can map to their regulatory controls. Each organization must independently validate compliance with applicable regulations (FISC, HIPAA, GDPR, NARA, local labor law, etc.).

NetApp / ONTAP Operational Notes

For production deployments on FSx for ONTAP, review the ONTAP-specific guidance in docs/ontap-integration-notes.md, including:

SVM / volume / protocol scope assumptions
NFS/SMB visibility of S3 AP-generated outputs (file ownership = AP file system identity)
IAM + S3 AP policy + ONTAP file identity behavior, separate from NFS export policy evaluation
Snapshot / SnapMirror / retention impact on output artifacts
Scheduler vs FPolicy trigger mode selection
FlexCache / FlexClone combination patterns per UC
NetApp support diagnostic bundle
OT/manufacturing safety caveat

FlexCache/FlexClone note: UC × FC combination patterns describe adjacent architecture patterns. Validate current AWS/FSx feature support before assuming direct S3 AP access to cached or cloned paths.

Benchmark scope: Results are from Single-AZ, First-generation FSx ONTAP. Validate separately for Multi-AZ or newer generation file systems.

Regulated research workflows (UC7, UC28, FC5): Capture input dataset version, model/prompt version, reviewer action, and output checksum as lineage metadata. See shared/lineage.py v2 fields.

Stats

New patterns: 11 (UC18-UC28)
New Lambda functions: 44 (4 per pattern average)
New tests: 699
New documentation files: ~300 (across 8 languages)
New shared modules: data_classification.py, human_review.py, schemas/events.py
Deployment verified: All 28 UCs achieved SUCCEEDED status in ap-northeast-1
Benchmark runs: 2 additional (256/512 MBps small-file comparison)
Cost: ~$10 total for deployment verification (Lambda + Step Functions + Bedrock Nova Lite)

Try It Today

git clone https://github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns.git
cd FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

# Quick test (no AWS account needed)
make test-quick

# Deploy any pattern with DemoMode (no FSx ONTAP needed)
cd telecom-network-analytics
cp samconfig.toml.example samconfig.toml
sam build && sam deploy --guided

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

Full series: FSx for ONTAP S3 Access Points on DEV.to

Evidence Expansion, Presigned URL Discovery, and Operational Surprises — FSx for ONTAP S3 Access Points, Phase 14

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 07 Jun 2026 02:53:08 +0000

TL;DR

Phase 14 shifts from building patterns to hardening the evidence base. After publishing Phase 13's field-ready reference architecture, I focused on post-publication refinement: Partner/SI delivery assets, benchmark methodology standardization, S3 AP compatibility clarification (Presigned URLs work despite documentation), and an unexpected operational discovery — S3 Access Points become unavailable during FSx throughput capacity changes.

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

Why Phase 14?

Phase 13 delivered the field-ready baseline. Phase 14 answers the question: "Now that the patterns exist, how do we make them easier to evaluate, adopt, and operate?"

The work falls into four categories:

Partner/SI delivery acceleration — one-pager, improved PoC templates, FC1-FC6 conversation starters
Benchmark methodology — standardized run IDs, hypothesis-driven testing, Range GET plans
Compatibility clarification — Presigned URL behavior confirmed with AWS Support
Operational discovery — S3 AP unavailability during throughput capacity changes

1. Partner/SI One-Pager: What / When / How / Where

Partners and SIs told us the existing 7-step delivery checklist was comprehensive but too long for a first conversation. Phase 14 adds a single-page overview that answers four questions:

Section	Content
What	28 UCs + 6 FC patterns, CloudFormation templates, 4-level maturity model
When	Customer has FSx for ONTAP + needs serverless file processing + permission-aware access
How	Identify UC → Deploy template → Measure baseline → Evaluate Go/No-Go
Where	Links to Success Metrics, Governance, Production Readiness, Benchmarks

Available in both Japanese and English.

FC1-FC6 Recommended First Questions

Each FlexCache/FlexClone pattern now has a recommended first conversation question — the question a Partner/SI should ask to determine if the pattern is relevant:

Pattern	First Question
FC1 (Anycast/DR)	"What is your current read latency from remote sites, and what target would justify a caching layer?"
FC2 (Render)	"How many concurrent render jobs share the same source data, and what is the job lifecycle?"
FC3 (RAG)	"Which file shares contain the knowledge base, and do access permissions need to be preserved in RAG results?"
FC4 (CAE)	"What is the typical solver output size and how quickly must results be available for post-processing?"
FC5 (Life Sciences)	"How do you currently share research datasets between teams while maintaining data governance?"
FC6 (Gaming)	"What is your current build pipeline duration and which asset validation steps are bottlenecks?"

2. Presigned URLs: "Not Supported" but Working

⚠️ Production Warning: AWS Support explicitly states that operations marked "Not supported" should NOT be relied upon for production workloads, even when they return success today. The behavior may change without deprecation notice, return inconsistent results across regions, or stop working after service updates. Design alternatives for any workflow that requires presigned URL access to FSx for ONTAP S3 Access Points.

The Discovery

The FSx for ONTAP S3 AP compatibility table lists Presign — Not supported. However, testing showed presigned URLs for GetObject work successfully.

AWS Support Clarification

After raising this with AWS Support, the explanation was clear:

Presigning is client-side only — aws s3 presign computes a SigV4 signature locally. No network request is made.
The presigned URL executes a standard GetObject — signature is in query parameters instead of the Authorization header.
Since GetObject is Supported, presigned URLs cannot be blocked without breaking GetObject itself.
The documentation likely intended to indicate that presigned URL workflows are not officially tested.

Production Guidance

Feature	Status	Guidance
GetObject, PutObject, ListObjectsV2	Supported	Build on freely
Conditional writes (If-None-Match)	Blocked	Returns NotImplemented
Presigned URLs	Not supported (doc)	Works but do not rely on for production

AWS Support has escalated documentation clarification to the FSx for ONTAP service team. The distinction between "Not supported + hard-blocked" (returns error) and "Not supported + may incidentally work" (no guarantees) is being reviewed.

Readers should verify the latest AWS documentation before relying on this behavior, as the status may change.

Alternatives to Presigned URLs

If you need time-limited or delegated file access without relying on unsupported behavior:

API Gateway + Lambda proxy with IAM or JWT authorization
CloudFront signed URLs backed by a controlled Lambda@Edge origin
Temporary STS credentials with scoped IAM permissions (time-limited, per-object or prefix)
Application-level download broker with audit logging and access revocation

For a broader comparison with standard S3 bucket semantics, see the S3 Bucket User Guide.

3. Benchmark Methodology: Hypothesis-Driven Testing

Why 1769 MB Lambda Memory?

Lambda memory directly controls CPU and network bandwidth allocation. At 1769 MB, Lambda receives exactly 1 vCPU equivalent, providing consistent and reproducible network throughput for benchmark measurements. Lower memory settings would introduce variable network bandwidth as a confounding factor.

Benchmark Run ID Convention

Every benchmark run now follows a standardized format:

s3ap-bench-{YYYY-MM-DD}-{seq}

With mandatory fixed conditions:

Region: ap-northeast-1
Lambda memory: 1769 MB (1 vCPU)
Lambda architecture: arm64
FSx Throughput Capacity: [128 / 256 / 512] MBps
Iterations per data point: 50
Statistics: p50, p90, p95, p99, min, max
Concurrent NFS/SMB workload: [None / Light / Production-level]

Hypothesis: Throughput Capacity vs Practical Concurrency Point

Based on 128 MBps observations where I observed concurrency=10 as the practical upper limit in this specific test environment (1 MB objects, single Lambda invocation pattern, no concurrent NFS/SMB workload), I hypothesize:

Practical concurrency point may shift with FSx throughput capacity increase.

FSx Capacity	Predicted Practical Concurrency	Rationale
128 MBps	10 (observed)	Baseline — P99 exceeded 420 ms
256 MBps	~15-25	Sub-linear scaling is plausible due to ONTAP WAFL overhead and TCP connection management
512 MBps	~25-45	Step-function behavior possible if a different bottleneck emerges

Note: Linear scaling (2x capacity = 2x concurrency) is one possible outcome, but sub-linear or step-function behavior is equally plausible. The actual relationship depends on ONTAP data plane queuing, TCP connection overhead, and whether the bottleneck shifts from throughput to IOPS or latency at higher capacities.

Initial verification was blocked by the S3 AP issue described below. Results were published after recovery on 2026-05-25.

Update (2026-05-25): S3 AP recovered. Benchmarks completed. See Section 7 for results and hypothesis verification.

4. Operational Discovery: S3 AP Unavailability During Throughput Changes

What Happened

While preparing to run 256 MBps benchmarks, I changed the FSx throughput capacity from 128 to 256 MBps. After the change completed successfully:

All S3 Access Points on the file system returned ServiceUnavailable
All SVMs were affected (not just one)
Reverting to 128 MBps did not immediately restore S3 AP access
The file system itself remained AVAILABLE throughout

Timeline

Time	Event
T+0	`update-file-system` ThroughputCapacity 128 → 256
T+25 min	Change completed (256 MBps confirmed)
T+25+ min	All S3 APs return ServiceUnavailable
T+40 min	Revert initiated (256 → 128)
T+65 min	Revert completed, S3 APs still unavailable

Impact and Recommendation

This is now tracked as an AWS Support case. Key takeaways:

Plan throughput capacity changes during maintenance windows. S3 AP workloads may be disrupted for an extended period.

Unlike standard S3 buckets, FSx for ONTAP S3 AP availability can be affected by FSx file system operational changes such as throughput capacity updates.

Important context: AWS documentation states that NFS/SMB access typically remains available during throughput capacity changes. The S3 AP disruption I observed appears to be specific to the S3 Access Point data plane — not the file system's NFS/SMB data LIFs. This distinction matters for environments that use both protocols.

For regulated environments (FISC, healthcare, government): Throughput capacity changes must be included in change management procedures. If S3 AP-based workloads have SLA requirements, the change should be approved through the organization's change advisory board with documented rollback procedures.

This finding has been added to the Production Readiness document as a Level 3+ operational consideration.

5. DEV.to Series Cleanup

Phase 14 also cleaned up the article series:

Update Notes added to Phase 1, 9, 10, 12 articles linking to Phase 13
Permission-Aware RAG articles moved to a separate series (was incorrectly mixed into FSx S3AP series)
Series now has 14 articles in "FSx for ONTAP S3 Access Points" (down from 16 after RAG separation)

Blockers and Current Status

Item	Blocker	Resolution
~~256/512 MBps benchmark~~	~~S3 AP ServiceUnavailable~~	✅ Resolved 2026-05-25 — Results below
FC1 Recovery Metrics	FlexCache × S3 AP integration	Pending AWS feature availability
~~Hypothesis verification~~	~~Depends on benchmark~~	✅ Partially confirmed — see results

6. Benchmark Results: 128 / 256 / 512 MBps Concurrency Comparison

S3 AP ServiceUnavailable was resolved on 2026-05-25. We immediately executed the planned benchmark across all three throughput tiers.

Test Environment

Note on methodology divergence: The benchmark methodology (Section 3) defines 1769 MB Lambda + 50 iterations as the standard. The Internet tests below used a macOS client + 10 iterations per concurrency level due to the initial exploratory nature of these measurements. The Lambda egress test (Section 8) follows the 1769 MB / 50 iteration standard. Treat Internet results as directional sizing guidance, not as statistically rigorous benchmarks.

Parameter	Value
Region	ap-northeast-1 (Tokyo)
FSx for ONTAP	Single-AZ, First-generation
S3 AP	NetworkOrigin=Internet
Client	macOS, boto3, Python 3.9 (public Internet)
Object sizes	1 KB, 100 KB, 1 MB
Concurrency	1, 5, 10, 20, 50
Iterations	10 per concurrency level (exploratory)

Key Results: 1 MB GetObject P99

Concurrency	128 MBps	256 MBps	512 MBps
1	76 ms	93 ms	96 ms
5	160 ms	175 ms	308 ms
10	239 ms	236 ms	229 ms
20	981 ms	481 ms	738 ms
50	—	850 ms	4,495 ms

Analysis

P50 (median) is largely independent of throughput capacity — Internet baseline latency (connection + TLS) dominates
P99 (tail latency) shows the difference — 128→256 MBps improved P99 by 51% at concurrency=20
512 MBps shows no improvement over 256 MBps via Internet — client-side bandwidth (~100 Mbps) becomes the bottleneck
Hypothesis partially confirmed: Practical concurrency point does shift with throughput capacity, but the relationship is non-linear and bounded by client bandwidth in Internet-origin tests

Sizing Guidance

Workload	128 MBps	256 MBps	512 MBps
Small files (< 10 KB)	MaxConcurrency=20	50	50
Medium files (100 KB)	10	20	50
Large files (1 MB+)	5	10	20

These are sizing references from a specific test environment, not service limits. A VPC-internal Lambda + VPC-origin S3 AP path is expected to reduce public Internet overhead, but remains untested and must be validated separately. Always validate with your own workload profile.

What This Means for Production

For PoC (128 MBps): Keep Step Functions Map state MaxConcurrency ≤ 5 for 1 MB+ files
For Production (256+ MBps): MaxConcurrency=10-20 is safe for most workloads
For VPC-internal Lambda (untested): Expected to further reduce latency by eliminating public Internet path, but requires VPC-origin S3 AP (not yet measured)
Throughput capacity changes: Plan during maintenance windows (S3 AP disruption risk confirmed)
Small files (< 1 KB): Throughput capacity increase has no effect — bottleneck is connection overhead, not bandwidth. Save costs by staying at 128 MBps for metadata-heavy workloads

7. Lambda Egress Path Benchmark: Reducing Connection Overhead

Terminology clarification: This test uses a VPC-external Lambda (no VpcConfig) accessing an Internet-origin S3 AP. The Lambda egress path goes through AWS-managed networking, which is faster than public Internet but is NOT a VPC-internal path. A true VPC-internal test would require a VPC-origin S3 AP + VPC-internal Lambda — that remains untested.

Path	Status	What it measures
Public Internet client → Internet-origin S3 AP	✅ Measured (Section 7)	End-user/CI baseline
VPC-external Lambda → Internet-origin S3 AP	✅ Measured (this section)	AWS-managed Lambda egress, NOT VPC-private
VPC-internal Lambda → VPC-origin S3 AP	❌ Not yet measured	True private path (requires new AP)

We deployed a benchmark Lambda (1769 MB, ARM64, no VpcConfig) to measure GetObject latency via AWS-managed Lambda egress.

Lambda Egress vs Internet: 1 MB GetObject P50

Concurrency	Internet P50	Lambda P50	Improvement
1	68 ms	62 ms	9%
5	117 ms	61 ms	48%
10	175 ms	73 ms	58%
20	256 ms	122 ms	52%
50	N/A	128 ms	—

Key Findings

P50 dramatically improved at concurrency > 1: Lambda egress eliminates public Internet TCP connection overhead
P99 remains high (~1s): Even from Lambda, concurrency=20 shows P99 of 1,318 ms — this is the S3 AP data plane's internal queuing
concurrency=50 P50 is only 128 ms: Lambda threads are efficient against S3 AP
The bottleneck is the FSx for ONTAP S3 AP data plane, not Lambda network bandwidth

Production Sizing (Lambda)

Workload	Recommended MaxConcurrency	Expected P50	Expected P99
Small files (1 KB)	50	~63 ms	~994 ms
Medium files (100 KB)	20	~79 ms	~1,044 ms
Large files (1 MB)	10	~73 ms	~928 ms

Set Lambda timeout to 30s+ and use Step Functions Retry to handle P99 spikes. These results are from VPC-external Lambda (AWS-managed egress), not true VPC-internal path.

8. SQS Replay Storm Simulation: Zero Message Loss Under Load

Scope clarification: This test validates the downstream SQS ingestion and Lambda consumer drain path under replay-like burst conditions. It does NOT validate ONTAP Persistent Store buffering or FPolicy TCP-level server reconnection replay. Those require a live FPolicy server environment (future work).

We simulated FPolicy server reconnection by injecting 1,000 and 10,000 events directly into SQS, mimicking the burst that occurs when a Persistent Store replays buffered events after server reconnection.

Results

Scenario	Events	Loss Rate	Throughput	Batch P99
5 min downtime	1,000	0%	188 eps	177 ms
30 min downtime	10,000	0%	464 eps	79 ms
Consumer drain	1,000	0%	341 msgs/sec	85 ms

SLO Validation

SLO Metric	Threshold	Observed	Status
Event loss rate	< 0.1%	0%	✅
Injection throughput	> 100 eps	464 eps	✅
Consumer drain rate	> injection rate	341 > 188	✅
Batch latency P99	< 200 ms	79 ms	✅
DLQ messages	0	0	✅

Implications

30-min downtime accumulates ~835K events at 464 eps. With Lambda auto-scaling (10 consumers), drain completes in < 5 minutes.
Persistent Store sizing estimate: Based on simulated event payload size, 10K events ≈ 5 MB. Real ONTAP Persistent Store sizing must be validated with live FPolicy replay.
No backpressure issues: SQS Standard queue handles burst without message loss.

9. Operational Runbook: S3 AP Disruption Response

When S3 AP becomes unavailable (e.g., during throughput capacity changes):

Check S3 AP health: ListObjectsV2 / GetObject against S3 AP alias
Check NFS/SMB separately: mount + read test (may still be functional)
Check FSx file system status: describe-file-systems → Lifecycle
Check CloudWatch alarms: Lambda errors, Step Functions failures
Pause ingestion: Disable EventBridge Schedules for affected UC pipelines
Wait and retry: S3 AP recovery may take 15-60 min after throughput changes
Escalate: If unavailable > 60 min, contact AWS Support with file system ID

See Incident Response Playbook for full procedures.

What's Next (Phase 15 candidates)

Update: Phase 15 expanded the pattern library from 17 to 28 industry-specific use cases. Items 1-2 below remain pending AWS feature availability. Items 3-4 are carried forward in Phase 15's What's Next.

FlexCache × S3 AP integration — pending AWS feature availability (not yet supported)
FC1 Recovery Metrics — route decision latency, cache health detection, failover timing (depends on #1)
Replay Storm with real FPolicy server — TCP-level replay characteristics (requires ECS re-deploy)
VPC-internal Lambda with VPC Origin S3 AP — true VPC-internal path (requires new AP with NetworkOrigin=VPC)

Multi-Account OAM validation completed 2026-05-25 — cross-account CloudWatch Metrics, Logs, and X-Ray Traces confirmed working.

Field feedback tracked: S3 AP disruption during throughput change, presigned URL documentation gap, VPC-origin benchmark gap, and FlexCache × S3 AP feature dependency. Details in docs/ontap-integration-notes.md.

Stats

Files changed: 200+ (documentation, translations, shared modules, templates)
New documents: Partner/SI one-pager (JP/EN/KO/ZH-CN), cost calculator, customization guide, incident response playbook, demo mode guide, comparison alternatives, PoC Go/No-Go template
New shared modules: data_classification.py, human_review.py, schemas/events.py
Benchmark runs: 7 (128/256/512 MBps Internet × 2 file sizes + Lambda egress + SQS replay storm simulation)
Templates fixed: 5 (cfn-lint errors: RecursiveDeleteOption, SNSPublishMessagePolicy, Handler path)
Translations added: 20 files (FC1-FC6 ko/zh-CN + FC1/FC3 full 8-lang)
samconfig.toml.example: 24 patterns
Output JSON samples: 24 patterns
DEV.to articles updated: 6 (4 Update Notes + 2 Series changes)
AWS Support cases: 1 resolved (S3 AP ServiceUnavailable — throughput change related)
Operational discoveries: 1 (throughput change → S3 AP disruption, now resolved)
Cost savings: ~$346/month (v4-test-demo + FPolicy server + VPC Endpoints + EC2 停止)
SQS Replay Storm Simulation: 10,000 events, 0% loss in downstream SQS/consumer path

Who Should Care About Phase 14?

Partners and SIs get a one-pager for first conversations and recommended questions for each FC pattern
Operations teams learn that throughput capacity changes can disrupt S3 AP access
Architects get standardized benchmark methodology with hypothesis-driven testing
Developers get Presigned URL clarification — works but don't depend on it
Standard S3 bucket users learn where FSx for ONTAP S3 AP differs from S3 bucket semantics, especially presigned URLs, availability, and operational dependencies
Serverless-first teams learn where the serverless processing plane ends and FSx for ONTAP operational considerations begin

What You Can Do Today

Phase 14 delivers immediately usable assets:

Use the Partner/SI one-pager for your next customer conversation about FSx for ONTAP + serverless
Check the S3AP Compatibility Notes for the latest Presigned URL and troubleshooting guidance
Plan throughput changes carefully — add S3 AP health checks to your maintenance runbook
Use the Sizing Guidance tables (Sections 7-8) to set MaxConcurrency for your workload
Review the S3 Bucket User Guide before porting existing S3 applications to FSx for ONTAP S3 AP
Review the ONTAP Integration Notes before attaching S3 AP workflows to production SVMs and volumes

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

Full series: FSx for ONTAP S3 Access Points on DEV.to

9 Services, One Architecture: What We Learned Shipping FSx for ONTAP Logs to Every Major Observability Platform

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 31 May 2026 01:42:23 +0000

TL;DR

We built and E2E-verified serverless integrations shipping FSx for ONTAP audit logs to 9 observability platforms — all from the same architecture:

For decision makers: 90% cost reduction vs EC2-based collectors ($66/month → $5-8/month), 9 vendor choices instead of 1, 30-minute deploy instead of hours, zero operational burden. Four vendors offer permanent free tiers covering most FSx for ONTAP deployments (New Relic 100 GB, Grafana Cloud 50 GB, Honeycomb 20M events, Sumo Logic 500 MB/day).

                    ┌─────────────────────────────────────────────┐
                    │         One Architecture, 9 Backends        │
                    ├─────────────────────────────────────────────┤
                    │                                             │
                    │  FSx for ONTAP ──→ S3 Access Point          │
                    │       │                                     │
                    │       ▼                                     │
                    │  EventBridge Scheduler (5 min)              │
                    │       │                                     │
                    │       ▼                                     │
                    │  Lambda (vendor-specific handler)           │
                    │       │                                     │
                    │       ├──→ Datadog (Logs API v2)            │
                    │       ├──→ New Relic (Log API v1)           │
                    │       ├──→ Splunk (HEC)                     │
                    │       ├──→ Grafana Cloud (OTLP Gateway)     │
                    │       ├──→ Elastic (Bulk API)               │
                    │       ├──→ Dynatrace (Log Ingest v2)        │
                    │       ├──→ Sumo Logic (HTTP Source)         │
                    │       ├──→ Honeycomb (Events Batch API)     │
                    │       └──→ OTel Collector (OTLP/HTTP)       │
                    │                                             │
                    └─────────────────────────────────────────────┘

12 articles, 9 vendors, 3 event sources (audit logs, EMS webhooks, FPolicy), all CloudFormation-templated, all tested with real FSx for ONTAP data. This post distills what we learned.

This is Part 13 — the series finale — of Serverless Observability for FSx for ONTAP.

The Architecture That Survived 9 Integrations

After implementing 9 vendor integrations, the core pattern remained unchanged:

def lambda_handler(event, context):
    # 1. Get cached credentials (Secrets Manager + TTL, default 5 min)
    creds = auth.get()

    # 2. List new files since checkpoint (S3 AP + SSM)
    new_keys = list_new_keys(s3_ap_arn, prefix, checkpoint)

    # 3. Read, parse, format, ship per file (vendor-specific)
    #    (Simplified — actual implementation batches events across files
    #     and respects vendor-specific batch size limits)
    for key in new_keys:
        logs = read_and_parse(key)
        payload = format_for_vendor(logs)  # Only this changes per vendor

    # 4. Ship with retry (vendor API)
        ship_to_vendor(payload, creds)

    # 5. Advance checkpoint (only after confirmed delivery)
        update_checkpoint(key)

What changes per vendor: only the formatting and HTTP call (~50-100 lines). Everything else — S3 AP access, checkpoint management, DLQ handling, credential caching, retry logic — is shared.

Cross-Vendor Comparison: The Numbers

API Characteristics

Vendor	Endpoint	Auth Model	Max Batch	Success Code	Firehose
Datadog	Logs API v2	Header (`DD-API-KEY`)	5 MB / 1000 items	200	Yes
New Relic	Log API v1	Header (`Api-Key`)	1 MB	202	Yes
Splunk	HEC	Header (`Splunk <token>`)	No hard limit	200	Yes (built-in)
Grafana	OTLP Gateway	Basic Auth (base64)	~4 MB	200	No
Elastic	Bulk API	Header (`ApiKey <b64>`)	~10 MB	200	No
Dynatrace	Log Ingest v2	Header (`Api-Token`)	1 MB	204	Via ActiveGate
Sumo Logic	HTTP Source	URL-embedded token	1 MB	200	No
Honeycomb	Events Batch	Header (`x-honeycomb-team`)	5 MB (impl: 100/batch)	200	No
OTel Collector	OTLP/HTTP	Configurable	Configurable	200	No

Cost at 10 GB/month

Vendor	Vendor Cost	AWS Infra	Total	Free Tier
Sumo Logic	$0	~$5	~$5	500 MB/day
Honeycomb	$0	~$5	~$5	20M events/month
New Relic	$0	~$5	~$5	100 GB/month
Grafana Cloud	$0	~$5	~$5	50 GB logs/month
Datadog	~$15	~$5	~$20	Logs: 14-day trial only
Dynatrace	~$25	~$5	~$30	14-day trial
Elastic Cloud	~$95	~$5	~$100	14-day trial
Splunk Cloud	~$150+	~$5	~$155+	N/A

AWS infrastructure cost is consistent across all vendors (~$5/month for Lambda + EventBridge + Secrets Manager). The vendor platform cost is the differentiator.

Data Residency

Vendor	Tokyo (JP)	US	EU	Self-Hosted
Sumo Logic	Yes	Yes	Yes	No
Elastic	Yes	Yes	Yes	Yes
Dynatrace	Yes (region-specific)	Yes	Yes	Yes (Managed)
Datadog	No	Yes	Yes	No
New Relic	No (July 2026 planned)	Yes	Yes	No
Grafana Cloud	Dedicated only	Yes	Yes	No (Alloy self-hosted)
Splunk	No	Yes	Yes	Yes
Honeycomb	No	Yes	No	No

Governance note: This table provides technical awareness for vendor selection. Grafana Cloud offers Tokyo region on Dedicated tier (not Free/Pro). Data residency alone does not constitute regulatory compliance. Evaluate your specific requirements (APPI, GDPR, FISC, ISMAP) with your compliance team. See the Retention Policy Matrix for regulation-to-vendor mapping.

Unique Strengths

Vendor	Best For
Datadog	Full-stack APM correlation, broadest feature set
New Relic	Generous free tier (100 GB), NRQL power
Splunk	Existing Splunk shops, SPL expertise, Firehose native
Grafana Cloud	OTLP-native, LogQL, open-source ecosystem
Elastic	Data sovereignty (self-hosted), ECS/SIEM, Kibana
Dynatrace	Davis AI root cause analysis, APM correlation
Sumo Logic	JP region data residency, generous free tier
Honeycomb	High-cardinality analysis (BubbleUp, Heatmaps)
OTel Collector	Multi-backend, vendor portability, redaction

Note on Grafana ecosystem: Grafana Alloy (formerly Grafana Agent) provides a Grafana-native alternative to the OpenTelemetry Collector with the same OTLP compatibility. Grafana Cloud's OTLP Gateway is available on all tiers including Free (US/EU regions only). For Tokyo data residency, Grafana Cloud Dedicated is required.

7 Patterns That Survived All 9 Integrations

1. Polling > Event-Driven (for FSx for ONTAP S3 AP)

FSx for ONTAP S3 Access Points don't support S3 Event Notifications. We evaluated CloudTrail data events as an alternative — however, CloudTrail data events for FSx for ONTAP S3 AP access are not consistently available across all configurations. The 5-minute EventBridge Scheduler poll is simpler, cheaper, and sufficient for audit log use cases where near-real-time (not real-time) delivery is acceptable.

2. Checkpoint-After-Delivery

Never advance the checkpoint before confirming vendor delivery. This single rule prevents data loss across all failure modes:

# CORRECT: checkpoint after confirmed delivery
ship_to_vendor(payload)  # Raises on failure
update_checkpoint(key)   # Only reached on success

# WRONG: checkpoint before delivery
update_checkpoint(key)   # What if ship_to_vendor fails next?
ship_to_vendor(payload)  # Data loss if this fails

3. Credential Caching with Reload-on-401

Every vendor integration uses the same SecretBackedAuth pattern: cache credentials at cold start, reload on TTL expiry or 401/403. This handles credential rotation without Lambda redeployment.

4. Reserved Concurrency = 1

The audit poller must not run concurrently (checkpoint race condition). ReservedConcurrentExecutions: 1 is the simplest guard. For higher throughput, move to DynamoDB-based per-object locking.

5. DLQ for Every Async Path

Every template includes a KMS-encrypted DLQ. In 9 integrations, the DLQ caught: vendor outages, credential expiry, malformed files, and Lambda timeouts. Without it, these failures would be silent data loss.

6. Vendor-Specific Batch Limits Matter

The biggest implementation difference across vendors is batch size handling:

Vendor	Limit	Lambda Behavior
Honeycomb	100 events	Split into chunks of 100
Dynatrace / Sumo Logic	1 MB	Measure payload size, split at boundary
Datadog	5 MB / 1000 items	Dual limit check
Elastic	~10 MB	Rarely hit with audit logs

7. OTLP as the Universal Format

If you're unsure which vendor you'll use long-term, start with OTLP. The OTel Collector integration (Part 5) proved that a single Lambda producing OTLP can feed Datadog, Grafana, and Honeycomb simultaneously — with zero code changes when adding or removing backends.

Beyond multi-backend delivery, the OTel Collector provides:

Enrichment: Resource detection, Kubernetes attributes, custom metadata injection
Sampling: Tail-based sampling for high-volume environments
Redaction: PII field removal/masking before data leaves your account (see PII Redaction Cookbook)
Format conversion: OTLP ↔ vendor-native format translation

Verified version: All OTel Collector configurations in this series were tested with OpenTelemetry Collector Contrib v0.152.0. OTel Collector has frequent releases with potential breaking changes — pin your version in production and test before upgrading.

What We'd Do Differently

Start with OTel Collector for Multi-Vendor Evaluation

If evaluating multiple vendors, deploy the OTel Collector path first. It lets you send the same data to 2-3 vendors simultaneously for comparison, without deploying separate Lambda stacks per vendor.

Define SLOs Before Building

We defined Pipeline SLOs after building all 9 integrations. In hindsight, defining "< 10 min delivery latency" and "< 0.01% data loss" upfront would have guided design decisions earlier (e.g., checkpoint granularity, retry policy).

Data Classification First

Audit logs contain PII (usernames, file paths). We documented this in the Data Classification Guide after implementation. For regulated environments, classify fields before choosing a vendor — it may eliminate options that don't support your data residency requirements.

Production Readiness Framework

After 9 integrations, we formalized a 4-level production readiness model:

Level	What	Go/No-Go to Next
Level 1: Quickstart	Audit poller + DLQ	Logs arrive, checkpoint advances, DLQ empty 24h
Level 2: Operational PoC	+ Dashboard + alerts	SLOs met 7 days, security review done
Level 3: Production	+ DynamoDB ledger + poison-pill	SLOs met 30 days, compliance pack
Level 4: Enterprise	+ OTel Collector + redaction	Multi-backend, PII redaction, DR tested

Most PoCs should target Level 2. Production deployments need Level 3. Enterprise pipelines with compliance requirements need Level 4.

Recommended transition timeline:

Level 1 → Level 2: ~1 week (add dashboards, define SLOs, validate 7-day stability)
Level 2 → Level 3: ~2-4 weeks (deploy DynamoDB ledger, implement poison-pill handling, complete security review)
Level 3 → Level 4: ~1-2 months (deploy OTel Collector, implement PII redaction, test DR failover, complete compliance evidence pack)

Full criteria: Pipeline SLO Definitions

Vendor Selection Decision Tree

Start
  |
  +-- Need JP data residency?
  |   +-- Yes -> Sumo Logic (JP) or Elastic (self-hosted in Tokyo VPC)
  |   +-- No  |
  |           v
  +-- Need self-hosted (air-gapped)?
  |   +-- Yes -> Elastic or Splunk
  |   +-- No  |
  |           v
  +-- Already have an observability platform?
  |   +-- Yes -> Use that vendor (all 9 are supported)
  |   +-- No  |
  |           v
  +-- Budget constraint (free tier needed)?
  |   +-- Yes -> Sumo Logic (500 MB/day) or Honeycomb (20M events) or New Relic (100 GB)
  |   +-- No  |
  |           v
  +-- Need AI-powered root cause analysis?
  |   +-- Yes -> Dynatrace (Davis AI)
  |   +-- No  |
  |           v
  +-- Need high-cardinality analysis?
  |   +-- Yes -> Honeycomb (BubbleUp)
  |   +-- No  |
  |           v
  +-- Need multi-backend / vendor portability?
  |   +-- Yes -> OTel Collector
  |   +-- No  |
  |           v
  +-- Default -> Datadog (broadest) or Grafana (OTLP-native, open ecosystem)

The FSx for ONTAP S3 AP Constraint That Shaped Everything

The single most impactful technical constraint: FSx for ONTAP S3 Access Points do not support S3 Event Notifications.

This one fact drove:

EventBridge Scheduler polling pattern (not event-driven)
SSM Parameter Store checkpointing (track what's been processed)
Reserved concurrency = 1 (prevent checkpoint races)
Safety threshold (stop before Lambda timeout)
MAX_KEYS_PER_RUN (bound processing per invocation)

If FSx for ONTAP S3 APs add event notification support in the future, the architecture could simplify significantly. As of May 2026, this feature is not supported, and the polling pattern is battle-tested across 9 vendors.

Cost Reality: EC2 vs Serverless

The original motivation: replace the EC2-based Splunk pattern (2x EC2 instances) with serverless.

Metric	EC2 Pattern	Serverless Pattern
Monthly AWS cost	~$66	~$5-8
OS patching	Required	None
Scaling	Manual	Automatic
Vendor support	Splunk only	9 vendors
Deploy time	Hours	30 minutes
Recovery from failure	Manual restart	Automatic (DLQ + retry)

90% cost reduction with zero operational burden. The serverless pattern wins on every dimension except one: real-time latency (EC2 syslog can be sub-second; our poller is 5-minute intervals). For audit logs, 5 minutes is acceptable. For real-time needs, use the FPolicy path (< 30 seconds).

What's Next

This series covered the foundation. The project continues with:

Phase 3 (delivered): Multi-account deployment (AWS Organizations + StackSets)
Phase 3 (delivered): DynamoDB object ledger for per-object processing state
Phase 3 (delivered): SQS buffering pattern for backpressure handling
Phase 3 (delivered): Cross-region DR with Active-Passive failover
Phase 3 (delivered): OTel Collector PII redaction cookbook (7 recipes for APPI/GDPR)
Phase 4: Terraform module equivalents
Phase 4: CDK construct library

See the full ROADMAP.

Resources

GitHub: github.com/Yoshiki0705/fsxn-observability-integrations
Pipeline SLO: docs/en/pipeline-slo.md
Data Classification: docs/en/data-classification.md
S3 AP Throughput Benchmark: docs/en/s3ap-throughput-benchmark.md
Vendor Comparison: docs/en/vendor-comparison.md
Partner FAQ: docs/en/partner-faq.md
Workshop Guide: docs/en/workshop-hands-on-half-day.md
Compliance Evidence Pack: docs/en/compliance-evidence-pack.md

Series Navigation

Part 1: Why Your FSx for ONTAP Audit Logs Deserve Better Than EC2
Part 2: Shipping FSx for ONTAP Logs to Datadog — The Serverless Way
Part 3: Event-Driven Ransomware Detection with ONTAP ARP + Datadog
Part 4: FPolicy File Activity Pipeline — ONTAP to Datadog via ECS Fargate
Part 5: Escape Vendor Lock-in: Multi-Backend Log Delivery with OTel Collector for FSx for ONTAP.
Part 6: Direct-to-Grafana: Shipping FSx for ONTAP Logs to Grafana Cloud Loki via OTLP Gateway
Part 7: Ship FSx for ONTAP Audit Logs to New Relic via Serverless Lambda Pipeline
Part 8: EC2 to Serverless: Modernizing FSx for ONTAP Splunk Integration
Part 9: Data Sovereignty: FSx for ONTAP Logs in Your VPC with Elastic
Part 10: High-Cardinality File Access Analysis with Honeycomb + OTel
Part 11: AI-Powered Root Cause: Correlating File Access with APM via Dynatrace
Part 12: FSx for ONTAP Audit Logs with Data Residency in your region with Sumo Logic
Part 13: 9 Vendors, One Architecture (this post)

Thank you for following this series. If you've deployed any of these integrations, I'd love to hear about your experience — drop a comment or open a GitHub issue.

GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

FSx for ONTAP Audit Logs with Data Residency in your region with Sumo Logic

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 31 May 2026 00:37:19 +0000

TL;DR

We built a serverless Lambda pipeline that ships FSx for ONTAP audit logs to Sumo Logic's JP (Tokyo) region deployment. For Japanese enterprises with data residency requirements under APPI (Act on the Protection of Personal Information), this means audit logs never leave Japan.

FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda → Sumo Logic HTTP Source (JP)
                                                                         │
                                                                         ▼
                                                              ┌───────────────────┐
                                                              │ Sumo Logic JP     │
                                                              │ (Tokyo)           │
                                                              │                   │
                                                              │ • 500 MB/day FREE │
                                                              │ • Data stays in   │
                                                              │   Japan           │
                                                              │ • 7-day retention │
                                                              │   (free tier)     │
                                                              └───────────────────┘

Key advantages:

500 MB/day free tier (~15 GB/month) — covers most FSx for ONTAP deployments at zero vendor cost
JP region deployment — data residency in Tokyo
Simplest auth model — URL-embedded token, no header management
30-minute end-to-end — HTTP Source URL is the only credential needed

Verified on Sumo Logic JP region. Logs searchable via _sourceCategory=aws/fsxn/audit.

This is Part 12 of the Serverless Observability for FSx for ONTAP series.

Why Sumo Logic for Japanese Enterprises?

For organizations operating under Japanese data protection regulations, the choice of observability platform often comes down to one question: where does the data physically reside?

Requirement	Sumo Logic JP	Other Options
Data residency in Japan	✅ Tokyo deployment	Varies by vendor
APPI compliance consideration	✅ Data stays in JP	May require cross-border assessment
Free tier for validation	✅ 500 MB/day	Most offer 14-day trials only
No agent installation	✅ HTTP Source (agentless)	Some require collectors

Sumo Logic's JP deployment (service.jp.sumologic.com) processes and stores all data within Japan, making it a straightforward choice for organizations that need to demonstrate data residency compliance.

Compliance note: This integration provides a technical path for data residency. Evaluate your specific regulatory requirements with your compliance team — data residency alone does not constitute full regulatory compliance.

Architecture

┌─────────────────────────────────────────────────────────┐
│ Event Sources                                           │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  EventBridge Scheduler                                  │
│  rate(5 minutes) ──→ Lambda                             │
│                       │ lists new files via             │
│                       │ S3 Access Point                 │
│                       │ (checkpoint in SSM)             │
│                       ▼                                 │
│           Sumo Logic HTTP Source                        │
│           (URL-embedded auth)                           │
│                       │                                 │
│  EMS Webhook          │                                 │
│  ──→ API GW ──→ Lambda ─────────────┤                   │
│     (ems_handler)                   │                   │
│                                     ▼                   │
│  FPolicy                       Sumo Logic               │
│  ──→ ECS Fargate ──→ SQS      (Log Search,              │
│  ──→ Bridge Lambda              Dashboards,             │
│  ──→ EventBridge                Alerts)                 │
│  ──→ Lambda (fpolicy_handler) ──────────────────────────┤
└─────────────────────────────────────────────────────────┘

The Simplest Auth Model

Sumo Logic's HTTP Source embeds the authentication token directly in the URL. No separate auth headers, no API key management, no token rotation complexity:

https://collectors.jp.sumologic.com/receiver/v1/http/<TOKEN>

Lambda just POSTs JSON to this URL. That's it. The metadata (source category, name, host) is sent via X-Sumo-* headers:

headers = {
    "Content-Type": "application/json",
    "X-Sumo-Category": "aws/fsxn/audit",
    "X-Sumo-Name": "fsxn-ontap-audit",
    "X-Sumo-Host": "fsxn-ontap"
}

⚠️ Security: The HTTP Source URL contains the auth token. Store it in Secrets Manager, never log it, and never expose it in environment variables or source control.

Quick Start (30 Minutes)

1. Create Sumo Logic Account (JP Region)

Sign up at service.jp.sumologic.com
Select APAC: Tokyo (JP) as your deployment region
Free tier: 500 MB/day, 7-day retention, full feature access for 30 days

2. Create Hosted Collector + HTTP Source

Go to Manage Data → Collection → Add Collector
Select Hosted Collector (name: fsxn-audit-collector)
Add HTTP Logs & Metrics Source:
- Name: fsxn-ontap-audit
- Source Category: aws/fsxn/audit
- Timestamp Parsing: Auto-detect
Copy the generated HTTP Source URL

3. Store HTTP Source URL

aws secretsmanager create-secret \
  --name "sumo-logic/fsxn-http-source" \
  --secret-string '{"url":"https://collectors.jp.sumologic.com/receiver/v1/http/<YOUR_TOKEN>"}' \
  --region ap-northeast-1

4. Deploy CloudFormation Stack

aws cloudformation deploy \
  --template-file integrations/sumo-logic/template.yaml \
  --stack-name fsxn-sumo-logic-integration \
  --parameter-overrides \
    S3AccessPointArn=arn:aws:s3:ap-northeast-1:123456789012:accesspoint/fsxn-audit-ap \
    SumoLogicHttpSourceSecretArn=arn:aws:secretsmanager:ap-northeast-1:123456789012:secret:sumo-logic/fsxn-http-source-XXXXXX \
    S3BucketName=my-fsxn-audit-bucket \
  --capabilities CAPABILITY_NAMED_IAM \
  --region ap-northeast-1

5. Verify in Sumo Logic

Run this search query:

_sourceCategory=aws/fsxn/audit

Note: First-time indexing on a new JP region account takes ~10 minutes. Subsequent ingestion is near-instant.

Query Examples

Sumo Logic uses a pipe-based query language:

Basic Investigation

-- All failed access attempts with user and path
_sourceCategory=aws/fsxn/audit
| json "Result", "UserName", "ObjectName"
| where Result = "Failure"
| count by UserName, ObjectName
| sort by _count desc

-- Top operations by volume
_sourceCategory=aws/fsxn/audit
| json "Operation"
| count by Operation
| sort by _count desc

-- Access pattern timeline (5-minute buckets)
_sourceCategory=aws/fsxn/audit
| json "Operation", "UserName"
| timeslice 5m
| count by _timeslice, Operation

Security Investigation

-- After-hours access (outside 9am-6pm JST)
_sourceCategory=aws/fsxn/audit
| json "UserName", "Operation", "ObjectName"
| where _messagetime < today() + 9h OR _messagetime > today() + 18h
| count by UserName, Operation

-- Bulk delete detection (potential data exfiltration)
_sourceCategory=aws/fsxn/audit
| json "Operation", "UserName"
| where Operation = "Delete"
| timeslice 5m
| count by _timeslice, UserName
| where _count > 50

-- Sensitive path access
_sourceCategory=aws/fsxn/audit
| json "ObjectName", "UserName", "Result"
| where ObjectName matches "*confidential*" OR ObjectName matches "*restricted*"
| count by UserName, Result

Operational Monitoring

-- Log volume trend (capacity planning)
_sourceCategory=aws/fsxn/audit
| timeslice 1h
| count by _timeslice
| outlier _count

-- SVM activity comparison
_sourceCategory=aws/fsxn/audit
| json "SVMName"
| timeslice 15m
| count by _timeslice, SVMName

Sumo Logic Metadata Headers

Lambda sends structured metadata with each request:

Header	Value	Purpose
`X-Sumo-Category`	`aws/fsxn/audit`	Primary search dimension
`X-Sumo-Name`	`fsxn-ontap-audit`	Source identification
`X-Sumo-Host`	`fsxn-ontap`	Host-level grouping

These headers enable efficient searching without parsing the log body:

-- Search by metadata (fast, no JSON parsing)
_sourceCategory=aws/fsxn/audit _sourceHost=fsxn-ontap

Cost Analysis: The Free Tier Advantage

Sumo Logic's free tier is the most generous for small-to-medium FSx for ONTAP deployments:

Monthly Log Volume	Daily Average	Sumo Logic Tier	Monthly Cost
1 GB	~33 MB/day	Free	$0
5 GB	~167 MB/day	Free	$0
10 GB	~333 MB/day	Free	$0
15 GB	~500 MB/day	Free (at limit)	$0
30 GB	~1 GB/day	Professional	~$108/month

Component	Monthly Cost (10 GB/month)
Lambda (5-min polling)	~$3
EventBridge Scheduler	~$1
Secrets Manager	~$1
Sumo Logic	$0 (within free tier)
Total	~$5

For most FSx for ONTAP deployments generating < 15 GB/month of audit logs, the total cost is just the AWS infrastructure (~$5/month). The observability platform itself is free.

Sumo Logic Deployment Regions

Region	URL	Data Residency
JP (Tokyo)	`service.jp.sumologic.com`	Japan
US1	`service.sumologic.com`	US
US2	`service.us2.sumologic.com`	US
EU (Ireland)	`service.eu.sumologic.com`	EU
AU (Sydney)	`service.au.sumologic.com`	Australia
IN (Mumbai)	`service.in.sumologic.com`	India
CA (Montreal)	`service.ca.sumologic.com`	Canada

Select the deployment matching your data residency requirements at account creation time. Region cannot be changed after account creation.

Gotchas & Lessons Learned

#	Discovery	Impact
1	JP region new accounts have ~10 minute initial indexing lag	First search returns empty; wait and retry
2	Search queries use `_sourceCategory` (underscore prefix)	Common mistake: `sourceCategory` without underscore returns nothing
3	HTTP Source URL contains embedded auth token	Rotate by creating a new HTTP Source, updating the Secrets Manager secret, then deleting the old Source
4	Free tier has 7-day retention only	Sufficient for real-time monitoring; archive to S3 for long-term
5	No built-in Firehose support	Lambda direct delivery only
6	Max 1MB per request (newline-delimited JSON)	Lambda batches accordingly
7	Region is permanent — cannot migrate data between deployments	Choose JP at signup for data residency

Free Tier vs Professional

Feature	Free (500 MB/day)	Professional
Daily ingestion	500 MB	1+ GB (configurable)
Retention	7 days	30-365 days
Users	3	Unlimited
Alerts	✅	✅
Dashboards	✅	✅
API access	✅	✅
Data residency	✅ (region-specific)	✅

Tip: Enable Field Extraction Rules (FER): After first ingestion, create a FER for _sourceCategory=aws/fsxn/audit with "Auto-parse JSON" enabled. This automatically extracts all JSON fields (UserName, Operation, ObjectName, etc.) as searchable metadata — no manual field definition needed. Go to Manage Data → Logs → Field Extraction Rules → Add Rule.

For most PoC and small production deployments, the free tier is sufficient. Upgrade when you need longer retention or higher volume.

Data Residency & Classification

Sumo Logic JP deployment keeps all data in Japan, but audit logs still contain PII fields:

Field	Classification	Sumo Logic Handling
`UserName`	PII	Use RBAC to restrict search access; consider field extraction rules for masking
`ObjectName`	Sensitive	Path may reveal business context; restrict dashboard sharing
`ClientIP`	Internal	Generally acceptable

For APPI compliance considerations:

Data stays in JP region (no cross-border transfer for log data)
Configure retention policies matching your regulatory requirements (7 days free tier vs 30-365 days paid)
Implement Sumo Logic RBAC to restrict PII field access by role

See the Data Classification Guide for full field classification and regulatory mapping.

Production Readiness

This integration follows the project's Production Readiness Levels:

Level	What You Get	Go/No-Go to Next
Level 1 (this Quick Start)	Audit poller + DLQ	Logs arrive, checkpoint advances, DLQ empty 24h
Level 2	+ Sumo dashboards + alerts	SLOs met 7 days, security review done
Level 3	+ DynamoDB ledger + poison-pill	SLOs met 30 days, compliance pack
Level 4	+ OTel Collector + redaction	Multi-backend, PII redaction, DR tested

Full criteria: Pipeline SLO Definitions | DLQ Replay Runbook

Enterprise scale: For multi-account deployments across your Organization, see the StackSets deployment guide. For compliance evidence collection (ISMAP, FISC, SOC 2), see the Compliance Evidence Pack. For regulation-to-vendor retention mapping, see the Retention Policy Matrix.

CloudFormation Templates

Template	Purpose	Key Parameters
`template.yaml`	FSx audit log poller	S3AccessPointArn, SumoLogicHttpSourceSecretArn
`template-ems.yaml`	EMS webhook handler	SumoLogicHttpSourceSecretArn
`template-fpolicy.yaml`	FPolicy EventBridge handler	SumoLogicHttpSourceSecretArn, EventBusName

Resources

GitHub: integrations/sumo-logic/
Sumo Logic JP: service.jp.sumologic.com
HTTP Source Docs: Configure HTTP Source
Search Query Language: Search Operators
Series GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

Series Navigation

Part 1: Why Your FSx for ONTAP Logs Deserve Better
Part 2: Shipping FSx for ONTAP Logs to Datadog — The Serverless Way
Part 3: Event-Driven Ransomware Detection with ONTAP ARP + Datadog
Part 4: FPolicy File Activity Pipeline — ONTAP to Datadog via ECS Fargate
Part 5: Escape Vendor Lock-in: Multi-Backend Log Delivery with OTel Collector for FSx for ONTAP.
Part 6: Direct-to-Grafana: Shipping FSx for ONTAP Logs to Grafana Cloud Loki via OTLP Gateway
Part 7: Ship FSx for ONTAP Audit Logs to New Relic via Serverless Lambda Pipeline
Part 8: EC2 to Serverless: Modernizing FSx for ONTAP Splunk Integration
Part 9: Data Sovereignty: FSx for ONTAP Logs in Your VPC with Elastic
Part 10: High-Cardinality File Access Analysis with Honeycomb + OTel
Part 11: AI-Powered Root Cause: Correlating File Access with APM via Dynatrace
Part 12: FSx for ONTAP Audit Logs with Data Residency in your region with Sumo Logic (this post)
Part 13: 9 Vendors, One Architecture

Questions about the Sumo Logic JP integration or data residency? Drop a comment below.

GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

Requirement	Recommended path
Governed metadata analytics only	UC Foreign Catalog (if validated) or sync metadata to UC Delta
Raw file processing in Databricks	DataSync → S3 → UC External Volume
Zero-copy raw file access from Databricks	Not supported in validated paths (NFS mount works but without UC governance)
Business discovery / BI	Sync redacted metadata to UC Delta table

DEV Community: Yoshiki Fujiwara(藤原 善基)@AWS Community Builder

42 Patterns, Category Architecture, and HA LifeKeeper Monitoring — FSx for ONTAP S3 Access Points, Phase 18

TL;DR

Why Restructure?

The Move: git mv with History Preservation

HA LifeKeeper Monitoring — New Pattern

Key Design Decisions

Example Operational Metrics

Deploy (DemoMode)

Category Architecture Diagrams

Project Infrastructure Improvements

pyproject.toml (PEP 621)

Dependency Pinning

.cfnlintrc

Additional Tooling

CI/CD Changes

Multi-Perspective Review

What This Means for You

What's Next

Getting Started

Amazon Quick Agentic Workspace Powered by FSx for ONTAP S3 Access Points — Phase 17

TL;DR

Amazon Quick × S3 AP Data Mapping

Architecture

Hybrid RAG Flow (generate_brief_with_web)

Security Design

Authentication + Per-Action Authorization

Enforced Human-in-the-Loop (HITL)

Additional Controls

Web-Augmented Brief Generation (opt-in)

The Problem

The Solution

Usage

Response

Design Properties

Activation

Verification Findings

Lake Formation + Athena

Amazon Quick × FSx for ONTAP S3 AP Integration Boundary

Glue Tables

Data Classification

Cost

Getting Started

Governance Note

Amazon Bedrock Knowledge Bases + FSx for ONTAP S3 Access Points: Self-Service AI Curation via Windows Drag & Drop — Phase 16

TL;DR

Why This Pattern?

Architecture

Hybrid RAG Flow (opt-in)

Scenario B: Scheduled Automation (Step Functions)

Scenario C: FPolicy Event-Driven Real-Time Sync

Lost-Update Window (Critical)

Namespace Pitfall

Hybrid RAG: Internal KB + Web Search (opt-in)

How It Works

Key Design Decisions

Deployment

Example Response

Verification Highlights

Windows-Identity S3 Access Point with Dedicated AD

Deletion Lifecycle

Performance Considerations

Access Control — Three Layers

Vector Store: Why S3 Vectors

Data Classification

Cost

Getting Started

Governance Note

From Logs to Detection: Building a File Security Pipeline in Datadog for FSx for ONTAP

TL;DR

Why Post-Ingestion Processing Matters

Architecture

The Log Pipeline

Why a Pipeline?

Creating the Pipeline via API

EventID Mapping Table

Security Monitors

Monitor 1: Mass File Deletion

Monitor 2: Abnormal Access Volume

Monitor 3: Access Failure Spike

DEV Community: Yoshiki Fujiwara(藤原善基)@AWS Community Builder