DEV Community: Yoshiki Fujiwara(藤原善基)@AWS Community Builder

Governance & Cross-Platform Access: Lake Formation, PII Anonymization, and Multi-Engine Reality for S3 Tables

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Mon, 08 Jun 2026 16:49:37 +0000

Previously...

In Part 1, we built the metadata catalog. In Part 2, we added AI classification and vector search. Now we need to answer the hard questions:

Who can see what? (governance)
What about PII? (anonymization)
Can Databricks/Snowflake access this? (cross-platform)

Lake Formation: Governance on Unstructured Data

The Problem

Unstructured data on NAS storage may be well protected at the file-system layer, but it is often not consistently classified, searchable, or governed from analytics and AI workflows:

No unified classification → you may not know what's sensitive across the entire corpus
File-system permissions exist, but analytics/AI tools can't leverage them for discovery
Audit trails may exist at the file-system layer, but they are often not unified with analytics and AI query activity

The Solution

With metadata in S3 Tables (Iceberg), Lake Formation provides:

┌───────────────────────────────────────────────────┐
│  Lake Formation                                   │
│                                                   │
│  Table-level:  SELECT, DESCRIBE                   │
│  Column exposure: controlled via Athena Views     │
│                   (hide embedding_vector, paths)  │
│  Row filtering: WHERE sensitivity_level = 'public'│
│  Audit:        CloudTrail logs metadata queries   │
└───────────────────────────────────────────────────┘

Verified: Access Control in Action

Step 1: Authorized user queries metadata
  → ✅ SUCCEEDED (3 rows returned)

Step 2: Revoke SELECT permission
  → 🔒 BLOCKED: "Column 'file_name' cannot be resolved
     or requester is not authorized"

Step 3: Restore permission
  → ✅ SUCCEEDED (access restored)

Step 4: CloudTrail audit
  → All queries logged with user identity and timestamp

Every query against the metadata table is governed and audited. This gives you 100% metadata query governance coverage in this PoC. Raw file access remains governed separately by FSx for ONTAP file-system permissions, S3 Access Point policies, and application-specific access paths.

Lake Formation Governance Status

Capability	Status	Notes
Table-level SELECT / DESCRIBE	✅ Verified	Grant/revoke works correctly
Athena query governance	✅ Verified	Unauthorized access blocked
CloudTrail audit logging	✅ Verified	All queries logged with user identity
Column-level exclusion (ColumnWildcard)	⚠️ Failed	On tested S3 Tables federated catalog path
Row-level filtering / LF-Tags	📋 Design pattern	Taxonomy defined, needs validation
Column exposure via Athena Views	✅ Workaround	Recommended alternative to column-level grants

Observed Limitation: Column-Level Grants on This S3 Tables Federated Catalog Path

In this PoC, table-level Lake Formation SELECT grants worked as expected. However, column exclusion grants using ColumnWildcard with ExcludedColumnNames returned InvalidInputException: Permissions modification is invalid against the s3tablescatalog/... federated catalog path we tested.

AWS documentation describes table, column, and row-level permissions for S3 Tables integrated with Lake Formation. Therefore, treat this as an observed limitation in our specific validation path (CLI command, region, catalog ID, engine version), not a confirmed general product limitation. The exact error and test conditions are recorded in the verification evidence.

Workaround: Create Athena Views that expose only permitted columns:

-- View for general users (no embeddings, no PII paths)
CREATE VIEW metadata.public_files AS
SELECT file_id, file_name, file_type, classification, confidence_score
FROM "s3tablescatalog/fsxn-metadata-catalog"."metadata"."unstructured_files"
WHERE is_deleted = false AND sensitivity_level = 'public';

-- Apply Lake Formation on the view
-- Users query the view, not the base table

Governance model choice: For simple use cases, table/column-level permissions suffice. For dynamic, attribute-based access (e.g., "only files classified as 'public'"), use LF-Tags. For enterprise SSO integration, combine with IAM Identity Center. For enterprise governance, map sensitivity_level, path_classification, tenant_id, and pii_status to LF-Tags. See governance/lf-tag-taxonomy.yaml.

Untested alternative: Registering the S3 Tables table in a standard (non-federated) Glue Catalog may enable column-level permissions. This requires manual Iceberg metadata location configuration and has not been verified.

PII Detection: English + Japanese

The Challenge

Amazon Comprehend's detect_pii_entities API supports only English and Spanish. For Japanese PII (names, addresses, My Number), we need a different approach.

Dual-Engine Architecture

Language	Engine	Detectable PII	Latency	Cost
English	Amazon Comprehend	NAME, EMAIL, PHONE, ADDRESS, SSN, CREDIT_CARD, DATE_TIME	~200ms	$0.0001/100 chars
Japanese	Bedrock Claude	氏名, メール, 電話, 住所, マイナンバー, クレジットカード, 生年月日	~2-5s	~$0.003/request

Data privacy note: When using Bedrock Claude for PII detection, document text is sent to the Bedrock API. Per AWS's data privacy policy, Bedrock does not store or use your inputs/outputs to train models. For highly sensitive workloads, consider VPC endpoints and AWS PrivateLink for Bedrock access.

Japanese PII Detection (Verified)

# Bedrock Claude detects Japanese PII via prompt
response = bedrock.invoke_model(
    modelId="anthropic.claude-3-haiku-20240307-v1:0",
    body=json.dumps({
        "messages": [{"role": "user", "content":
            f"Detect all PII in this text. Return JSON array: "
            f'[{{"type":"...","value":"...","begin":N,"end":N}}]\n\n'
            f"Text:\n{japanese_text}"}]
    })
)

Results on a controlled synthetic sample (not real personal data):

PII Type	Detected Value
NAME	山田太郎
EMAIL	taro.yamada@example.co.jp
PHONE	090-1234-5678
ADDRESS	〒150-0002 東京都渋谷区渋谷1-2-3
MY_NUMBER	1234 5678 9012
CREDIT_CARD	4111-1111-1111-1111
DATE_OF_BIRTH	1985年3月15日

Anonymization Pipeline

Original document
       │
       ▼
PII Detection (Comprehend or Bedrock)
       │
       ├─ No PII → has_pii = false (no action needed)
       │
       └─ PII found → has_pii = true
                          │
                          ▼
              Redaction: all PII → [REDACTED]
                          │
                          ▼
              Store anonymized version
              anonymization_status = "completed"

Before:

Name: Taro Yamada
Email: taro.yamada@example.com
Phone: 090-1234-5678
SSN: 123-45-6789

After:

Name: [REDACTED]
Email: [REDACTED]
Phone: [REDACTED]
SSN: [REDACTED]

Data Clean Room Pattern

┌─────────────────────────────────────────┐
│  Restricted Table (full metadata)       │
│  • has_pii, anonymized_path, raw paths  │
│  • Access: Security team only           │
│  • Lake Formation: strict SELECT grant  │
└─────────────────────────────────────────┘

┌─────────────────────────────────────────┐
│  Public Table (anonymized metadata)     │
│  • classification, summary (redacted)   │
│  • No PII, no raw file paths            │
│  • Access: All analysts                 │
│  • Lake Formation: broad SELECT grant   │
└─────────────────────────────────────────┘

Encryption and Data Residency

At rest: S3 Tables uses SSE-S3 encryption by default. All metadata is encrypted.
In transit: All API calls use TLS 1.2+.
Data residency: Both metadata (S3 Tables) and raw files (FSx for ONTAP) remain in the same AWS region. No cross-border data transfer occurs in the default architecture.

For detailed data sovereignty analysis, see the Architecture Document — Data Sovereignty section.

Audit Log Retention

CloudTrail: Default 90-day event history. For long-term retention, create a Trail delivering to S3 (recommended: 1+ year for regulated industries)
Lake Formation: Data access audit logs are recorded via CloudTrail
OpenSearch: Access logs can be delivered to CloudWatch Logs
Analysis: Use CloudTrail Lake (SQL queries) or Athena + S3 (cost-efficient) for audit analysis

For detailed operational monitoring setup, see the Operational Monitoring section in the architecture document.

Path Sensitivity Model

File paths can reveal sensitive context even when file contents are not exposed (e.g., /hr/layoffs/2026/ or /legal/mna/target-company/).

Recommended controls:

Store raw_path only in the restricted metadata table
Expose hashed_path or anonymized_path to general users
Use path_classification: public / internal / restricted / confidential
Apply Lake Formation grants to curated views, not the base table

Raw Data Access Boundary

This architecture governs metadata access through S3 Tables and Lake Formation. It does not automatically replace:

ONTAP/NFS/SMB file-system permissions
S3 Access Point resource policies
IAM permissions for raw file reads
Application-level authorization
Downstream use of presigned URLs or copied files

Treat metadata governance and raw data governance as two linked but separate control planes. Both must be configured for end-to-end security.

S3 Access Point Identity Boundary

Each FSx for ONTAP S3 Access Point has an associated file-system identity (OntapFileSystemIdentity — UNIX UID/GID or Windows domain user). All file access through that AP is authorized as that identity.

For each access point, document:

IAM principals allowed to use the access point
Access point policy (allowed S3 actions)
Associated UNIX or Windows file-system identity
Allowed volume / prefix scope
Whether the identity can access files beyond what metadata governance intends
Audit evidence location

If the AI enrichment access point uses a broad UNIX identity (e.g., root or a service account with wide read access), metadata-level Lake Formation controls do not prevent raw file reads through that AP. Scope the AP identity to minimum required access.

See security/s3-access-point-identity-matrix.yaml for the template.

Permission Identity Strategy

For multiprotocol environments (NFS + SMB + S3 AP):

Record discovery_protocol: nfs / smb / s3ap
Record access_point_identity_type: unix / windows
Record effective_reader_identity
Record permission_source: nfs_mode / ntfs_acl / mixed
Do not assume metadata visibility implies raw file readability

Retention and Deletion Semantics

This PoC uses metadata records to represent file discovery and enrichment state. For regulated workloads, define:

Metadata retention period (how long to keep catalog records)
Raw file retention period (governed by storage policy, not this catalog)
Anonymized metadata retention period
Deletion request workflow (who can request, who approves, how it's executed)
Snapshot expiration impact on deletion (Iceberg time travel may expose deleted metadata until snapshots expire)
Audit evidence retention (keep deletion evidence longer than the data itself)

Important: Iceberg time travel is useful for recovery, but it means deleted metadata may still be queryable during the snapshot retention window. Align snapshot expiration with your data deletion SLA.

Snowflake-side retention: If redacted metadata is synced into Snowflake-managed tables, define Snowflake-side retention, Time Travel (default 1 day, up to 90 days), and Fail-safe (7 days, non-configurable) separately from Iceberg snapshot retention. Deletion from the Snowflake copy does not delete from the Iceberg source, and vice versa.

Approval Evidence Template (for Regulated Industries)

For organizations requiring formal access approval documentation:

Approval ID: <unique-id>
Data owner: <name/group>
Security owner: <name/group>
Platform owner: <name/group>
Allowed metadata columns: <columns>
Allowed raw file prefixes: <prefixes>
Allowed operations: metadata query only / raw file read / anonymized export
Review date: <date>
Expiration date: <date>
Evidence location: verification-evidence/<path>

Regulated Workload Readiness

For public sector, healthcare, financial services, and other regulated industries, validate the following before production deployment:

Area	Requirement	Status in this PoC
Data residency	Metadata and raw files in same AWS Region	✅ Single region (ap-northeast-1)
Encryption at rest	S3 Tables: SSE-S3; FSx: at-rest encryption	✅ Default encryption
Encryption in transit	TLS 1.2+ for all API calls	✅ AWS default
Raw data access boundary	File reads governed by S3 AP policy + ONTAP permissions	✅ Documented
Metadata access boundary	Lake Formation table-level + CloudTrail audit	✅ Verified
AI processing data flow	Content sent to Bedrock API, not stored by provider	✅ Per AWS data protection policy
PII detection limitations	English (Comprehend) + Japanese (Claude) only	⚠️ Other languages not covered
Human review workflow	Low-confidence queue defined	✅ Design documented
Audit log retention	CloudTrail 90-day default; configure Trail for longer	⚠️ Requires Trail setup
Deletion SLA	Define separately for metadata, raw files, and snapshots	⚠️ Requires policy definition
Legal/compliance sign-off	Not in scope for this PoC	❌ Required before production

AI governance note: AI enrichment in this pattern is assistive metadata generation. It does not constitute authoritative regulatory classification. Final classification decisions, data handling approvals, and compliance certifications must be confirmed by data owners, security teams, legal counsel, and compliance officers.

Cross-Platform Access: The Current Reality

Fully Verified ✅

Platform	Access Method	Status
Athena	Direct query via Glue federated catalog	✅ Fully verified
Lambda/Python	PyIceberg SDK	✅ Fully verified
EMR Spark	Glue Iceberg REST (EMR 7.13.0+)	✅ Fully verified (SELECT, COUNT, time travel)
Snowflake	Glue Iceberg REST + VENDED_CREDENTIALS	✅ Fully verified (CREATE TABLE, SELECT, COUNT, DESCRIBE, AUTO_REFRESH)
Snowflake	External Stage (FSx S3 AP) + TO_FILE + Cortex AI	✅ Fully verified

Expected / Requires Validation ⚠️

Platform	Access Method	Status
EMR Trino	Glue Iceberg REST (EMR 7.13.0+)	⚠️ Expected (same EMR SigV4 handling as Spark)
Redshift Spectrum	Same as Athena (Glue catalog)	⚠️ Expected, not fully validated

What Doesn't Work (Yet) ⚠️

Platform	Tested method	Result	Tested	Status
Databricks SQL Warehouse	`CREATE CONNECTION TYPE iceberg_rest` to S3 Tables REST	`CONNECTION_TYPE_NOT_SUPPORTED`	2026-05-31	Observed limitation in this path
Databricks Spark cluster	Iceberg REST + SigV4 via spark.conf.set / cluster config	`NO_SUCH_CATALOG_EXCEPTION` (UC blocks external catalog registration)	2026-06-01	Confirmed: UC Foreign Catalog required
Databricks Delta Sharing	Delta Sharing server accessing S3 AP-backed storage	Sharing server uses same UC storage credentials; cannot bypass session policy	2026-06-01	Confirmed limitation (not a workaround for S3 AP)
Databricks NFS → UC Volume	NFS mount path as UC External Volume	Cloud storage URIs only (s3://, abfss://, gs://); NFS/FUSE paths not supported	2026-06-01	Confirmed limitation; internal feature request exists
Snowflake	External Iceberg Table with S3 Tables direct REST endpoint	Not a supported catalog type (use Glue REST instead)	2026-05-31	Use Glue REST + VENDED_CREDENTIALS (✅ verified)
Snowflake	CATALOG INTEGRATION with default ACCESS_DELEGATION_MODE	Defaults to EXTERNAL_VOLUME_CREDENTIALS which triggers ListObjectsV2 (rejected by S3 Tables)	2026-06-02	✅ Resolved: set explicit `ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS`
Snowflake	Lake Formation column-level via VENDED_CREDENTIALS	AllowFullTableExternalDataAccess=false blocks all VENDED_CREDENTIALS access	2026-06-08	Use Snowflake Horizon (Row Access Policy / Dynamic Masking) for column governance
Snowflake Open Catalog	Polaris as Iceberg catalog	Not tested	TBD	Strategic alternative

Databricks: Three Integration Paths to Validate

Update note (2026-06-09): We revalidated the S3 Tables path after Databricks announced GA for Foreign Iceberg and credential vending (May 28, 2026). Glue Connection creation and credential configuration succeeded, but Unity Catalog External Location validation still failed because S3 Tables internal buckets reject standard S3 API validation (HeadBucket/ListBucket). The S3 Tables path remains blocked in this tested Databricks UC configuration. A new Databricks support case has been submitted.

For Databricks business users: The value is not only table access. The value is turning previously invisible NAS files into governed metadata assets that can be searched, explained, lineage-tracked, and consumed from Databricks SQL, AI/BI, and dashboards.

In this PoC, CREATE CONNECTION TYPE iceberg_rest to the S3 Tables REST endpoint returned CONNECTION_TYPE_NOT_SUPPORTED on Databricks SQL Warehouse (tested 2026-05-31). This does not mean Databricks lacks Iceberg REST support — Databricks provides Unity Catalog Iceberg REST endpoints and Foreign Iceberg capabilities that evolve rapidly.

Confirmed Limitations (2026-06-01)

Path	Result	Confirmed by
Spark cluster + Iceberg REST (spark.conf.set / cluster config)	❌ UC blocks external catalog registration	Databricks support + our testing
Delta Sharing via S3 Access Point	❌ Sharing server uses same UC storage credentials	Databricks support
NFS mount path as UC External Volume	❌ Cloud storage URIs only (s3://, abfss://, gs://)	Databricks support
DataSync → S3 → UC External Delta Table → Delta Sharing	✅ Works (Delta format required)	Databricks support

Delta Sharing note: Delta Sharing is not a workaround for the FSx S3 Access Point session policy limitation in our tested path. The sharing server uses the same UC storage credentials and cannot bypass the session policy that blocks S3 AP ARNs. Note that Databricks has announced first-class Iceberg format support in Delta Sharing (Jan 2026), enabling providers to share Iceberg tables via the Iceberg REST Catalog API. This broader capability is not contradicted by our finding — our limitation is specific to S3 AP-backed storage access through UC credentials, not Delta Sharing's format support in general.

NFS Volume note: UC External Volumes require cloud storage URIs. An internal feature request (AHA) exists for EFS/NFS access via UC. Until this is implemented, DataSync → S3 → UC External Location remains the only supported path.

📢 Databricks users: If S3 Tables access from Databricks is important for your workflow, the UC Foreign Catalog for S3 Tables feature is being tracked internally by Databricks (request DB-I-15824). Contact your Databricks account team to express interest and increase prioritization. Snowflake achieved full S3 Tables access via VENDED_CREDENTIALS in June 2026 — the same architectural pattern should be feasible for UC.

Immediate workaround for Databricks: Use DataSync → S3 → UC External Table to sync metadata into a standard S3 location accessible by Unity Catalog. This is not zero-copy for the synced metadata, but raw files remain on FSx for ONTAP.

Path 1: Spark cluster + Iceberg REST (SigV4)

Best for technical validation and batch processing. Two endpoint options:

Tested 2026-06-01: On Databricks with Unity Catalog enabled, external Iceberg catalogs cannot be registered via spark.conf.set or cluster Spark config. Unity Catalog controls catalog registration exclusively. Both Serverless (CONFIG_NOT_AVAILABLE) and All-Purpose clusters (NO_SUCH_CATALOG_EXCEPTION) fail. Unity Catalog Foreign Catalog (Path 2) is the required approach.

# Path 1a: Direct S3 Tables REST endpoint (used in this PoC)
spark.sql.catalog.s3tables.uri=https://s3tables.ap-northeast-1.amazonaws.com/iceberg
spark.sql.catalog.s3tables.rest.signing-name=s3tables

# Path 1b: AWS Glue Iceberg REST endpoint (recommended for production + Lake Formation)
spark.sql.catalog.s3tables.uri=https://glue.ap-northeast-1.amazonaws.com/iceberg
spark.sql.catalog.s3tables.rest.signing-name=glue

Common config for both:

spark.sql.catalog.s3tables=org.apache.iceberg.spark.SparkCatalog
spark.sql.catalog.s3tables.catalog-impl=org.apache.iceberg.rest.RESTCatalog
spark.sql.catalog.s3tables.warehouse=arn:aws:s3tables:ap-northeast-1:<ACCOUNT>:bucket/fsxn-metadata-catalog
spark.sql.catalog.s3tables.rest.sigv4-enabled=true
spark.sql.catalog.s3tables.rest.signing-region=ap-northeast-1

Path 2: Unity Catalog Foreign Iceberg

Register external Iceberg tables into Unity Catalog if supported for the target catalog/storage path. Best for Databricks governance, lineage, and discovery. Verify refresh semantics and read/write limitations. Retested for S3 Tables on 2026-06-09: Glue Connection and credentials succeeded, but UC External Location validation failed because S3 Tables internal buckets reject standard S3 API validation. This path remains blocked for the tested S3 Tables configuration.

Documentation/version note: Databricks Iceberg capabilities are evolving rapidly. Earlier documentation and our initial validation showed limitations around Foreign Iceberg credential vending and automatic refresh behavior. After the May 2026 GA announcement, we revalidated the S3 Tables path on 2026-06-09. Credential configuration progressed further, but the tested path still failed at UC External Location validation against S3 Tables internal storage. Additionally, Databricks supports catalog federation with AWS Glue (Hive Metastore type), which can expose Glue-registered tables in UC. Whether a future Iceberg REST catalog federation path could bypass the S3 Tables internal bucket constraint is an open question.

Refresh semantics: If UC Foreign Iceberg works for S3 Tables via Glue REST, define refresh semantics explicitly. Our metadata catalog is append-only (new records added on file events). Analysts should know whether Databricks reads the latest Iceberg snapshot automatically or only after REFRESH FOREIGN TABLE. Without auto-refresh, Athena and Databricks may show temporarily different results until the next refresh cycle. Plan for a scheduled refresh job or event-driven trigger.

AWS reference for this path: AWS has published guidance on accessing S3 Iceberg tables from Databricks using the Glue Iceberg REST Catalog. This validates the architectural direction of B-4/B-5, though S3 Tables-specific compatibility requires separate validation.

Path 3: AWS Glue Catalog Federation with Databricks

AWS Glue can federate metadata from Databricks Unity Catalog for Iceberg tables. This is the reverse direction but useful for cross-platform governance patterns.

Federation Directionality

Pattern	Direction	Primary governance	Best for
UC Foreign Catalog / Catalog Federation to Glue	Databricks reads AWS-managed metadata	Unity Catalog	Databricks users querying AWS Iceberg (S3 Tables)
AWS Glue federation to UC	AWS reads Databricks-managed metadata	Lake Formation / Glue	Athena/EMR/Redshift reading UC Iceberg/UniForm

AWS reference: AWS has published guidance on accessing S3 Iceberg tables from Databricks using AWS Glue Iceberg REST Catalog, and on federating Databricks Unity Catalog data into AWS Glue Data Catalog. Both directions are documented.

Why Iceberg here (not Delta Lake)? This architecture uses Iceberg because S3 Tables is Iceberg-native, and the Iceberg REST endpoint enables multi-engine access (Athena, EMR, Snowflake). For Databricks-only environments, Delta Lake on S3 remains the natural choice. This pattern targets multi-platform scenarios.

Databricks UC Audit Logging for External Engines (Confirmed 2026-06-01)

External engine access via the UC Iceberg REST Catalog endpoint is fully auditable:

Audit aspect	Confirmed behavior
Metadata requests (listNamespaces, listTables, loadTable)	✅ Logged in `system.access.audit` under `uniformIcebergRestCatalog`
Vended credential issuance	✅ Logged as `loadTableCredentials` / `generateTemporaryTableCredential`
Audit fields	user_identity, source_ip_address, user_agent, event_time, action_name, request_params
Distinguish external vs internal	✅ `service_name = 'uniformIcebergRestCatalog'` (external) vs `'unityCatalog'` (internal)

Note: Databricks audit logs record credential issuance, not individual S3 file reads after credentials are vended. Complement with AWS CloudTrail + S3 access logging for file-level audit.

Databricks integration documentation:

For UC Foreign Catalog validation steps, see databricks/uc-foreign-iceberg-validation.md

For coexistence planning, see databricks/coexistence-roadmap.md

For audit investigation, see databricks/audit-correlation-guide.md

Snowflake: S3 Tables via Glue REST + VENDED_CREDENTIALS ✅

Working Configuration (Verified 2026-06-05)

Snowflake can directly query S3 Tables Iceberg tables via the Glue Iceberg REST endpoint with VENDED_CREDENTIALS. Here's the complete working setup:

-- 1. Catalog Integration (CRITICAL: explicit ACCESS_DELEGATION_MODE)
CREATE OR REPLACE CATALOG INTEGRATION s3tables_glue_rest_int
  CATALOG_SOURCE = ICEBERG_REST
  TABLE_FORMAT = ICEBERG
  CATALOG_NAMESPACE = 'metadata'
  REST_CONFIG = (
    CATALOG_URI = 'https://glue.ap-northeast-1.amazonaws.com/iceberg'
    CATALOG_API_TYPE = AWS_GLUE
    CATALOG_NAME = '<ACCOUNT_ID>:s3tablescatalog/fsxn-metadata-catalog'
    ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS  -- MUST be explicit
  )
  REST_AUTHENTICATION = (
    TYPE = SIGV4
    SIGV4_IAM_ROLE = 'arn:aws:iam::<ACCOUNT_ID>:role/fsxn-snowflake-verification-role'
    SIGV4_SIGNING_REGION = 'ap-northeast-1'
  )
  ENABLED = TRUE;

-- 2. Schema WITHOUT default EXTERNAL_VOLUME (critical)
CREATE SCHEMA FSXN_LAKEHOUSE.S3TABLES_VENDED;
USE SCHEMA FSXN_LAKEHOUSE.S3TABLES_VENDED;

-- 3. Table WITHOUT EXTERNAL_VOLUME parameter (critical)
CREATE ICEBERG TABLE s3tables_vended_creds_test
  CATALOG = 's3tables_glue_rest_int'
  CATALOG_TABLE_NAME = 'unstructured_files';

AWS prerequisites (must be completed before Snowflake configuration):

# Register S3 Tables resource with Lake Formation (--with-federation is REQUIRED)
aws lakeformation register-resource \
  --resource-arn "arn:aws:s3tables:ap-northeast-1:<ACCOUNT_ID>:bucket/fsxn-metadata-catalog" \
  --role-arn "arn:aws:iam::<ACCOUNT_ID>:role/S3TablesRoleForLakeFormation" \
  --with-federation \
  --region ap-northeast-1

# Grant SELECT + DESCRIBE to Snowflake's IAM role (table-level)
aws lakeformation grant-permissions \
  --principal '{"DataLakePrincipalIdentifier":"arn:aws:iam::<ACCOUNT_ID>:role/fsxn-snowflake-verification-role"}' \
  --resource '{"Table":{"CatalogId":"<ACCOUNT_ID>:s3tablescatalog/fsxn-metadata-catalog","DatabaseName":"metadata","Name":"unstructured_files"}}' \
  --permissions SELECT DESCRIBE \
  --region ap-northeast-1

IAM role policy must include: glue:GetTable, glue:GetDatabase, glue:GetCatalog, lakeformation:GetDataAccess, s3tables:GetTableBucket, s3tables:GetTable, s3tables:GetNamespace, s3tables:GetTableData, s3tables:GetTableMetadataLocation.

Verify integration (expected DESCRIBE output after creation):

DESCRIBE CATALOG INTEGRATION s3tables_glue_rest_int;

Property	Value
ENABLED	true
CATALOG_SOURCE	ICEBERG_REST
TABLE_FORMAT	ICEBERG
CATALOG_NAMESPACE	metadata
REST_CONFIG	{CATALOG_URI=https://glue.ap-northeast-1.amazonaws.com/iceberg, ...}
REST_AUTHENTICATION	{TYPE=SIGV4, SIGV4_IAM_ROLE=arn:aws:iam::<ACCOUNT_ID>:role/..., ...}
API_AWS_IAM_USER_ARN	arn:aws:iam::465774455528:user/<snowflake-user-id>
API_AWS_EXTERNAL_ID	<external-id-for-trust-policy>
REFRESH_INTERVAL_SECONDS	30

Setup step: Copy API_AWS_IAM_USER_ARN and API_AWS_EXTERNAL_ID from this output into your IAM role's trust policy to allow Snowflake to assume the role.

Verified Capabilities (2026-06-08)

Operation	Result	Performance
CREATE ICEBERG TABLE	✅	6.5s
SELECT * LIMIT 5	✅ (5 rows)	1.9s
COUNT(*)	✅ (170 rows)	66ms
DESCRIBE TABLE	✅ (23 columns)	69ms
ALTER ... SET AUTO_REFRESH = TRUE	✅	131ms
SHOW ICEBERG TABLES	✅ (UNMANAGED type)	567ms
Time Travel	✅ (available, snapshot-dependent)	—

Screenshots (URL bar excluded, S3 Tables internal bucket masked):

COUNT() returns 170 rows in 66ms*

DESCRIBE TABLE shows all 23 Iceberg columns

SHOW ICEBERG TABLES confirms UNMANAGED type with S3TABLES_GLUE_REST catalog

SELECT * LIMIT 5 returns actual file metadata from S3 Tables

AUTO_REFRESH + Time Travel verification:

AUTO_REFRESH verified: PyIceberg appended 1 record → Snowflake COUNT() automatically updated from 170 to 171 within 30 seconds*

-1200) = 170" width="800" height="457">
Time Travel: querying 20 minutes ago returns 170 (before the append), confirming snapshot history is accessible

About FILE_PATH: The FILE_PATH column shows the S3 path used during metadata ingestion (via FSx for ONTAP S3 Access Point). This is the path recorded in the Iceberg metadata catalog — it does not mean the files were copied to S3. The actual files remain on FSx for ONTAP and are accessible via NFS, SMB, or S3 Access Point depending on your application's protocol.

Key Insight: Why Previous Attempts Failed

ACCESS_DELEGATION_MODE defaults to EXTERNAL_VOLUME_CREDENTIALS when not explicitly specified. In this default mode, Snowflake validates storage access through the External Volume path, which triggers ListObjectsV2 against S3 Tables internal buckets — an operation that returns MethodNotAllowed.

With VENDED_CREDENTIALS explicit:

Snowflake calls Glue REST loadTable
Lake Formation (via GetTemporaryGlueTableCredentials) returns temporary storage credentials in the loadTable response config map
Snowflake uses these credentials to access data files directly by exact path
No ListObjectsV2 is required — Snowflake reads files by exact path from Iceberg metadata

Note: The Glue REST endpoint does not implement the standard Iceberg REST /credentials endpoint. Credential vending works through Lake Formation's proprietary mechanism embedded in the loadTable response. This is transparent to Snowflake when configured correctly.

Governance Limitation: Lake Formation Column-Level (2026-06-08)

Lake Formation column-level filtering is NOT enforced via the VENDED_CREDENTIALS path:

When AllowFullTableExternalDataAccess = false, the entire VENDED_CREDENTIALS path is blocked
Explicit column/table-level grants + ExternalDataFilteringAllowList do not resolve this
AllowFullTableExternalDataAccess = true is required for VENDED_CREDENTIALS to function

Technical context: AllowFullTableExternalDataAccess controls whether external engines (those using Lake Formation credential vending) can access table data without per-table SELECT grants. When set to false, fine-grained column/row filtering is the intended enforcement mechanism — but for S3 Tables accessed via VENDED_CREDENTIALS, this currently results in complete access denial rather than filtered access. This may be a service-specific constraint of the S3 Tables federated catalog path, or it may require additional AllowExternalDataFiltering + ExternalDataFilteringAllowList configuration that was not effective in our testing. A feature request has been submitted to AWS.

Workaround: Use Snowflake Horizon for column-level governance:

-- Row Access Policy: restrict by sensitivity_level
CREATE OR REPLACE ROW ACCESS POLICY metadata_sensitivity_filter AS
  (sensitivity_level VARCHAR) RETURNS BOOLEAN ->
    CASE
      WHEN IS_ROLE_IN_SESSION('SECURITY_ADMIN') THEN TRUE
      WHEN sensitivity_level IN ('public', 'internal') THEN TRUE
      ELSE FALSE
    END;

ALTER TABLE s3tables_vended_creds_test ADD ROW ACCESS POLICY
  metadata_sensitivity_filter ON (sensitivity_level);

-- Dynamic Data Masking: hide embedding vectors from non-ML roles
CREATE OR REPLACE MASKING POLICY mask_embedding AS
  (val BINARY) RETURNS BINARY ->
    CASE
      WHEN IS_ROLE_IN_SESSION('ML_ENGINEER') THEN val
      ELSE NULL
    END;

ALTER TABLE s3tables_vended_creds_test MODIFY COLUMN
  embedding_vector SET MASKING POLICY mask_embedding;

Snowflake Iceberg Access Modes (Summary)

Access mode	Best for	Status
Glue REST + VENDED_CREDENTIALS	S3 Tables direct query	✅ VERIFIED
External Stage (FSx S3 AP) + TO_FILE	File AI analysis (Cortex COMPLETE)	✅ VERIFIED
Metadata sync to Snowflake table	BI / Cortex Search / governance	Available
Object Store Catalog	Direct metadata file read	❌ Blocked (S3 Tables internal bucket)
Snowflake Open Catalog (Polaris)	Alternative Iceberg catalog	Not tested

📖 Investigation History (2026-06-01 to 2026-06-05) — click to expand

2026-05-31: Tested S3 Tables direct REST endpoint as External Iceberg catalog → not a supported catalog type.

2026-06-01: Created CATALOG INTEGRATION using ICEBERG_REST + AWS_GLUE + VENDED_CREDENTIALS. DESCRIBE succeeded but CREATE ICEBERG TABLE failed with "Failed to retrieve credentials from the Catalog". Root cause identified: Glue REST does not implement /credentials endpoint (UnknownOperationException).

2026-06-02: AWS Support confirmed Lake Formation uses proprietary mechanism (GetTemporaryGlueTableCredentials) for credential vending, not standard Iceberg REST /credentials. Snowflake Support confirmed Error 004174 occurs when s3.access-key-id/secret/token absent from loadTable response.

2026-06-02: Tested Object Store catalog and EXTERNAL_VOLUME_CREDENTIALS mode — both blocked by S3 Tables internal bucket rejecting ListObjectsV2.

2026-06-03: Discovered register-resource --with-federation was missing. After setup, loadTable response included credentials. However, CREATE TABLE still failed at storage validation (ListObjectsV2).

2026-06-05: Snowflake Support identified the critical distinction: ACCESS_DELEGATION_MODE defaults to EXTERNAL_VOLUME_CREDENTIALS. Explicitly setting VENDED_CREDENTIALS + schema without External Volume + CREATE TABLE without External Volume parameter → SUCCESS. CREATE TABLE + SELECT both working.

2026-06-08: Additional testing confirmed COUNT(*), DESCRIBE, AUTO_REFRESH, SHOW ICEBERG TABLES all working. Lake Formation column-level filtering NOT enforced via this path (AllowFullTableExternalDataAccess=false blocks all access).

External Stage note: Snowflake External Stage against the FSx S3 Access Point alias was verified in this PoC (2026-05-31, ap-northeast-1). Update (2026-06-02): TO_FILE (Cortex COMPLETE multimodal) also verified working — Claude Sonnet 4.5 can directly read files from FSx for ONTAP via S3 AP-backed External Stage. See snowflake/external-stage-fsx-s3ap-validation.md for exact DDL and verified operations.

Snowflake Metadata Activation Pattern

If you sync only the metadata into Snowflake (not raw files), you preserve the zero-copy principle for actual data while enabling Snowflake-native use cases:

Governed metadata analytics and executive dashboards
File inventory and PII coverage reporting
Cortex Search over redacted summaries (RAG on metadata)
Snowflake Intelligence / Cortex Analyst style business Q&A
Row Access Policies and Dynamic Masking on synced metadata

Horizon Catalog note: When metadata reaches Snowflake, Snowflake governance features such as Row Access Policies and Dynamic Masking can be applied to Snowflake-managed access paths. For external engine access via Iceberg REST, validate the exact Open Catalog / Horizon behavior for your target engine and security model.

Metadata sync best practice: Sync curated latest-record metadata, not the append-only base table, unless analysts explicitly need history. Preserve scan_run_id, change_type, and is_deleted for audit and reconciliation. Use MERGE INTO keyed by file_id or path_hash to make metadata activation idempotent. See snowflake/metadata-sync-example.sql for the full pattern.

Governance policy mapping: When syncing metadata into Snowflake, map AWS-side fields such as sensitivity_level, tenant_id, pii_status, and path_classification to Snowflake tags, masking policies, and row access policies. Track policy drift between Lake Formation and Snowflake governance. See snowflake/path-decision-guide.md for the full policy mapping.

Snowflake Cortex Search Activation Pattern

If redacted metadata and summaries are synced into Snowflake, Cortex Search can provide Snowflake-native enterprise search and RAG over metadata — without managing embeddings, infrastructure, or search quality tuning.

Why Cortex Search here:

Business users can search approved metadata without operating a separate vector database
RAG and enterprise search can run over redacted summaries already governed in Snowflake
Search quality, embedding management, and index refresh are delegated to Snowflake-managed services
This is best suited for Snowflake-first organizations that want business-facing discovery inside the AI Data Cloud

Use Cortex Search for:

Executive metadata search (natural language queries over file inventory)
File inventory Q&A (powered by LLM + retrieval)
PII coverage reporting and compliance dashboards
Governed search over redacted summaries

OpenSearch Serverless NextGen remains the AWS-native serving index for this PoC. Cortex Search is an optional Snowflake-native alternative for organizations that standardize on Snowflake for business discovery.

Role separation: S3 Tables / Iceberg remains the metadata source of truth. OpenSearch (AWS path) or Cortex Search (Snowflake path) are serving indexes for search UX. Choose based on your primary platform. Cortex Search operates over redacted summaries and metadata synced into Snowflake, not raw files, unless the customer explicitly chooses to copy/extract document content into Snowflake (which would break the zero-copy raw data principle).

Cortex Search scope: Cortex Search should operate on redacted metadata and summaries by default. If raw document content is extracted or copied into Snowflake for Cortex use cases, treat that as a separate data movement decision with its own governance, retention, and cost model.

Snowflake activation cost drivers: Snowflake activation introduces separate cost drivers from the AWS-native catalog: warehouse compute for metadata sync tasks and dashboards, Cortex Search service usage (based on corpus size and query volume), task/stream orchestration for refresh, and small metadata storage. These costs should be modeled separately from the AWS-native catalog cost ($114/month estimate in Part 1 does not include Snowflake-side compute).

Retention alignment: Confirm Snowflake account edition, table type, and retention settings before promising deletion SLAs. Snowflake Time Travel (1–90 days) and Fail-safe (7 days) operate independently from Iceberg snapshot expiration. Snowflake-side deletion evidence should be retained separately from Iceberg snapshot expiration evidence.

Snowflake Metadata Product Contract

When activating metadata in Snowflake, expose a curated subset as the governed metadata product:

Recommended curated columns:

Column	Purpose	Governance
file_id	Unique identifier	—
business_domain	Organizational grouping	Row access policy
file_type	File format	—
classification	AI-generated classification	—
sensitivity_level	Data sensitivity tier	Snowflake tag + masking policy
pii_status	PII detection result	Access policy / dashboard filter
redacted_summary	AI-generated (PII-free) summary	Cortex Search source column
owner_team	Business ownership	Business glossary / stewardship
last_seen_at	Last scan timestamp	—
data_quality_status	Enrichment quality flag	—

Snowflake governance mapping:

sensitivity_level → Snowflake tag + masking policy
tenant_id / business_domain → row access policy
pii_status → access policy / dashboard filter
redacted_summary → Cortex Search source column
owner_team → business glossary / stewardship workflow

Databricks Metadata Activation Pattern

If UC Foreign Catalog is not yet validated for your S3 Tables path, sync only the redacted metadata into a UC-managed Delta table. This preserves the zero-copy principle for raw files while enabling Databricks-native use cases:

Databricks SQL dashboards and executive reporting
AI/BI Genie over curated metadata (natural language queries)
UC lineage and audit on metadata usage
ML feature generation from file metadata
Operational reporting on PII coverage and enrichment backlog

Raw files remain on FSx for ONTAP. Only the small metadata table (~MB scale for 100K files) is synced.

This is analogous to the Snowflake metadata activation pattern: it copies only curated metadata, not the original unstructured files. Both patterns preserve the zero-copy principle for raw data.

Databricks Raw File Access Decision:

Requirement Recommended path

Governed metadata analytics only UC Foreign Catalog (if validated) or sync metadata to UC Delta

Raw file processing in Databricks DataSync → S3 → UC External Volume

Zero-copy raw file access from Databricks Not supported in validated paths (NFS mount works but without UC governance)

Business discovery / BI Sync redacted metadata to UC Delta table

If metadata is synced into Databricks for BI, include Databricks SQL / Jobs compute cost in the activation model. This does not affect raw-file zero-copy storage, but it is part of the business-facing analytics cost.

Requirement	Recommended path
Governed metadata analytics only	UC Foreign Catalog (if validated) or sync metadata to UC Delta
Raw file processing in Databricks	DataSync → S3 → UC External Volume
Zero-copy raw file access from Databricks	Not supported in validated paths (NFS mount works but without UC governance)
Business discovery / BI	Sync redacted metadata to UC Delta table

Other Lakehouse Engines to Validate

Beyond Databricks and Snowflake, the most natural validation targets for this metadata catalog are:

Engine	Access path	Likely fit	Validation priority
Trino / Starburst	Glue Iceberg REST or S3 Tables REST	Federated SQL, ad hoc query	High
EMR Spark	Glue Iceberg REST (native since EMR 7.5.0+)	Bulk backfill, batch enrichment	High
Redshift Spectrum	Glue catalog (external schema)	DWH integration, BI	Medium
Dremio	Glue catalog or Iceberg REST	Query acceleration, BI	Medium
StarRocks / Doris	Glue Iceberg REST	Low-latency serving queries	Medium
Apache Flink	Glue Iceberg REST	Streaming metadata updates	Low
dbt (via Athena)	dbt-athena + Iceberg materialization	Analytics engineering, governed marts	Medium
Apache NiFi	Iceberg REST or Polaris	Event-driven ingestion	Low

These engines should be validated against:

S3 Tables direct REST vs AWS Glue Iceberg REST
Read vs write capability
Lake Formation behavior (credential vending, column/row filtering)
Snapshot freshness after external writes
Latest-record view compatibility
Case-sensitivity and lowercase naming requirements

Key finding from validation (2026-06-08): AWS Glue Iceberg REST supports SigV4-authenticated catalog access. Lake Formation credential vending works through a proprietary mechanism (GetTemporaryGlueTableCredentials). Snowflake requires explicit ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS — the default mode fails. Engines that can sign requests with their own IAM credentials (EMR Spark ✅ verified, Trino on EMR expected, PyIceberg ✅ verified) work out of the box. Snowflake also works when configured correctly (✅ verified 2026-06-05). EMR requirement: 7.13.0+ (7.5.0 has a credential resolution bug). Governance note: Lake Formation column-level filtering is NOT enforced via the VENDED_CREDENTIALS path for Snowflake.

Trino note: AWS has published guidance on querying S3 Tables from Trino using the Iceberg REST endpoint. Trino's Iceberg connector supports REST catalogs natively, making it one of the most straightforward third-party validation targets.

EMR Spark note: For large-scale backfill or re-enrichment (100K+ files), Spark on EMR Serverless or EMR on EC2 can be used as an alternative to Lambda/Fargate. Use Glue Iceberg REST for centralized metadata access with Lake Formation governance. Verified (2026-06-02): EMR Serverless Spark 7.13.0 successfully reads S3 Tables metadata via Glue Iceberg REST — SHOW NAMESPACES, SHOW TABLES, SELECT, COUNT, and snapshot history all work. Requires EMR 7.13.0+ (7.5.0 has a credential resolution bug for S3 Tables warehouse format).

Redshift note: Validate separately from Athena — external schema setup, Glue statistics, Lake Formation permissions, and query latency against latest-record views may differ.

For the full compatibility matrix, see lakehouse-tools/tool-compatibility-matrix.yaml.

Catalog Authority Rule

For each Iceberg table, define exactly one authoritative catalog for metadata pointer and commit coordination. Do not operate S3 Tables, Polaris, Gravitino, Nessie, and Glue as independent writable catalogs for the same table unless the integration explicitly supports federation without dual writes.

                    ┌──────────────────────┐
                    │ Authoritative Catalog│
                    │ (ONE per table)      │
                    │ • S3 Tables + Glue   │
                    │   (this PoC)         │
                    └──────────┬───────────┘
                               │
              ┌────────────────┼────────────────┐
              │                │                │
              ▼                ▼                ▼
         Read-only        Read-only        Read-only
         consumers        consumers        consumers
         (Trino,          (Databricks,     (Snowflake,
          Dremio,          UC Foreign       Cortex,
          StarRocks)       Catalog)         Open Catalog)

Split-brain warning: If two catalogs independently write to the same Iceberg table, snapshot pointers can diverge, causing data loss or corruption. Federation (one writer, many readers) is safe. Dual-write is not.

The Bigger Picture

                    S3 Tables (Iceberg)
                           │
              ┌────────────┼────────────┐
              │            │            │
              ▼            ▼            ▼
         Athena ✅    Databricks ❌  Snowflake ✅
         EMR Spark ✅  (UC Foreign    (Glue REST +
         PyIceberg ✅   path still    VENDED_CREDENTIALS
                      blocked in     verified 2026-06-05)
                      tested config)

Databricks integration summary (confirmed 2026-06-01):

Direct S3 AP access: ❌ (UC session policy)
NFS mount → UC Volume: ❌ (cloud URI only)
Delta Sharing via S3 AP: ❌ (same credentials)
DataSync → S3 → UC: ✅ (supported workaround, not zero-copy for synced data)
UC Foreign Catalog / Foreign Iceberg via Glue: ❌ Retested 2026-06-09; Glue Connection and credentials succeeded, but UC External Location validation failed against S3 Tables internal storage. Support case submitted.

For capability-level details such as read, write, time travel, metadata tables, and governance behavior, see verification-evidence/cross-platform-compatibility.yaml.

This is a temporary gap. S3 Tables is relatively new (GA Dec 2024), and cross-platform federation is actively being developed. Feature requests have been filed with both platforms. Timeline for native S3 Tables support is unknown, but the Iceberg ecosystem is converging rapidly — Unity Catalog 2.0's native Iceberg support and Snowflake's Open Catalog (Polaris) both point toward broader interoperability.

Catalog Decision Guide

In the Iceberg world, the catalog is the system of record for table metadata pointers and atomic operations. Choose based on your primary platform:

Primary platform	Recommended catalog	Notes
AWS-first / Athena-first	S3 Tables + Glue/Lake Formation	Used in this PoC
Databricks-first	Unity Catalog Managed/Foreign Iceberg	Best for UC governance, lineage, discovery
Snowflake-first	Snowflake Open Catalog (Polaris)	Best for Snowflake-governed Iceberg interoperability; validate external engine governance behavior
Neutral / OSS-first	Apache Polaris or other REST catalog	Maximum portability

Dual catalog warning: Avoid running two authoritative catalogs for the same Iceberg table. Use Snowflake Open Catalog / Polaris when Snowflake or a neutral REST catalog should be authoritative. Use S3 Tables when AWS-native Athena / Lake Formation / Glue governance is authoritative. If both platforms need access, use federation (one authoritative catalog, others read via REST).

When to Consider Snowflake Open Catalog / Polaris

Use S3 Tables + Glue/Lake Formation when AWS-native governance is authoritative (this PoC).

Consider Snowflake Open Catalog / Polaris when:

Snowflake should be the primary governance and interoperability plane
Multiple engines need Iceberg REST access through a neutral catalog
Snowflake-managed Iceberg or Snowflake-first AI/Data Cloud workflows are the center of gravity
You want managed Polaris instead of operating your own REST catalog

This would be a different authoritative-catalog design from the current PoC and should not be mixed as a second writer for the same table.

Databricks-first note: For organizations standardizing on Databricks, consider whether the metadata catalog itself should be managed in Unity Catalog as Managed Iceberg or Delta + UniForm, then exposed to AWS engines through Glue federation to UC or the UC Iceberg REST endpoint. Use S3 Tables when AWS-native Athena/Lake Formation is the primary governance path. The choice depends on which governance plane (UC or Lake Formation) is authoritative for your organization.

Format Decision for Databricks Environments

Option	Best for	Tradeoff
S3 Tables Iceberg	AWS-first Athena/LF governance	UC integration pending (Foreign Catalog validation)
UC Managed / Foreign Iceberg	Databricks-first open format governance	Validate current feature availability, region support, and limitations
Delta + UniForm	Databricks-native pipelines + Iceberg read compatibility	Iceberg metadata generated asynchronously; non-Databricks writes constrained
Metadata sync to Delta	BI activation in Databricks SQL	Metadata copy, but raw files remain zero-copy

Summary: What We Built

Layer	Technology	Status
Storage	FSx for ONTAP (files) + S3 Tables (metadata)	✅ Verified
AI	Bedrock Claude Vision + Titan Embeddings V2	✅ Verified
Search	OpenSearch Serverless NextGen (scale-to-zero)	✅ Verified
Governance	Lake Formation (table-level) + CloudTrail	✅ Verified
PII	Comprehend (EN) + Bedrock Claude (JA)	✅ Verified
Cross-platform	Athena ✅, EMR Spark ✅, PyIceberg ✅, Snowflake ✅, Databricks ⚠️	Mostly verified

The Numbers

42 seconds: Full demo execution time
$0.07: Total demo cost
Near $0 idle compute/search cost: Persistent metadata, logs, and audit trails may still incur small charges
$114/month: Projected cost at 100K files, 1000 changes/day
95%: Storage cost reduction vs S3 full copy
0.95: AI classification confidence (invoice detection)
7/7: PII entities detected and redacted

For regulated workloads, align Iceberg snapshot retention with deletion SLAs and audit evidence retention.

What's Next for This Project

Monitor support cases: Databricks UC Foreign Catalog for S3 Tables — timeline unknown
Production hardening: SQS batching, DLQ alerting, reconciliation jobs
Multi-language PII: Extend beyond EN/JA to other languages
Cost optimization: Provisioned Throughput for high-volume Bedrock usage
Production semantics: File identity, latest-record views, index reconciliation, and snapshot retention alignment
ONTAP production hardening: S3 Access Point identity matrix, FPolicy event filtering, SnapMirror catalog rebinding, and FSx performance dashboard
Snowflake governance: Implement Horizon Row Access Policies and Dynamic Data Masking for column-level protection (since Lake Formation column-level is not enforced via VENDED_CREDENTIALS)

Get Involved

⭐ Star the repo if this was useful
🐛 Open an Issue for questions or suggestions
🍴 Fork and adapt for your own unstructured data catalog

This concludes the 3-part series. All code is at github.com/Yoshiki0705/fsxn-lakehouse-integrations. Questions? Open a GitHub Issue.

Governance disclaimer: This article provides governance guidance and architectural patterns. It does not substitute for legal or compliance judgment. Final regulatory determinations should be confirmed with legal and compliance teams.

AI Enrichment Pipeline: From Sample Classification to 100K-File Metadata Search with Bedrock and OpenSearch NextGen

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Mon, 08 Jun 2026 16:37:44 +0000

Quick Recap: What We Built in Part 1

In Part 1, we built a metadata catalog on Apache Iceberg (S3 Tables) that makes unstructured files on FSx for ONTAP instantly searchable via Athena SQL — in under 2 seconds, at $5-15/month for 100K files, without bulk-copying raw files.

But basic metadata (file name, size, type) only gets you so far. What if you could ask: "Find all invoices from Q4" or "Show me files similar to this contract"?

That requires AI enrichment: automatically classifying files and generating vector embeddings for similarity search.

What We're Building

File on FSx for ONTAP
       │
       │ S3 Access Point (read)
       ▼
┌─────────────────────────────────────────┐
│  Bedrock Claude Vision                  │
│  "What is this file?"                   │
│  → classification: "invoice"            │
│  → confidence: 0.95                     │
│  → summary: "Invoice #INV-2024-..."     │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  Titan Embeddings V2                    │
│  "Represent this file as a vector"      │
│  → 1024-dimensional embedding           │
│  → normalized for cosine similarity     │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  S3 Tables (Iceberg)                    │
│  classification, confidence_score,      │
│  summary, embedding_vector              │
└──────────────────┬──────────────────────┘
                   │
                   ▼
┌─────────────────────────────────────────┐
│  OpenSearch Serverless NextGen          │
│  kNN vector search                      │
│  "Find files similar to X"              │
│  Scale-to-zero: $0 when idle            │
└─────────────────────────────────────────┘

AI Classification: Bedrock Claude Vision

How It Works

For image files (PNG, JPEG, TIFF), we send the file to Claude Vision with a simple prompt:

response = bedrock.invoke_model(
    modelId="anthropic.claude-3-haiku-20240307-v1:0",
    body=json.dumps({
        "anthropic_version": "bedrock-2023-05-31",
        "max_tokens": 512,
        "messages": [{"role": "user", "content": [
            {"type": "image", "source": {
                "type": "base64",
                "media_type": "image/png",
                "data": image_b64
            }},
            {"type": "text", "text": 
                'Classify this image. Respond JSON only: '
                '{"classification":"...","confidence_score":0.X,"summary":"..."}'}
        ]}]
    })
)

Results (Measured)

File	Classification	Confidence	Latency	Cost
invoice_sample.png	Invoice	0.95	~3s	$0.01
product_inspection.png	Pie Chart	1.0	~2s	$0.01
sensor_dashboard.png	IoT Sensor Dashboard	0.9	~3s	$0.01

Key insight: In this demo, Claude 3 Haiku classified sample images in ~2-3 seconds at roughly $0.01/image. Production accuracy and cost depend on image size, prompt length, model version, and document type.

Model version note: Model ID anthropic.claude-3-haiku-20240307-v1:0 was used at time of testing. Check Bedrock model IDs for the latest available version.

For Non-Image Files

File Type	Enrichment Strategy	Cost
PDF	Extract text → summarize with Claude	$0.01-0.05
CSV/Parquet	Schema extraction + row count	~$0 (metadata only)
Audio	Transcribe → summarize	$0.02-0.10
Video	Frame sampling → Vision	$0.05-0.50
CAD/3D	Metadata extraction only	~$0

Vector Embeddings: Titan Embeddings V2

Every file gets a 1024-dimensional vector embedding based on its content or AI-generated description:

response = bedrock.invoke_model(
    modelId="amazon.titan-embed-text-v2:0",
    body=json.dumps({
        "inputText": summary_text,  # AI-generated description
        "dimensions": 1024,
        "normalize": True
    })
)
embedding = json.loads(response["body"].read())["embedding"]
# → [0.023, -0.041, 0.089, ...] (1024 floats)

Why 1024 Dimensions?

Dimensions	Cost	Accuracy	Storage	Use Case
256	Lowest	Good	1KB/file	High-volume, cost-sensitive
512	Low	Better	2KB/file	General purpose
1024	Medium	High	4KB/file	Recommended balance
1536	Higher	Highest	6KB/file	Maximum precision

1024 dimensions was a practical default for this PoC. Validate 256/512/1024/1536 dimensions against your own top-k relevance and storage/cost targets (~4KB per file × 100K files = 400MB total at 1024-dim).

Pricing note: Titan Embeddings V2 charges per 1K input tokens ($0.00002). The cost is the same whether you request 256, 512, or 1024 dimensions — so there's no cost penalty for choosing higher dimensions.

Embedding Storage in Iceberg

Embeddings are stored as binary type in the Iceberg table:

import struct

# Convert float list to binary for Iceberg storage
embedding_bytes = struct.pack(f"{len(embedding)}f", *embedding)

# Write to Iceberg table
arrow_table = pa.table({
    "file_id": [file_id],
    "embedding_vector": [embedding_bytes],
    "enrichment_status": ["completed"],
    ...
})
table.append(arrow_table)

Important: Append-Only Writes and Deduplication

Iceberg on S3 Tables uses append-only writes. If you enrich the same file twice (e.g., after a retry), you'll get duplicate records. Use this dedup pattern in queries:

WITH ranked AS (
  SELECT *, ROW_NUMBER() OVER (PARTITION BY file_id ORDER BY modified_at DESC) as rn
  FROM "s3tablescatalog/fsxn-metadata-catalog"."metadata"."unstructured_files"
)
SELECT * FROM ranked WHERE rn = 1 AND is_deleted = false;

S3 Tables auto-compaction handles the storage overhead of duplicates over time.

Vector Search: OpenSearch Serverless NextGen

The Scale-to-Zero Revolution

Before May 2026, OpenSearch Serverless had a minimum cost of ~$350/month (2 OCUs always running). This made it impractical for PoC and dev environments.

OpenSearch Serverless NextGen (GA May 2026) introduces scale-to-zero:

State	Cost	Latency
Idle (no queries)	$0/month	—
Cold start (first query)	$0.24/OCU-hour	10-30 seconds
Warm (subsequent queries)	$0.24/OCU-hour	~54ms

This changes the economics completely: you can keep vector search compute cost near zero until you actually need it.

kNN Search Implementation

from opensearchpy import OpenSearch
from requests_aws4auth import AWS4Auth

# Generate query embedding
query_embedding = get_embedding("find invoice or payment documents")

# kNN search
results = client.search(index="fsxn-metadata-embeddings", body={
    "size": 5,
    "query": {
        "knn": {
            "embedding_vector": {
                "vector": query_embedding,
                "k": 5
            }
        }
    }
})

Note: Vector search requires OpenSearch — you cannot perform kNN queries directly on the embedding_vector binary column in Athena. The Iceberg table stores embeddings for durability; OpenSearch provides the search interface.

Search Results (Measured)

Query: "find invoice or payment documents"

Results:
  1. invoice_sample.png (score: 0.6749)
     Classification: Invoice
     Summary: "Invoice #INV-2024-..."

  2. (other similar files ranked by cosine similarity)

Score interpretation:

0.9+: Near-identical content
0.7-0.9: Highly similar
0.5-0.7: Related topic
< 0.5: Weak or no relation

Our score of 0.67 for "invoice or payment documents" → invoice_sample.png is reasonable — the query is broad, and the match is correct.

Improving search scores: Use more specific queries ("Q4 2024 invoice from vendor ABC" vs "find invoices"), enrich files with more detailed summaries, or increase embedding dimensions to 1536 for higher precision at ~50% more storage cost.

These score bands are demo heuristics, not universal thresholds. Calibrate thresholds with labeled examples for each document type and business workflow.

The Complete Pipeline

Processing Flow

New file detected (FPolicy event or batch scan)
       │
       ▼
┌─ Is it an image? ──────────────────────────┐
│  YES → Claude Vision classification        │
│  NO  → Metadata-only (file type, size)     │
└────────────────────────────────────────────┘
       │
       ▼
┌─ Generate embedding ──────────────────────┐
│  Input: classification + summary text     │
│  Output: 1024-dim normalized vector       │
└───────────────────────────────────────────┘
       │
       ▼
┌─ Write to S3 Tables (Iceberg) ────────────┐
│  classification, confidence_score,        │
│  summary, embedding_vector,               │
│  enrichment_status = "completed"          │
│  index_status = "pending"                 │
└───────────────────────────────────────────┘
       │
       ▼
┌─ Index in OpenSearch ─────────────────────┐
│  file_id, embedding_vector, metadata      │
│  (for kNN similarity search)              │
│  index_status = "indexed" / "stale" / "failed" │
└───────────────────────────────────────────┘

Error Handling

Error	Strategy	Result
Bedrock ThrottlingException	Exponential backoff (1s, 2s, 4s)	Retry up to 3 times
Bedrock ModelNotReadyException	Wait 5s, retry	Model warming up (first invocation)
File read failure (S3 AP)	Mark as `failed`, retry next cycle	No data loss
Low confidence (< 0.3)	Mark as `low_confidence`	Human review queue
Lambda timeout (large files)	Fallback to ECS Fargate	No timeout limit

Monitoring the Pipeline

How do you know when something goes wrong? Set up these CloudWatch alarms:

Metric	Source	Alert Condition	Action
DLQ message count	CloudWatch (SQS)	> 0	Inspect DLQ messages, redrive
Lambda error rate	CloudWatch (Lambda)	> 5% for 5 min	Check logs, Iceberg commit conflict?
Bedrock throttling	CloudWatch (Bedrock)	> 10/min	Reduce request rate, adjust backoff
Enrichment backlog	Athena query (pending count)	> 1000	Increase Lambda concurrency or batch size
OpenSearch index size	OpenSearch metrics	> 80% capacity	Add shards or rotate index

# Quick health check: DLQ + Lambda errors in one command
aws cloudwatch get-metric-data --metric-data-queries '[
  {"Id":"dlq","MetricStat":{"Metric":{"Namespace":"AWS/SQS","MetricName":"ApproximateNumberOfMessagesVisible","Dimensions":[{"Name":"QueueName","Value":"fsxn-metadata-sync-dlq"}]},"Period":300,"Stat":"Sum"}},
  {"Id":"errors","MetricStat":{"Metric":{"Namespace":"AWS/Lambda","MetricName":"Errors","Dimensions":[{"Name":"FunctionName","Value":"fsxn-metadata-sync"}]},"Period":300,"Stat":"Sum"}}
]' --start-time $(date -u -v-1H +%Y-%m-%dT%H:%M:%S) --end-time $(date -u +%Y-%m-%dT%H:%M:%S)

For detailed operational monitoring guidance, see the Operational Monitoring section in the architecture document.

Cost at Scale

Volume	AI Cost	Embedding Cost	OpenSearch	Total
100 files/day	$1/day	$0.002/day	$0 (idle)	~$30/month
1,000 files/day	$10/day	$0.02/day	~$42/month	~$342/month
10,000 files/day	$100/day	$0.20/day	~$84/month	~$3,084/month

At 10K files/day, consider batch processing during off-hours and Provisioned Throughput for Bedrock to reduce per-request cost.

Cost optimization tip: Not all files need AI enrichment. A common pattern: images → Vision classification, documents → text extraction + embedding, data files (CSV/Parquet) → metadata only (no AI cost). This can reduce AI costs by 60-80% depending on your file mix.

Batch Inference: For initial bulk enrichment (10K+ files), Bedrock Batch Inference can reduce costs by ~50% compared to real-time invocations. Use real-time for incremental new files, batch for backfill.

# Batch Inference example — submit a batch job for bulk classification
import boto3, json

bedrock = boto3.client("bedrock", region_name="ap-northeast-1")

# 1. Prepare input JSONL file in S3 (one request per line)
# Each line: {"recordId":"file-001","modelInput":{"anthropic_version":"bedrock-2023-05-31",...}}

# 2. Create batch inference job
response = bedrock.create_model_invocation_job(
    jobName="metadata-enrichment-backfill",
    modelId="anthropic.claude-3-haiku-20240307-v1:0",
    roleArn="arn:aws:iam::<ACCOUNT>:role/BedrockBatchRole",
    inputDataConfig={
        "s3InputDataConfig": {
            "s3Uri": "s3://my-bucket/batch-input/enrichment-requests.jsonl"
        }
    },
    outputDataConfig={
        "s3OutputDataConfig": {
            "s3Uri": "s3://my-bucket/batch-output/"
        }
    }
)
# Job runs asynchronously — results written to S3 when complete
# Typical processing: 10K files in ~30 minutes at ~50% cost reduction

The batch input JSONL contains prompts, file references, or extracted/redacted text depending on your design. It does not require copying the original raw files from FSx for ONTAP to S3. If images are included as base64, treat the JSONL as temporary processing data.

Batch job monitoring: Use EventBridge rules to detect Bedrock batch job state changes (COMPLETED, FAILED). Route to SNS → Lambda to automatically write results back to S3 Tables.

Prompt Caching: If using the same system prompt across all classifications (recommended), Bedrock's Prompt Caching feature can reduce input token costs by up to 90% for repeated prompts.

EMR Spark for large-scale backfill: For initial backfill or re-enrichment of 100K+ files, Spark on EMR Serverless or EMR on EC2 can be used as an alternative to Lambda/Fargate. EMR 7.13.0+ supports Glue as an Iceberg REST catalog, enabling distributed metadata writes with Lake Formation governance. Verified 2026-06-02: SELECT, COUNT, and time travel all work on EMR Serverless 7.13.0. Use Lambda for incremental (event-driven) processing and Spark for bulk operations.

Search Index Consistency

OpenSearch is a derived index, not the system of record. S3 Tables / Iceberg remains the metadata source of truth.

Recommended controls:

Store iceberg_snapshot_id in OpenSearch documents for traceability
Store embedding_model_id and prompt_version in both Iceberg and OpenSearch
Reconcile OpenSearch index against latest Iceberg view periodically
Mark index_status: pending / indexed / stale / failed
If search returns a stale result, fall back to Athena query on the base table

FPolicy Event Design

For incremental metadata sync via ONTAP FPolicy:

Use batch scan for initial backfill (not FPolicy)
Use FPolicy only for incremental changes after initial catalog population
Prefer create / close-with-modification / rename / delete events
Avoid read / open events (excessive volume, no catalog value)
Apply path and extension filters to reduce event noise
Add backpressure via SQS batching (not fan-out Lambda per event)

FPolicy can significantly impact file system throughput if configured too broadly. Filter to only the operations and paths that matter for catalog updates.

Hybrid Search Pattern

For production discovery, vector search should be combined with lexical filters and keyword search:

Lexical search: file_name, path, classification, summary, tags
Vector search: embedding similarity (kNN)
Filters: tenant_id, sensitivity_level, file_type, path_classification, last_modified

OpenSearch supports both search and vector collection types. Use a single index with both text fields and vector fields for hybrid queries. S3 Tables / Iceberg remains the metadata source of truth; OpenSearch is the serving index.

For sensitive workloads, use VPC interface endpoints for Bedrock Runtime and S3 VPC endpoints for batch input/output. See genai/bedrock-private-connectivity.md.

Storage Tier Impact During Backfill

Initial AI enrichment may read cold files from capacity pool storage, causing higher latency and throughput consumption.

Recommended controls:

Run backfill during off-hours (minimize impact on production NFS/SMB)
Limit Lambda concurrency during backfill
Enrich only selected file types first (images → documents → data files)
Monitor FSx capacity pool read activity via CloudWatch
Separate backfill cost from steady-state cost in planning

Backfill vs Incremental Cost Model

Separate cost planning for:

Phase	Scope	Cost driver	Optimization
Initial backfill	All existing files (e.g., 100K)	Bedrock AI at scale	Batch Inference (~50% savings)
Daily incremental	New/modified files (e.g., 1000/day)	Real-time Lambda + Bedrock	Selective enrichment by file type
Re-enrichment	After prompt/model change	Full re-scan of enriched files	Batch + compare confidence delta
OpenSearch reindex	After schema/embedding change	Index rebuild	Off-hours, parallel shards

The largest cost spike is typically the initial backfill, not steady-state. Plan Bedrock Batch Inference and off-peak scheduling for the first catalog population.

For adjustable assumptions, see verification-evidence/cost-assumptions.yaml.

Try It Yourself

# Enrich pending files with AI
python3 demo/scripts/demo-enrich.py \
  --table-bucket-arn <TABLE_BUCKET_ARN> \
  --ap-alias <AP_ALIAS> \
  --max-files 10

# Search by natural language
python3 demo/scripts/demo-search.py \
  --query "find documents about contracts or agreements"

AI Safety and Human Review Boundary

AI enrichment should not be treated as authoritative classification for regulated data without human review.

For regulated industries: AI enrichment is assistive metadata generation, not authoritative classification. Final regulatory classification must be confirmed by data owners, security, legal, and compliance teams. This system provides AI-generated signals to accelerate human review — it does not replace it.

Deterministic vs AI boundary: AI generates classifications and summaries, but pipeline state transitions, retry logic, deduplication, access controls, and audit evidence are deterministic and version-controlled. The deterministic pipeline guarantees reproducibility; AI provides enrichment quality.

Recommended controls:

Human review queue for low-confidence classifications (< 0.7)
Sampling review for high-confidence results (periodic spot-check)
False negative testing for PII detection
Model/prompt version recorded in metadata (enriched_at + model ID)
Re-enrichment policy when model or prompt changes

Recommended metadata columns for AI lineage:

classification_model_id — which model produced the classification
embedding_model_id — which model produced the embedding
prompt_version — version of the classification prompt
enrichment_code_version — version of the enrichment Lambda/script
enriched_at — timestamp of enrichment
human_review_status — pending / approved / rejected
human_reviewed_by — reviewer identity (if applicable)
human_reviewed_at — review timestamp

Evaluation Plan

For production use, do not rely only on model-reported confidence. Create a labeled validation set and measure:

Classification accuracy (overall and per document type)
Precision / recall per category
False positive rate for PII detection
False negative rate for PII detection
Embedding search top-k relevance (nDCG@5, MRR)
Human review acceptance rate
Cost per accepted classification

Business acceptance metrics (beyond model accuracy):

Time saved per analyst for file discovery
Dataset discovery lead-time reduction (days → hours target)
Business owner approval rate for AI classifications
Cost per useful search result
False negative risk by document category (which misses matter most?)
Governance coverage (% of assets searchable in BI/AI tools)

The 7/7 PII detection result was measured on a controlled synthetic sample. Production use requires evaluation with domain-specific documents, false-positive/false-negative review, human approval workflow, and legal/compliance sign-off.

Snowflake users: Snowflake can now directly query S3 Tables Iceberg metadata via Glue REST + VENDED_CREDENTIALS (verified 2026-06-05). Additionally, you can sync redacted metadata into Snowflake-managed tables for Cortex Search / Snowflake Intelligence business-facing discovery. In this PoC, OpenSearch remains the AWS-native vector search component.

What's Next

In Part 3, we'll cover:

Lake Formation governance: Column-level access control on metadata
PII detection and anonymization: Comprehend (English) + Bedrock Claude (Japanese)
Cross-platform access: What works and what doesn't with Databricks and Snowflake
Data Clean Room pattern: Separate tables for sensitive vs. anonymized metadata

Full code: github.com/Yoshiki0705/fsxn-lakehouse-integrations

From Hours to Seconds: An AI-Powered Metadata Catalog for Unstructured Data on FSx for ONTAP

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Mon, 08 Jun 2026 16:26:50 +0000

What Works Now vs What Requires Validation

This article separates verified AWS-native capabilities from cross-platform paths that still require validation. The core pattern — keeping raw files on FSx for ONTAP and cataloging only metadata in S3 Tables — is verified. Databricks paths are still evolving. Snowflake Glue REST + VENDED_CREDENTIALS and External Stage paths are verified in this PoC, with governance limitations noted below. Validate all cross-platform paths in your own environment before production use.

Component	Status	Notes
AWS Native PoC (Athena + S3 Tables + Bedrock + OpenSearch + Lake Formation)	✅ Verified	Full end-to-end in 42 seconds
Glue Iceberg REST endpoint access	✅ Verified	Both S3 Tables REST and Glue REST confirmed
Lake Formation table-level governance	✅ Verified	Grant/revoke/audit working
Lake Formation column-level exclusion	⚠️ Observed limitation	Failed on tested federated catalog path
Databricks SQL Warehouse direct	⚠️ Observed limitation	`iceberg_rest` connection type not supported
Databricks Spark + Iceberg REST	❌ Blocked by UC	spark.conf.set and cluster config both fail; UC Foreign Catalog required
Databricks UC Foreign Catalog	❌ Still blocked	Retested post-Foreign Iceberg GA (2026-06-09): Glue Connection ✅, Credentials ✅, but External Location fails — S3 Tables internal bucket rejects standard S3 API validation. No bypass available.
Databricks Delta Sharing via S3 AP	❌ Confirmed	Sharing server uses same UC credentials; not a workaround for S3 AP session policy
Databricks NFS → UC Volume	❌ Confirmed	Cloud storage URIs only; internal feature request exists
Databricks UC audit logging	✅ Confirmed	External engine access fully logged
Snowflake via Glue REST (VENDED_CREDENTIALS)	✅ Verified	Explicit `ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS`; CREATE TABLE + SELECT + COUNT + AUTO_REFRESH all working (2026-06-05)
Snowflake External Stage (FSx S3 AP)	✅ Verified	LIST, SELECT/COPY, and TO_FILE + Cortex AI all verified

Important distinction: This pattern does not use FSx for ONTAP S3 Access Points as an Iceberg warehouse. Raw files stay on FSx for ONTAP, while only the metadata catalog is written to S3 Tables. Direct Iceberg table writes to FSx for ONTAP S3 Access Points are tracked separately as a known limitation because Iceberg commit behavior and S3FileIO compatibility require additional validation.

This is an Iceberg Adoption Pattern, Not a Raw-Data Migration

This pattern does not convert the original unstructured files into Iceberg table data. Instead, it adopts Iceberg for the metadata layer only.

Scope	What happens
Data files	Not migrated. Raw files remain on FSx for ONTAP.
Metadata table	Newly created as an Iceberg table on S3 Tables.
Processing jobs	Metadata scan and AI enrichment jobs write append-only metadata.
Consumers	Athena, EMR, Snowflake, Databricks, and BI/search tools consume curated metadata views.

Storage Boundary: What Moves and What Doesn't

FSx for ONTAP S3 Access Point:
  ✅ Raw file READ path only (AI enrichment input)
  ❌ NOT an Iceberg warehouse
  ❌ NOT a table commit target
  ❌ NOT bulk-copied to S3

S3 Tables:
  ✅ Iceberg METADATA table (file catalog)
  ✅ Metadata source of truth
  ✅ Query and governance target

Data movement disclosure (for regulated environments): Raw files are NOT bulk-copied to S3. However, during AI enrichment, selected file content is temporarily read via the S3 Access Point and sent to Amazon Bedrock APIs for classification/embedding. Per AWS Bedrock data protection policy, model providers have no access to customer prompts or completions. Extracted/redacted metadata and embeddings are written to S3 Tables, OpenSearch, and optionally to Snowflake or Databricks depending on the activation path. Define your data flow boundary documentation before regulated-workload deployment.

The Problem: Most Enterprise Unstructured Data is Difficult to Discover and Govern

Most organizations store terabytes of unstructured data — PDFs, images, CAD files, sensor logs — on network-attached storage. This data is:

Undiscoverable: "Where is that invoice from last quarter?" requires manual searching or asking colleagues
Governed at the file-system layer, but not classified or searchable from analytics and AI workflows
Audit trails may exist at the file-system layer, but they are often not unified with analytics and AI query activity

Think of this as unstructured-data modernization: inventory first, classify selectively, govern metadata, and activate only what is needed — without bulk-copying the raw files.

Business Outcomes (Beyond Technical Metrics)

This pattern is not only about faster file search. It is about:

Reducing dataset discovery lead time for AI projects (days → hours)
Improving PII visibility across the organization (unknown → 95%+ coverage target)
Lowering duplicate storage cost ($230-256/month eliminated for 10TB)
Creating governed metadata products for analytics and AI teams
Enabling AI-readiness without raw-data copy or migration
Activating governed metadata in Snowflake AI Data Cloud for Cortex Search, semantic Q&A, executive dashboards, and business-facing file discovery

The traditional solution? Copy everything to S3 and build a catalog. But at 10TB, that's ~$230-256/month just for the copy — plus sync pipelines, duplicate governance, and data drift.

The Solution: Hot Metadata × Cold Data

What if we could catalog every file without moving it?

┌─────────────────────────────────────────────────────────┐
│  HOT: Metadata (Apache Iceberg on S3 Tables)            │
│  • File path, type, size, timestamps                    │
│  • AI classification + confidence score                 │
│  • Vector embedding (1024-dim, similarity search)       │
│  • PII detection flag                                   │
│  • Cost: ~$5-15/month for 100K files                    │
└────────────────────────┬────────────────────────────────┘
                         │ file_path reference
┌────────────────────────▼────────────────────────────────┐
│  COLD: Actual Files (FSx for ONTAP)                     │
│  • PDF, images, CAD, video, audio, logs                 │
│  • Deduplication (50-70% storage savings typical*)      │
│  • NFS/SMB (existing workflows) + S3 AP (AI/analytics)  │
│  • No bulk raw-data copy required                       │
└─────────────────────────────────────────────────────────┘

Key insight: Keep the data where it is. Move only the metadata into a queryable format.

Architecture

FSx for ONTAP ──S3 Access Point──→ AI Enrichment (Bedrock)
       │                                    │
       │                                    ▼
       │                          S3 Tables (Iceberg)
       │                                    │
       │                                    ▼
       │                          ┌──────────────────┐
       │                          │ Query Engines    │
       │                          │ • Athena (SQL)   │
       │                          │ • OpenSearch     │
       │                          │   (vector kNN)   │
       │                          │ • Lake Formation │
       │                          │   (governance)   │
       │                          └──────────────────┘
       │
       └──NFS/SMB──→ Existing applications (unchanged)

Observability (production add-on):

       ┌──────────────────────────────────────┐
       │  • CloudWatch Metrics + Alarms       │
       │  • CloudWatch Logs (Lambda/SQS)      │
       │  • CloudTrail (governance audit)     │
       │  • OpenSearch Dashboards (search UX) │
       │  • FSx metrics (throughput, IOPS,    │
       │    latency, capacity pool reads)     │
       └──────────────────────────────────────┘

Components:

Component	Role	Cost
FSx for ONTAP S3 Access Point	Read files for AI processing (no copy)	Included with FSx
S3 Tables	AWS managed Apache Iceberg table service (auto-compaction, REST endpoint)	~$5/month metadata
Bedrock Claude Vision	Image classification	~$0.01/file in this demo
Titan Embeddings V2	1024-dim vectors for similarity search	$0.00002/1K input tokens
OpenSearch Serverless NextGen	kNN vector search (scale-to-zero)	$0 idle compute when inactive
Lake Formation	Metadata access governance	No additional Lake Formation charge

S3 Tables Iceberg REST endpoint: https://s3tables.<region>.amazonaws.com/iceberg
Check S3 Tables availability for regional support before deployment.

Deduplication ratio is a general ONTAP range. Actual savings depend on data characteristics and were not measured in this PoC.

PoC Results (Verified 2026-05-31)

We built and verified this end-to-end in a single day. Here's what we measured:

S3 Tables Access Paths: Which Endpoint Should You Use?

Access path	Best for	Governance path	Verified
S3 Tables Iceberg REST (`s3tables.<region>.amazonaws.com/iceberg`)	Direct Iceberg client / simple PoC	IAM + S3 Tables permissions	✅
AWS Glue Iceberg REST (`glue.<region>.amazonaws.com/iceberg`)	Production analytics integration	IAM + Lake Formation	✅
Athena via Glue federated catalog	SQL analytics	Lake Formation + Athena	✅
PyIceberg local client	Lightweight validation	IAM/LF depending on endpoint	✅

For production workloads with centralized governance, the AWS Glue Iceberg REST endpoint is recommended over the S3 Tables direct endpoint. See AWS docs.

Catalog authority rule: S3 Tables + Glue is the authoritative catalog for this metadata table in this PoC. Other engines should consume the table through the authoritative catalog or a controlled metadata activation path. Do not configure multiple writable catalogs for the same Iceberg table — dual-write causes split-brain and potential data corruption.

Athena Iceberg behavior depends on Athena engine version, Iceberg version, Glue/Lake Formation integration, and table maintenance state. Validate DDL/DML requirements separately before using this as a write-heavy production catalog.

Verification details are recorded in evidence-record.yaml and cross-platform-compatibility.yaml.

Before vs After

Metric	Before	After	Improvement
File discovery time	Minutes-hours	< 2 seconds	100x+ at scale
AI classification	Manual	Automatic (6 sec/file)	Fully automated
Storage cost (10TB)	~$250/month (S3 copy)	$5-15/month (metadata only)	95% reduction
Metadata query governance	Not applicable	100% in this PoC	Complete for metadata queries
Idle compute/search cost	N/A	Near $0 when inactive	Persistent metadata/logs may still incur small charges

Search Time Scaling (Measured + Projected)

Files	ListObjectsV2	Athena SQL	Speedup
40	892 ms	3.0 sec	0.3x
1,000	22.3 sec	1.8 sec	12x
10,000	3.7 min	1.8 sec	124x
100,000	37.2 min	1.8 sec	1,239x
1,000,000	371.7 min	1.8 sec	12,389x

At 40 files, ListObjectsV2 is faster — Athena has cold start overhead. Athena query time does not scale linearly with the number of files on FSx because it queries the Iceberg metadata table instead of listing the raw file namespace. In this controlled demo, the query stayed around ~1.8 seconds for projected file counts, but production latency depends on Iceberg metadata size, manifest count, predicate selectivity, Athena cold start, and table maintenance state.

Projection method: ListObjectsV2 latency was extrapolated linearly from the measured 40-file scan. This is intentionally conservative for demonstrating namespace-scan behavior, but it is not a service benchmark.

The 42-Second Demo

Our complete demo runs all 8 steps in 42 seconds:

Step 1: Before/After search comparison     ✅ (ListObjectsV2 vs Athena)
Step 2: Infrastructure deploy              ✅ (CloudFormation, skippable)
Step 3: Metadata scan (40 files)           ✅ (3 seconds)
Step 4: AI enrichment (Bedrock Vision)     ✅ (invoice → 0.95 confidence)
Step 5: Athena query + Time Travel         ✅ (< 2 seconds)
Step 6: Vector similarity search           ✅ (kNN score 0.67)
Step 7: PII detection + anonymization      ✅ (7/7 entities, all redacted)
Step 8: Cost & ROI analysis                ✅ ($0.07 total demo cost)

Total demo cost: $0.07. After the demo, the compute/search components can scale to zero. If you retain S3 Tables metadata, logs, or audit trails, small storage/logging charges may still apply.

AI Classification Results

File	Classification	Confidence
invoice_sample.png	Invoice	0.95
product_inspection.png	Pie Chart	1.0
sensor_dashboard.png	IoT Sensor Dashboard	0.9

In this demo, Bedrock Claude Vision classified sample images at roughly $0.01/file with sub-10-second latency. Production cost and latency depend on image size, prompt length, model version, and retry behavior.

Vector Similarity Search

Query: "find invoice or payment documents"
→ invoice_sample.png (score: 0.6749)

OpenSearch Serverless with scale-to-zero capability (GA May 2026) provides kNN search — no minimum cost when idle. Cold start is ~10-30 seconds, warm queries are ~54ms.

Verified in this PoC environment on 2026-05-31. Check the latest OpenSearch Serverless documentation and regional availability before deployment.

Governance: Lake Formation Access Control

Step 1: Authorized query    → ✅ SUCCEEDED (3 rows)
Step 2: Revoke SELECT       → 🔒 BLOCKED (access denied)
Step 3: Restore SELECT      → ✅ SUCCEEDED
Step 4: CloudTrail audit    → All queries logged with user identity

Metadata queries are governed and audited. Raw file access remains governed separately by FSx file-system permissions, S3 Access Point policies, and application access paths.

Cost Analysis

This Demo

Component	Cost
Bedrock AI (5 files)	$0.05
OpenSearch (~6 min)	$0.024
Lambda + Athena	$0.001
Total	$0.07

Projected Monthly (10TB, 100K files, 1000 changes/day)

Component	Monthly
S3 Tables (metadata)	$5
Lambda (sync + AI)	$36
Bedrock (AI enrichment)	$30
OpenSearch (business hours)	$42
SQS + misc	$1
Total	$114/month
S3 copy eliminated	-$230-256/month

Net effect: The AI-powered catalog costs less than the S3 copy it eliminates.

Without AI enrichment (metadata scan + Athena only): ~$42/month. AI processing is optional and can be enabled per-file-type.

S3 Standard pricing: us-east-1 $0.023/GB, ap-northeast-1 $0.025/GB. Verified 2026-06-01 via AWS Pricing API.

For reproducibility, see: evidence-record.yaml, cost-assumptions.yaml, comprehensive-test-results.yaml

Known Limitations (Honest Assessment)

Limitation	Impact	Workaround
Databricks SQL Warehouse `CREATE CONNECTION TYPE iceberg_rest` to S3 Tables REST failed in this validation (2026-05-31)	SQL Warehouse direct path unavailable in tested method	Retested 2026-06-09; still blocked in tested UC path. Use curated metadata sync to UC Delta as practical workaround; support case submitted.
Databricks Spark cluster: UC blocks external catalog registration (2026-06-01)	Cannot use spark.conf.set or cluster config for external Iceberg catalogs	UC Foreign Catalog tested 2026-06-09 — External Location validation fails against S3 Tables internal bucket. Sync metadata to UC Delta table instead.
Databricks Delta Sharing: cannot bypass S3 AP session policy (2026-06-01)	Sharing server uses same UC credentials	DataSync → S3 → UC → Delta Sharing works for copied data; validate target table format and catalog support separately
Databricks NFS mount: cannot register as UC External Volume (2026-06-01)	NFS/FUSE paths not supported for UC Volumes	DataSync → S3 → UC External Location; internal feature request exists
Snowflake External Iceberg Table with S3 Tables REST endpoint was not a supported catalog type in this validation (2026-05-31)	Direct S3 Tables REST path unavailable in tested method	✅ Resolved (2026-06-05): Use Glue REST + explicit `ACCESS_DELEGATION_MODE = VENDED_CREDENTIALS`. Schema must have no default External Volume. AWS prerequisite: `register-resource --with-federation`. Lake Formation column-level filtering NOT enforced via this path.
LF column exclusion grant failed in tested S3 Tables federated catalog path	Can't hide specific columns via tested grant pattern	Athena Views; track AWS support status
At 40 files, ListObjectsV2 is faster than Athena	Architecture value is at scale (100K+)	Expected — Athena has cold start overhead

Naming note: Use lowercase table, namespace, and column names for S3 Tables integrated with AWS analytics services. Mixed-case names may not be visible to Athena / Glue / Lake Formation. See S3 Tables naming rules.

Performance Boundaries Not Yet Validated

This PoC validates the architecture shape, not production scale limits. The following require separate testing:

FSx throughput impact under concurrent NFS/SMB/S3 access
S3 Access Point metadata operation impact under large namespace scans
S3 API request concurrency vs FSx provisioned throughput capacity
Impact of scan jobs on production SMB/NFS latency
ListObjectsV2 pagination behavior at 1M+ files
Lambda concurrency and S3 AP request throttling
Iceberg manifest growth and compaction behavior
Athena query latency with high snapshot counts
OpenSearch indexing throughput during bulk backfill
File size distribution and small-file amplification effects
Cold vs warm namespace access behavior (capacity pool reads during backfill)

ONTAP Object Model Mapping

ONTAP / FSx object	Role in this pattern
FSx file system	Performance / HA boundary
SVM	Protocol and administrative boundary
Volume	Catalog scope and S3 Access Point attachment target
Junction path / SMB share	Existing application namespace
S3 Access Point	S3 API boundary for AI/analytics (with associated file-system identity)
Iceberg table	Metadata catalog, not raw data store

Each S3 Access Point has an associated OntapFileSystemIdentity (UNIX UID/GID or Windows domain user) that authorizes all file access through that AP. IAM policy is evaluated first, then ONTAP file-system permissions. See security/s3-access-point-identity-matrix.yaml.

Iceberg Table Maintenance Plan

For production, define:

Snapshot retention period and table maintenance behavior — verify S3 Tables service-managed policies and any configurable retention settings
Manifest rewrite cadence (if metadata table grows large)
Orphan file cleanup policy
Deduplication view or materialized latest-record table
Time travel retention policy
Athena engine version and Iceberg version compatibility
Append-only dedup query as default named query for analysts

For operational steps, see ops/iceberg-maintenance-runbook.md. For details on Iceberg spec vs S3 Tables service behavior, see docs/standards-vs-service-behavior.md.

Iceberg does not enforce primary-key uniqueness in this PoC. Consumers should query curated latest-record views instead of the append-only base table. See ops/athena-named-queries/latest_records.sql in the repo.

Apache Iceberg is the open table format. Amazon S3 Tables is an AWS managed table bucket service that uses Apache Iceberg. Some operational behavior, endpoint support, and governance integration are AWS service-specific and should be validated separately from the Iceberg specification itself.

File Identity Strategy

file_id method	Best for	Tradeoff
`hash(volume_id + normalized_path)`	General purpose	Rename = new file_id
`hash(volume_id + file_handle/inode)`	Rename tracking	Requires inode access
Content hash (SHA-256)	Immutable documents	Expensive for large files
`path + last_modified + size`	Lightweight PoC only	Fragile under overwrites

Production should define how rename, overwrite, delete, and permission changes are represented in the metadata table.

Recommended production columns: source_system_id, volume_id, normalized_path, path_hash, content_hash, scan_run_id, change_type (created / modified / deleted / renamed / permission_changed).

For FlexClone-based dev/test datasets, decide whether cloned files should retain lineage to source files. If lineage matters, store clone_parent_volume_id, clone_parent_snapshot_id, and catalog_environment (prod / dev / test / dr). See dr/snapmirror-catalog-rebinding.md for DR failover considerations.

For manufacturing and engineering workloads, see schema/extensions/manufacturing_metadata.yaml for domain-specific metadata fields such as part number, revision, plant, machine, and inspection lot.

Multi-Tenant Deployment Considerations

If this pattern is provided by a partner or platform team to multiple business units or customers, define the isolation boundary explicitly.

Isolation model	Recommended when	Tradeoff
Table bucket per tenant	Strong isolation required	Higher operational overhead
Namespace per tenant	Balanced isolation and operations	Shared table bucket governance required
tenant_id column in one table	Internal multi-BU catalog	Requires strict LF-Tags / row filters
OpenSearch index per tenant	Search isolation required	More index management
Shared OpenSearch index + tenant filter	Lower cost	Must enforce filter in every query path

For partner-led deployments, document tenant onboarding automation, offboarding deletion/retention policy, per-tenant cost allocation tags, and audit evidence location.

Business KPI Mapping

Business problem	Baseline metric	Target metric	How this PoC measures it
Employees cannot find documents	Average search time	< 10 sec	Search latency + result relevance
Manual classification is slow	Files classified/day/person	10x improvement	AI enrichment throughput
Sensitive files are unknown	% files classified for PII	95%+ coverage target	PII scan completion rate
Duplicate S3 copy is costly	Monthly duplicate storage cost	Reduce by 50%+	Metadata-only architecture cost
AI projects lack data inventory	Dataset discovery lead time	Days → hours	Catalog completeness
Business users need governed discovery	% searchable assets in BI/AI tools	80%+ of approved metadata visible	Expose curated metadata views to Athena, Databricks, Snowflake, or BI tools

Try It Yourself

FSx for ONTAP prerequisites:

SVM and volume selected as catalog scope
S3 Access Point attached to the target volume
Associated UNIX or Windows identity documented
NFS/SMB production workload impact reviewed
CloudWatch metrics dashboard enabled

# Clone the repo
git clone https://github.com/Yoshiki0705/fsxn-lakehouse-integrations.git
cd fsxn-lakehouse-integrations/integrations/iceberg-metadata-catalog

# Install dependencies
pip install -r requirements.txt

# Run the demo (requires FSx for ONTAP with S3 Access Point)
cd demo/scripts
./run-demo.sh --ap-alias <your-ap-alias-ext-s3alias>

Don't have FSx for ONTAP? You can still explore the architecture:

What's Next

This is Part 1 of a 3-part series:

Part 1 (this article): Architecture & PoC Results
Part 2: AI Enrichment Pipeline — Bedrock Vision + Titan Embeddings + OpenSearch NextGen
Part 3: Governance & Cross-Platform Access — Lake Formation, PII Anonymization, Databricks/Snowflake Integration

Key Takeaways

Don't copy data to make it searchable — catalog the metadata instead. Apache Iceberg + S3 Tables gives you a managed metadata layer with time travel.
Selective AI enrichment plus scale-to-zero search can keep PoC and low-traffic environments cost-efficient — compute/search components idle near $0; persistent metadata and logs may incur small charges.
42 seconds, $0.07 — that's the barrier to entry for an AI-powered data catalog on your existing NAS storage.
Start small, grow incrementally — from metadata-only scan (Level 1) to full business workflow integration (Level 5). See the Production Maturity Model for the progression path.

All code and documentation is available at github.com/Yoshiki0705/fsxn-lakehouse-integrations. Feedback welcome via GitHub Issues.

28 Industry Reference Patterns with FSx for ONTAP S3 Access Points — Phase 15

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 07 Jun 2026 03:00:23 +0000

TL;DR

Phase 15 expands the pattern library from 17 to 28 industry-specific use cases, providing reference implementations across major AWS Industry verticals where FSx for ONTAP file processing is relevant. Each new pattern includes a CloudFormation template, Step Functions workflow, Python Lambda functions, 8-language documentation, and property-based tests. Combined with 6 FlexCache/FlexClone patterns and 1 SAP/ERP pattern, the repository now offers 35 deployable reference patterns for enterprise file processing on FSx for ONTAP.

The SAP/ERP pattern focuses on controlled document/report processing around ERP-adjacent file exports (IDoc, spool), not direct transactional SAP data manipulation.

Important: These are reference implementations with production-readiness guidance, not fully certified production systems. Customers must validate against their own regulatory, security, and operational requirements before production use.

For S3 standard bucket users: This library is not a replacement for S3 data lake patterns. It is a file-data integration pattern for customers who want to process FSx ONTAP-resident data through S3-compatible APIs while preserving NAS access paths. See docs/s3-bucket-user-guide.md for a detailed comparison.

Serverless boundary: Compute (Lambda), orchestration (Step Functions), eventing (EventBridge), and AI services (Bedrock, Textract, Rekognition) are serverless/managed. FSx for ONTAP is a fully managed file system with provisioned capacity and operational considerations — it is not scale-to-zero storage. This is a serverless processing pattern over existing enterprise file data, not a pure serverless storage pattern.

When NOT to use this: If your workload is already object-native, does not require NFS/SMB coexistence, and can use standard S3 data lake patterns — prefer S3-native serverless architecture.

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

Why 28 Use Cases?

AWS organizes customers into 22 industry verticals. When we mapped our existing 17 patterns against these verticals, several gaps stood out:

Telecommunications — No CDR/network log processing pattern
Advertising & Marketing — No creative asset management
Travel & Hospitality — No document processing for reservations
Agriculture & Food — No traceability or crop monitoring
Sustainability/ESG — No ESG metrics extraction
Nonprofit — No grant management automation
Utilities — No drone/SCADA-based asset inspection
Real Estate — No portfolio analysis
HR — No resume screening (with PII protection)
Chemicals — No SDS/lab notebook processing
Transportation (railway) — No deterioration detection

Phase 15 fills all of these, covering 19 of 22 AWS Industry verticals (remaining 3 — Consumer Packaged Goods, Mining, Software/Internet — have limited file-processing relevance for this pattern type). Combined with 11 Japan-market focus areas (all covered), the repository addresses the vast majority of enterprise file processing scenarios.

The 11 New Patterns

P0: Foundation Patterns

UC	Industry	Key AWS Services	Differentiator
UC18	Telecom	Athena, Bedrock	CDR/syslog anomaly detection with 7-day baseline
UC19	AdTech	Rekognition, Textract, Bedrock	Brand compliance scoring + moderation

P1: Document Intelligence

UC	Industry	Key AWS Services	Differentiator
UC20	Travel	Textract, Comprehend, Rekognition	Multilingual reservation extraction + facility inspection
UC21	Agriculture	Rekognition, Textract, Bedrock	GeoTIFF crop analysis + lot traceability
UC22	Transportation	Rekognition, Textract, Bedrock	Safety-critical escalation trigger + deterioration trends

P2: Specialized Processing

UC	Industry	Key AWS Services	Differentiator
UC23	Sustainability	Textract, Bedrock	ESG metric extraction + GRI/TCFD/ISSB mapping
UC24	Nonprofit	Textract, Comprehend, Bedrock	Grant application + outcome matching
UC25	Utilities	Rekognition, Bedrock, Athena	Drone + SCADA + thermal tri-modal inspection
UC26	Real Estate	Rekognition, Textract, Bedrock	Property analysis + lease extraction + PII flagging
UC27	HR	Textract, Comprehend, Bedrock	Recruiting document triage with PII protection
UC28	Chemicals	Textract, Rekognition, Bedrock	SDS hazard extraction + GHS compliance + lab notebook

Architecture: One Pattern, Many Industries

Architecture Classification

Layer	Classification
Workflow orchestration	Serverless (Step Functions)
Compute	Serverless (Lambda)
Eventing / scheduling	Serverless (EventBridge)
AI/ML services	Managed service consumption (Bedrock, Textract, Rekognition, Comprehend)
File storage	Managed/provisioned (FSx for ONTAP)
Operations model	Hybrid: serverless processing + managed file storage

Lambda concurrency must be bounded by FSx ONTAP S3 AP throughput behavior. Do not treat Lambda concurrency as the only scaling control.

Common Workflow Pattern

Every pattern follows the same proven architecture:

EventBridge Scheduler
       │
       ▼
Step Functions State Machine
       │
       ├── Discovery Lambda (VPC-internal, ONTAP API)
       │        │
       │        ▼
       │   S3 Access Point (list + classify files)
       │
       ├── Processing Map (parallel, Retry + Catch)
       │        │
       │        ▼
       │   [Rekognition | Textract | Comprehend | Bedrock | Athena]
       │
       └── Report Lambda
                │
                ├── Output → S3 AP (FSx ONTAP volume)
                └── SNS Notification

What changes per industry:

File prefixes and extensions (Discovery Lambda configuration)
AI/ML service selection (Rekognition for images, Textract for documents, Bedrock for reasoning)
Domain-specific schemas (ESG metrics, GHS sections, CDR fields)
Review thresholds (60% escalation trigger for safety-critical defects, 80% standard detection, 90% auto-approve threshold)
Compliance requirements (PII filtering for HR, data classification labels, audit trails)

For production deployments, validate how S3 AP-generated output files appear from existing NFS/SMB clients, including ownership, permissions, naming convention, and Snapshot/SnapMirror policy impact. See ONTAP Integration Notes.

Shared Modules: The Productivity Multiplier

The 11 new patterns reuse the same shared/ modules that power the original 17:

Module	Purpose	Used By
`s3ap_helper.py`	S3 Access Point abstraction (alias + ARN)	All 28 UCs
`exceptions.py`	Domain exceptions + error handler decorator	All 28 UCs
`observability.py`	EMF metrics + structured logging	All 28 UCs
`human_review.py`	Confidence-based review decisions	UC22, UC25, UC27
`data_classification.py`	Output data labeling (INTERNAL/CUI/etc.)	UC23, UC24, UC27, UC28
`schemas/events.py`	TypedDict event/response schemas	All 28 UCs

Adding a new industry pattern takes 2-3 hours (not days) because the infrastructure is already solved. A new pattern is considered field-shareable only after DemoMode execution, cfn-lint validation, unit/property tests, success metrics, data classification, and human review thresholds are documented.

Key Design Decisions for New Patterns

1. Safety-Critical Thresholds (UC22)

Railway infrastructure inspection cannot accept false negatives. We use a dual-threshold approach:

STANDARD_THRESHOLD = 80       # General defect detection trigger
SAFETY_CRITICAL_THRESHOLD = 60  # Bridges, signaling, rail joints — lower to reduce false negatives
HUMAN_REVIEW_THRESHOLD = 90    # Auto-approve only above this

Critical design intent: 60% is NOT an auto-approval threshold. It is an escalation trigger — any signal above 60% for safety-critical categories triggers mandatory human review. The system is designed to surface potential defects for expert evaluation, not to automate safety decisions. All detections below 90% confidence require human review regardless of category.

2. PII-First Design (UC27)

Recruiting document triage handles personal data. The pattern enforces:

No PII in logs — structured logging strips personal identifiers
Protected characteristic exclusion — Bedrock prompt explicitly excludes age, gender, ethnicity
Encrypted output — all results written with data classification labels
Audit trail — every scoring decision is logged with justification (not content)

Regulatory notice: UC27 is a document triage and summarization workflow, not an automated hiring decision system. Final hiring decisions must remain with qualified human reviewers. Customers must validate against local labor law, privacy regulations (GDPR, APPI, CCPA), and anti-discrimination requirements before any use in recruitment processes. Output must not include ranking by protected attributes, and explanation fields must cite only job-relevant qualifications.

3. Tri-Modal Inspection (UC25)

Utilities asset inspection combines three data modalities in a single workflow:

Visual (drone images) → Rekognition defect detection
Temporal (SCADA logs) → Athena time-series anomaly detection
Thermal (FLIR images) → Hot-spot classification (≥10°C differential)

The Step Functions workflow processes all three in parallel Map states, then merges results for a unified maintenance priority report.

4. ESG Framework Mapping (UC23)

Sustainability reporting requires mapping extracted metrics to multiple frameworks simultaneously:

GRI (Global Reporting Initiative)
TCFD (Task Force on Climate-related Financial Disclosures)
ISSB (International Sustainability Standards Board)

Bedrock performs the mapping using structured prompts with framework-specific indicator definitions.

Testing: 1,499+ Tests Across 28 Patterns

Each new pattern includes:

Unit tests with moto for AWS service mocking
Property-based tests (Hypothesis) for invariant verification
cfn-lint validation for all CloudFormation templates
ruff linting for Python code quality

Notable property tests:

UC22: severity_level ∈ {critical, major, minor, observation} for all inputs
UC25: SCADA thresholds within physical bounds (voltage ±5%, frequency ±0.5 Hz)
UC27: No protected characteristics appear in any output field
UC28: All GHS mandatory sections validated for completeness

Responsible AI and Human Review

These patterns are reference workflows, not fully automated decision systems. For regulated or safety-critical domains (healthcare, finance, transportation, HR, public sector), customers must define:

Human review thresholds — what confidence level requires expert validation
Appeal/escalation process — how incorrect classifications are corrected
Audit trail requirements — what decisions need immutable logging
Data retention policy — how long intermediate results are kept
Model evaluation criteria — accuracy, hallucination rate, bias testing on domain data
Local regulatory review — jurisdiction-specific compliance (FISC, HIPAA, GDPR, NARA, labor law)

The shared/human_review.py module provides a framework for confidence-based routing, but threshold values and escalation procedures must be defined by domain experts, not by template defaults.

Customers are responsible for validating these workflows against their own policies, risk classification, and regulatory obligations before production use.

Pattern Selection Guide

Customer Situation	Recommended Starting Pattern
FSx ONTAP already used for shared files	UC by industry + DemoMode=false
No FSx ONTAP yet, wants to evaluate workflow	Any UC + DemoMode=true
Document-heavy workload (PDF, contracts, reports)	UC20 / UC23 / UC24 / UC26 / UC27 / UC28
Image-heavy inspection workload	UC19 / UC21 / UC22 / UC25
Logs / time-series / analytics workload	UC18 / UC25 (SCADA)
Safety-critical review required	UC22 / UC25 with human_review module
PII-sensitive workflow	UC27 / UC26 with data_classification module
ESG / sustainability reporting	UC23 with framework mapping
Greenfield object-native workload (no NAS)	Prefer standard S3 + serverless-native architecture

DemoMode to Production Path

Area	DemoMode (evaluation)	Production (FSx ONTAP)
Input source	Regular S3 bucket	FSx ONTAP S3 Access Point
Permissions	S3 IAM only	IAM + S3 AP policy + ONTAP file identity
Network	Public AWS service path	Internet-origin or VPC-origin design decision (NetworkOrigin is immutable after creation)
Data	Sample/synthetic data	Customer-controlled NAS data
Governance	Demo labels only	Data classification + lineage + retention
Cost	~$0.10/execution	+ FSx ONTAP infrastructure (~$194/month base)
Code compatibility	Standard S3 bucket semantics	Validate the FSx ONTAP S3 AP API subset and unsupported S3 bucket features before production
Access point lifecycle	N/A	NetworkOrigin changes require creating a new S3 AP

Cost varies by region, deployment type, SSD capacity, throughput capacity, backups, and data transfer; the figure above is a baseline estimate for Single-AZ / 128 MBps / 1 TB SSD. This cost model is not scale-to-zero storage. Use this pattern when the value of processing existing NAS data in place outweighs the baseline FSx ONTAP infrastructure cost.

Deployment: 30 Minutes to First Result

Every pattern includes a samconfig.toml.example and step-by-step deployment:

# 1. Copy and configure
cp samconfig.toml.example samconfig.toml
# Edit: S3AccessPointAlias, VpcId, SubnetIds, etc.

# 2. Deploy
sam build && sam deploy --guided

# 3. Execute
aws stepfunctions start-execution \
  --state-machine-arn <ARN from outputs>

# 4. Verify
aws stepfunctions describe-execution --execution-arn <ARN>
# Status: SUCCEEDED

For patterns without FSx for ONTAP, DemoMode=true uses a regular S3 bucket — ideal for evaluation without infrastructure commitment.

Benchmark Insight: Small Files Don't Need More Throughput

During Phase 15 deployment verification, we ran benchmarks at 128/256/512 MBps throughput capacity with a 202-byte JSON manifest:

Throughput	P50 @ conc=1	P50 @ conc=25	P50 @ conc=50
256 MBps	56.9 ms	60.3 ms	257.9 ms
512 MBps	59.8 ms	59.9 ms	246.1 ms

Conclusion: For metadata-heavy workloads (JSON manifests, small config files, document headers), throughput capacity increase has zero effect on latency. The bottleneck is connection overhead (TLS + S3 AP routing), not bandwidth. Save costs by staying at 128 MBps for these workloads.

Sizing reference from a specific test environment, not a service limit.

Documentation: 8 Languages × 28 Patterns

Every pattern includes documentation in:
🇯🇵 Japanese (primary) · 🇺🇸 English · 🇰🇷 Korean · 🇨🇳 Chinese (Simplified) · 🇹🇼 Chinese (Traditional) · 🇫🇷 French · 🇩🇪 German · 🇪🇸 Spanish

Each language includes:

README.md — Overview, deployment, success metrics
docs/architecture.md — Mermaid data flow diagram
docs/demo-guide.md — Step-by-step demo with verification checklist

Each UC README includes Success Metrics with Business Outcome, Technical KPI, Quality KPI, Cost KPI, and Go/No-Go criteria. This article summarizes the portfolio; detailed success criteria live with each pattern.

What Changed Since Phase 14

Metric	Phase 14	Phase 15	Delta
Use cases	17	28	+11
Total patterns	24	35	+11
Test count	~800	1,499+	+699
Industries covered	14/22	19/22	+5
Languages	8	8	—
Shared modules	8	11	+3
Documentation files	~400	~700	+300

Who Should Use Each New Pattern?

Recommended Starting Patterns

Start here if...	Pattern	Why
You want document intelligence	UC20 or UC26	Multilingual extraction + property/lease analysis
You want log analytics	UC18	CDR/syslog anomaly detection with baseline
You need PII-safe document triage	UC27	Protected characteristic exclusion built-in
You need inspection workflows	UC22 or UC25	Safety-critical escalation + tri-modal
You want ESG extraction	UC23	Multi-framework mapping (GRI/TCFD/ISSB)

Full Pattern List

If you are...	Start with...	Why
Telecom operator with CDR data	UC18	Anomaly detection across network logs
Ad agency managing creative assets	UC19	Automated brand compliance scoring
Hotel chain with inspection photos	UC20	Facility condition monitoring at scale
Agricultural cooperative	UC21	Crop health + traceability in one workflow
Railway/transit operator	UC22	Safety-critical deterioration detection
ESG reporting team	UC23	Multi-framework metric extraction
Grant-making foundation	UC24	Application processing + outcome matching
Power utility with drone programs	UC25	Tri-modal inspection (visual + SCADA + thermal)
Real estate portfolio manager	UC26	Property analysis + lease extraction
Recruiting team (APAC/EMEA)	UC27	PII-compliant recruiting document triage
Chemical manufacturer	UC28	SDS compliance + lab notebook digitization

What's Next

VPC-internal Lambda benchmark — True VPC path performance (eliminates Internet latency)
FPolicy TCP-level Replay Storm — Real ONTAP event replay (requires ECS rebuild)
Cross-repository integration — Link patterns to fsxn-lakehouse-integrations for analytics pipelines
Glue Data Catalog integration — Schema versioning and data quality checks for output datasets
Community contributions — Pattern template for community-submitted industry use cases

Resolved from Phase 14: FlexCache × S3 AP integration confirmed as not currently supported by AWS — tracked in Field Feedback Log. FC1 Recovery Metrics depend on this feature. Both remain pending AWS feature availability.

Ownership Model

Layer	Recommended Owner
Shared modules (`shared/`)	Platform / DevOps team
UC business logic (`functions/`)	Application / data team
FSx ONTAP and S3 AP infrastructure	Storage / platform team
IAM, data classification, encryption	Security team
Success metrics and Go/No-Go	Business owner
Regulatory compliance mapping	GRC / legal team

Compliance Positioning

These templates do not certify compliance with any specific regulation. They provide implementation hooks for audit logging, retention, classification, and human review that customers can map to their regulatory controls. Each organization must independently validate compliance with applicable regulations (FISC, HIPAA, GDPR, NARA, local labor law, etc.).

NetApp / ONTAP Operational Notes

For production deployments on FSx for ONTAP, review the ONTAP-specific guidance in docs/ontap-integration-notes.md, including:

SVM / volume / protocol scope assumptions
NFS/SMB visibility of S3 AP-generated outputs (file ownership = AP file system identity)
IAM + S3 AP policy + ONTAP file identity behavior, separate from NFS export policy evaluation
Snapshot / SnapMirror / retention impact on output artifacts
Scheduler vs FPolicy trigger mode selection
FlexCache / FlexClone combination patterns per UC
NetApp support diagnostic bundle
OT/manufacturing safety caveat

FlexCache/FlexClone note: UC × FC combination patterns describe adjacent architecture patterns. Validate current AWS/FSx feature support before assuming direct S3 AP access to cached or cloned paths.

Benchmark scope: Results are from Single-AZ, First-generation FSx ONTAP. Validate separately for Multi-AZ or newer generation file systems.

Regulated research workflows (UC7, UC28, FC5): Capture input dataset version, model/prompt version, reviewer action, and output checksum as lineage metadata. See shared/lineage.py v2 fields.

Stats

New patterns: 11 (UC18-UC28)
New Lambda functions: 44 (4 per pattern average)
New tests: 699
New documentation files: ~300 (across 8 languages)
New shared modules: data_classification.py, human_review.py, schemas/events.py
Deployment verified: All 28 UCs achieved SUCCEEDED status in ap-northeast-1
Benchmark runs: 2 additional (256/512 MBps small-file comparison)
Cost: ~$10 total for deployment verification (Lambda + Step Functions + Bedrock Nova Lite)

Try It Today

git clone https://github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns.git
cd FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

# Quick test (no AWS account needed)
make test-quick

# Deploy any pattern with DemoMode (no FSx ONTAP needed)
cd telecom-network-analytics
cp samconfig.toml.example samconfig.toml
sam build && sam deploy --guided

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

Full series: FSx for ONTAP S3 Access Points on DEV.to

Evidence Expansion, Presigned URL Discovery, and Operational Surprises — FSx for ONTAP S3 Access Points, Phase 14

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 07 Jun 2026 02:53:08 +0000

TL;DR

Phase 14 shifts from building patterns to hardening the evidence base. After publishing Phase 13's field-ready reference architecture, I focused on post-publication refinement: Partner/SI delivery assets, benchmark methodology standardization, S3 AP compatibility clarification (Presigned URLs work despite documentation), and an unexpected operational discovery — S3 Access Points become unavailable during FSx throughput capacity changes.

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

Why Phase 14?

Phase 13 delivered the field-ready baseline. Phase 14 answers the question: "Now that the patterns exist, how do we make them easier to evaluate, adopt, and operate?"

The work falls into four categories:

Partner/SI delivery acceleration — one-pager, improved PoC templates, FC1-FC6 conversation starters
Benchmark methodology — standardized run IDs, hypothesis-driven testing, Range GET plans
Compatibility clarification — Presigned URL behavior confirmed with AWS Support
Operational discovery — S3 AP unavailability during throughput capacity changes

1. Partner/SI One-Pager: What / When / How / Where

Partners and SIs told us the existing 7-step delivery checklist was comprehensive but too long for a first conversation. Phase 14 adds a single-page overview that answers four questions:

Section	Content
What	28 UCs + 6 FC patterns, CloudFormation templates, 4-level maturity model
When	Customer has FSx for ONTAP + needs serverless file processing + permission-aware access
How	Identify UC → Deploy template → Measure baseline → Evaluate Go/No-Go
Where	Links to Success Metrics, Governance, Production Readiness, Benchmarks

Available in both Japanese and English.

FC1-FC6 Recommended First Questions

Each FlexCache/FlexClone pattern now has a recommended first conversation question — the question a Partner/SI should ask to determine if the pattern is relevant:

Pattern	First Question
FC1 (Anycast/DR)	"What is your current read latency from remote sites, and what target would justify a caching layer?"
FC2 (Render)	"How many concurrent render jobs share the same source data, and what is the job lifecycle?"
FC3 (RAG)	"Which file shares contain the knowledge base, and do access permissions need to be preserved in RAG results?"
FC4 (CAE)	"What is the typical solver output size and how quickly must results be available for post-processing?"
FC5 (Life Sciences)	"How do you currently share research datasets between teams while maintaining data governance?"
FC6 (Gaming)	"What is your current build pipeline duration and which asset validation steps are bottlenecks?"

2. Presigned URLs: "Not Supported" but Working

⚠️ Production Warning: AWS Support explicitly states that operations marked "Not supported" should NOT be relied upon for production workloads, even when they return success today. The behavior may change without deprecation notice, return inconsistent results across regions, or stop working after service updates. Design alternatives for any workflow that requires presigned URL access to FSx for ONTAP S3 Access Points.

The Discovery

The FSx for ONTAP S3 AP compatibility table lists Presign — Not supported. However, testing showed presigned URLs for GetObject work successfully.

AWS Support Clarification

After raising this with AWS Support, the explanation was clear:

Presigning is client-side only — aws s3 presign computes a SigV4 signature locally. No network request is made.
The presigned URL executes a standard GetObject — signature is in query parameters instead of the Authorization header.
Since GetObject is Supported, presigned URLs cannot be blocked without breaking GetObject itself.
The documentation likely intended to indicate that presigned URL workflows are not officially tested.

Production Guidance

Feature	Status	Guidance
GetObject, PutObject, ListObjectsV2	Supported	Build on freely
Conditional writes (If-None-Match)	Blocked	Returns NotImplemented
Presigned URLs	Not supported (doc)	Works but do not rely on for production

AWS Support has escalated documentation clarification to the FSx for ONTAP service team. The distinction between "Not supported + hard-blocked" (returns error) and "Not supported + may incidentally work" (no guarantees) is being reviewed.

Readers should verify the latest AWS documentation before relying on this behavior, as the status may change.

Alternatives to Presigned URLs

If you need time-limited or delegated file access without relying on unsupported behavior:

API Gateway + Lambda proxy with IAM or JWT authorization
CloudFront signed URLs backed by a controlled Lambda@Edge origin
Temporary STS credentials with scoped IAM permissions (time-limited, per-object or prefix)
Application-level download broker with audit logging and access revocation

For a broader comparison with standard S3 bucket semantics, see the S3 Bucket User Guide.

3. Benchmark Methodology: Hypothesis-Driven Testing

Why 1769 MB Lambda Memory?

Lambda memory directly controls CPU and network bandwidth allocation. At 1769 MB, Lambda receives exactly 1 vCPU equivalent, providing consistent and reproducible network throughput for benchmark measurements. Lower memory settings would introduce variable network bandwidth as a confounding factor.

Benchmark Run ID Convention

Every benchmark run now follows a standardized format:

s3ap-bench-{YYYY-MM-DD}-{seq}

With mandatory fixed conditions:

Region: ap-northeast-1
Lambda memory: 1769 MB (1 vCPU)
Lambda architecture: arm64
FSx Throughput Capacity: [128 / 256 / 512] MBps
Iterations per data point: 50
Statistics: p50, p90, p95, p99, min, max
Concurrent NFS/SMB workload: [None / Light / Production-level]

Hypothesis: Throughput Capacity vs Practical Concurrency Point

Based on 128 MBps observations where I observed concurrency=10 as the practical upper limit in this specific test environment (1 MB objects, single Lambda invocation pattern, no concurrent NFS/SMB workload), I hypothesize:

Practical concurrency point may shift with FSx throughput capacity increase.

FSx Capacity	Predicted Practical Concurrency	Rationale
128 MBps	10 (observed)	Baseline — P99 exceeded 420 ms
256 MBps	~15-25	Sub-linear scaling is plausible due to ONTAP WAFL overhead and TCP connection management
512 MBps	~25-45	Step-function behavior possible if a different bottleneck emerges

Note: Linear scaling (2x capacity = 2x concurrency) is one possible outcome, but sub-linear or step-function behavior is equally plausible. The actual relationship depends on ONTAP data plane queuing, TCP connection overhead, and whether the bottleneck shifts from throughput to IOPS or latency at higher capacities.

Initial verification was blocked by the S3 AP issue described below. Results were published after recovery on 2026-05-25.

Update (2026-05-25): S3 AP recovered. Benchmarks completed. See Section 7 for results and hypothesis verification.

4. Operational Discovery: S3 AP Unavailability During Throughput Changes

What Happened

While preparing to run 256 MBps benchmarks, I changed the FSx throughput capacity from 128 to 256 MBps. After the change completed successfully:

All S3 Access Points on the file system returned ServiceUnavailable
All SVMs were affected (not just one)
Reverting to 128 MBps did not immediately restore S3 AP access
The file system itself remained AVAILABLE throughout

Timeline

Time	Event
T+0	`update-file-system` ThroughputCapacity 128 → 256
T+25 min	Change completed (256 MBps confirmed)
T+25+ min	All S3 APs return ServiceUnavailable
T+40 min	Revert initiated (256 → 128)
T+65 min	Revert completed, S3 APs still unavailable

Impact and Recommendation

This is now tracked as an AWS Support case. Key takeaways:

Plan throughput capacity changes during maintenance windows. S3 AP workloads may be disrupted for an extended period.

Unlike standard S3 buckets, FSx for ONTAP S3 AP availability can be affected by FSx file system operational changes such as throughput capacity updates.

Important context: AWS documentation states that NFS/SMB access typically remains available during throughput capacity changes. The S3 AP disruption I observed appears to be specific to the S3 Access Point data plane — not the file system's NFS/SMB data LIFs. This distinction matters for environments that use both protocols.

For regulated environments (FISC, healthcare, government): Throughput capacity changes must be included in change management procedures. If S3 AP-based workloads have SLA requirements, the change should be approved through the organization's change advisory board with documented rollback procedures.

This finding has been added to the Production Readiness document as a Level 3+ operational consideration.

5. DEV.to Series Cleanup

Phase 14 also cleaned up the article series:

Update Notes added to Phase 1, 9, 10, 12 articles linking to Phase 13
Permission-Aware RAG articles moved to a separate series (was incorrectly mixed into FSx S3AP series)
Series now has 14 articles in "FSx for ONTAP S3 Access Points" (down from 16 after RAG separation)

Blockers and Current Status

Item	Blocker	Resolution
~~256/512 MBps benchmark~~	~~S3 AP ServiceUnavailable~~	✅ Resolved 2026-05-25 — Results below
FC1 Recovery Metrics	FlexCache × S3 AP integration	Pending AWS feature availability
~~Hypothesis verification~~	~~Depends on benchmark~~	✅ Partially confirmed — see results

6. Benchmark Results: 128 / 256 / 512 MBps Concurrency Comparison

S3 AP ServiceUnavailable was resolved on 2026-05-25. We immediately executed the planned benchmark across all three throughput tiers.

Test Environment

Note on methodology divergence: The benchmark methodology (Section 3) defines 1769 MB Lambda + 50 iterations as the standard. The Internet tests below used a macOS client + 10 iterations per concurrency level due to the initial exploratory nature of these measurements. The Lambda egress test (Section 8) follows the 1769 MB / 50 iteration standard. Treat Internet results as directional sizing guidance, not as statistically rigorous benchmarks.

Parameter	Value
Region	ap-northeast-1 (Tokyo)
FSx for ONTAP	Single-AZ, First-generation
S3 AP	NetworkOrigin=Internet
Client	macOS, boto3, Python 3.9 (public Internet)
Object sizes	1 KB, 100 KB, 1 MB
Concurrency	1, 5, 10, 20, 50
Iterations	10 per concurrency level (exploratory)

Key Results: 1 MB GetObject P99

Concurrency	128 MBps	256 MBps	512 MBps
1	76 ms	93 ms	96 ms
5	160 ms	175 ms	308 ms
10	239 ms	236 ms	229 ms
20	981 ms	481 ms	738 ms
50	—	850 ms	4,495 ms

Analysis

P50 (median) is largely independent of throughput capacity — Internet baseline latency (connection + TLS) dominates
P99 (tail latency) shows the difference — 128→256 MBps improved P99 by 51% at concurrency=20
512 MBps shows no improvement over 256 MBps via Internet — client-side bandwidth (~100 Mbps) becomes the bottleneck
Hypothesis partially confirmed: Practical concurrency point does shift with throughput capacity, but the relationship is non-linear and bounded by client bandwidth in Internet-origin tests

Sizing Guidance

Workload	128 MBps	256 MBps	512 MBps
Small files (< 10 KB)	MaxConcurrency=20	50	50
Medium files (100 KB)	10	20	50
Large files (1 MB+)	5	10	20

These are sizing references from a specific test environment, not service limits. A VPC-internal Lambda + VPC-origin S3 AP path is expected to reduce public Internet overhead, but remains untested and must be validated separately. Always validate with your own workload profile.

What This Means for Production

For PoC (128 MBps): Keep Step Functions Map state MaxConcurrency ≤ 5 for 1 MB+ files
For Production (256+ MBps): MaxConcurrency=10-20 is safe for most workloads
For VPC-internal Lambda (untested): Expected to further reduce latency by eliminating public Internet path, but requires VPC-origin S3 AP (not yet measured)
Throughput capacity changes: Plan during maintenance windows (S3 AP disruption risk confirmed)
Small files (< 1 KB): Throughput capacity increase has no effect — bottleneck is connection overhead, not bandwidth. Save costs by staying at 128 MBps for metadata-heavy workloads

7. Lambda Egress Path Benchmark: Reducing Connection Overhead

Terminology clarification: This test uses a VPC-external Lambda (no VpcConfig) accessing an Internet-origin S3 AP. The Lambda egress path goes through AWS-managed networking, which is faster than public Internet but is NOT a VPC-internal path. A true VPC-internal test would require a VPC-origin S3 AP + VPC-internal Lambda — that remains untested.

Path	Status	What it measures
Public Internet client → Internet-origin S3 AP	✅ Measured (Section 7)	End-user/CI baseline
VPC-external Lambda → Internet-origin S3 AP	✅ Measured (this section)	AWS-managed Lambda egress, NOT VPC-private
VPC-internal Lambda → VPC-origin S3 AP	❌ Not yet measured	True private path (requires new AP)

We deployed a benchmark Lambda (1769 MB, ARM64, no VpcConfig) to measure GetObject latency via AWS-managed Lambda egress.

Lambda Egress vs Internet: 1 MB GetObject P50

Concurrency	Internet P50	Lambda P50	Improvement
1	68 ms	62 ms	9%
5	117 ms	61 ms	48%
10	175 ms	73 ms	58%
20	256 ms	122 ms	52%
50	N/A	128 ms	—

Key Findings

P50 dramatically improved at concurrency > 1: Lambda egress eliminates public Internet TCP connection overhead
P99 remains high (~1s): Even from Lambda, concurrency=20 shows P99 of 1,318 ms — this is the S3 AP data plane's internal queuing
concurrency=50 P50 is only 128 ms: Lambda threads are efficient against S3 AP
The bottleneck is the FSx for ONTAP S3 AP data plane, not Lambda network bandwidth

Production Sizing (Lambda)

Workload	Recommended MaxConcurrency	Expected P50	Expected P99
Small files (1 KB)	50	~63 ms	~994 ms
Medium files (100 KB)	20	~79 ms	~1,044 ms
Large files (1 MB)	10	~73 ms	~928 ms

Set Lambda timeout to 30s+ and use Step Functions Retry to handle P99 spikes. These results are from VPC-external Lambda (AWS-managed egress), not true VPC-internal path.

8. SQS Replay Storm Simulation: Zero Message Loss Under Load

Scope clarification: This test validates the downstream SQS ingestion and Lambda consumer drain path under replay-like burst conditions. It does NOT validate ONTAP Persistent Store buffering or FPolicy TCP-level server reconnection replay. Those require a live FPolicy server environment (future work).

We simulated FPolicy server reconnection by injecting 1,000 and 10,000 events directly into SQS, mimicking the burst that occurs when a Persistent Store replays buffered events after server reconnection.

Results

Scenario	Events	Loss Rate	Throughput	Batch P99
5 min downtime	1,000	0%	188 eps	177 ms
30 min downtime	10,000	0%	464 eps	79 ms
Consumer drain	1,000	0%	341 msgs/sec	85 ms

SLO Validation

SLO Metric	Threshold	Observed	Status
Event loss rate	< 0.1%	0%	✅
Injection throughput	> 100 eps	464 eps	✅
Consumer drain rate	> injection rate	341 > 188	✅
Batch latency P99	< 200 ms	79 ms	✅
DLQ messages	0	0	✅

Implications

30-min downtime accumulates ~835K events at 464 eps. With Lambda auto-scaling (10 consumers), drain completes in < 5 minutes.
Persistent Store sizing estimate: Based on simulated event payload size, 10K events ≈ 5 MB. Real ONTAP Persistent Store sizing must be validated with live FPolicy replay.
No backpressure issues: SQS Standard queue handles burst without message loss.

9. Operational Runbook: S3 AP Disruption Response

When S3 AP becomes unavailable (e.g., during throughput capacity changes):

Check S3 AP health: ListObjectsV2 / GetObject against S3 AP alias
Check NFS/SMB separately: mount + read test (may still be functional)
Check FSx file system status: describe-file-systems → Lifecycle
Check CloudWatch alarms: Lambda errors, Step Functions failures
Pause ingestion: Disable EventBridge Schedules for affected UC pipelines
Wait and retry: S3 AP recovery may take 15-60 min after throughput changes
Escalate: If unavailable > 60 min, contact AWS Support with file system ID

See Incident Response Playbook for full procedures.

What's Next (Phase 15 candidates)

Update: Phase 15 expanded the pattern library from 17 to 28 industry-specific use cases. Items 1-2 below remain pending AWS feature availability. Items 3-4 are carried forward in Phase 15's What's Next.

FlexCache × S3 AP integration — pending AWS feature availability (not yet supported)
FC1 Recovery Metrics — route decision latency, cache health detection, failover timing (depends on #1)
Replay Storm with real FPolicy server — TCP-level replay characteristics (requires ECS re-deploy)
VPC-internal Lambda with VPC Origin S3 AP — true VPC-internal path (requires new AP with NetworkOrigin=VPC)

Multi-Account OAM validation completed 2026-05-25 — cross-account CloudWatch Metrics, Logs, and X-Ray Traces confirmed working.

Field feedback tracked: S3 AP disruption during throughput change, presigned URL documentation gap, VPC-origin benchmark gap, and FlexCache × S3 AP feature dependency. Details in docs/ontap-integration-notes.md.

Stats

Files changed: 200+ (documentation, translations, shared modules, templates)
New documents: Partner/SI one-pager (JP/EN/KO/ZH-CN), cost calculator, customization guide, incident response playbook, demo mode guide, comparison alternatives, PoC Go/No-Go template
New shared modules: data_classification.py, human_review.py, schemas/events.py
Benchmark runs: 7 (128/256/512 MBps Internet × 2 file sizes + Lambda egress + SQS replay storm simulation)
Templates fixed: 5 (cfn-lint errors: RecursiveDeleteOption, SNSPublishMessagePolicy, Handler path)
Translations added: 20 files (FC1-FC6 ko/zh-CN + FC1/FC3 full 8-lang)
samconfig.toml.example: 24 patterns
Output JSON samples: 24 patterns
DEV.to articles updated: 6 (4 Update Notes + 2 Series changes)
AWS Support cases: 1 resolved (S3 AP ServiceUnavailable — throughput change related)
Operational discoveries: 1 (throughput change → S3 AP disruption, now resolved)
Cost savings: ~$346/month (v4-test-demo + FPolicy server + VPC Endpoints + EC2 停止)
SQS Replay Storm Simulation: 10,000 events, 0% loss in downstream SQS/consumer path

Who Should Care About Phase 14?

Partners and SIs get a one-pager for first conversations and recommended questions for each FC pattern
Operations teams learn that throughput capacity changes can disrupt S3 AP access
Architects get standardized benchmark methodology with hypothesis-driven testing
Developers get Presigned URL clarification — works but don't depend on it
Standard S3 bucket users learn where FSx for ONTAP S3 AP differs from S3 bucket semantics, especially presigned URLs, availability, and operational dependencies
Serverless-first teams learn where the serverless processing plane ends and FSx for ONTAP operational considerations begin

What You Can Do Today

Phase 14 delivers immediately usable assets:

Use the Partner/SI one-pager for your next customer conversation about FSx for ONTAP + serverless
Check the S3AP Compatibility Notes for the latest Presigned URL and troubleshooting guidance
Plan throughput changes carefully — add S3 AP health checks to your maintenance runbook
Use the Sizing Guidance tables (Sections 7-8) to set MaxConcurrency for your workload
Review the S3 Bucket User Guide before porting existing S3 applications to FSx for ONTAP S3 AP
Review the ONTAP Integration Notes before attaching S3 AP workflows to production SVMs and volumes

Repository: github.com/Yoshiki0705/FSx-for-ONTAP-S3AccessPoints-Serverless-Patterns

Full series: FSx for ONTAP S3 Access Points on DEV.to

9 Services, One Architecture: What We Learned Shipping FSx for ONTAP Logs to Every Major Observability Platform

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 31 May 2026 01:42:23 +0000

TL;DR

We built and E2E-verified serverless integrations shipping FSx for ONTAP audit logs to 9 observability platforms — all from the same architecture:

For decision makers: 90% cost reduction vs EC2-based collectors ($66/month → $5-8/month), 9 vendor choices instead of 1, 30-minute deploy instead of hours, zero operational burden. Four vendors offer permanent free tiers covering most FSx for ONTAP deployments (New Relic 100 GB, Grafana Cloud 50 GB, Honeycomb 20M events, Sumo Logic 500 MB/day).

                    ┌─────────────────────────────────────────────┐
                    │         One Architecture, 9 Backends        │
                    ├─────────────────────────────────────────────┤
                    │                                             │
                    │  FSx for ONTAP ──→ S3 Access Point          │
                    │       │                                     │
                    │       ▼                                     │
                    │  EventBridge Scheduler (5 min)              │
                    │       │                                     │
                    │       ▼                                     │
                    │  Lambda (vendor-specific handler)           │
                    │       │                                     │
                    │       ├──→ Datadog (Logs API v2)            │
                    │       ├──→ New Relic (Log API v1)           │
                    │       ├──→ Splunk (HEC)                     │
                    │       ├──→ Grafana Cloud (OTLP Gateway)     │
                    │       ├──→ Elastic (Bulk API)               │
                    │       ├──→ Dynatrace (Log Ingest v2)        │
                    │       ├──→ Sumo Logic (HTTP Source)         │
                    │       ├──→ Honeycomb (Events Batch API)     │
                    │       └──→ OTel Collector (OTLP/HTTP)       │
                    │                                             │
                    └─────────────────────────────────────────────┘

12 articles, 9 vendors, 3 event sources (audit logs, EMS webhooks, FPolicy), all CloudFormation-templated, all tested with real FSx for ONTAP data. This post distills what we learned.

This is Part 13 — the series finale — of Serverless Observability for FSx for ONTAP.

The Architecture That Survived 9 Integrations

After implementing 9 vendor integrations, the core pattern remained unchanged:

def lambda_handler(event, context):
    # 1. Get cached credentials (Secrets Manager + TTL, default 5 min)
    creds = auth.get()

    # 2. List new files since checkpoint (S3 AP + SSM)
    new_keys = list_new_keys(s3_ap_arn, prefix, checkpoint)

    # 3. Read, parse, format, ship per file (vendor-specific)
    #    (Simplified — actual implementation batches events across files
    #     and respects vendor-specific batch size limits)
    for key in new_keys:
        logs = read_and_parse(key)
        payload = format_for_vendor(logs)  # Only this changes per vendor

    # 4. Ship with retry (vendor API)
        ship_to_vendor(payload, creds)

    # 5. Advance checkpoint (only after confirmed delivery)
        update_checkpoint(key)

What changes per vendor: only the formatting and HTTP call (~50-100 lines). Everything else — S3 AP access, checkpoint management, DLQ handling, credential caching, retry logic — is shared.

Cross-Vendor Comparison: The Numbers

API Characteristics

Vendor	Endpoint	Auth Model	Max Batch	Success Code	Firehose
Datadog	Logs API v2	Header (`DD-API-KEY`)	5 MB / 1000 items	200	Yes
New Relic	Log API v1	Header (`Api-Key`)	1 MB	202	Yes
Splunk	HEC	Header (`Splunk <token>`)	No hard limit	200	Yes (built-in)
Grafana	OTLP Gateway	Basic Auth (base64)	~4 MB	200	No
Elastic	Bulk API	Header (`ApiKey <b64>`)	~10 MB	200	No
Dynatrace	Log Ingest v2	Header (`Api-Token`)	1 MB	204	Via ActiveGate
Sumo Logic	HTTP Source	URL-embedded token	1 MB	200	No
Honeycomb	Events Batch	Header (`x-honeycomb-team`)	5 MB (impl: 100/batch)	200	No
OTel Collector	OTLP/HTTP	Configurable	Configurable	200	No

Cost at 10 GB/month

Vendor	Vendor Cost	AWS Infra	Total	Free Tier
Sumo Logic	$0	~$5	~$5	500 MB/day
Honeycomb	$0	~$5	~$5	20M events/month
New Relic	$0	~$5	~$5	100 GB/month
Grafana Cloud	$0	~$5	~$5	50 GB logs/month
Datadog	~$15	~$5	~$20	Logs: 14-day trial only
Dynatrace	~$25	~$5	~$30	14-day trial
Elastic Cloud	~$95	~$5	~$100	14-day trial
Splunk Cloud	~$150+	~$5	~$155+	N/A

AWS infrastructure cost is consistent across all vendors (~$5/month for Lambda + EventBridge + Secrets Manager). The vendor platform cost is the differentiator.

Data Residency

Vendor	Tokyo (JP)	US	EU	Self-Hosted
Sumo Logic	Yes	Yes	Yes	No
Elastic	Yes	Yes	Yes	Yes
Dynatrace	Yes (region-specific)	Yes	Yes	Yes (Managed)
Datadog	No	Yes	Yes	No
New Relic	No (July 2026 planned)	Yes	Yes	No
Grafana Cloud	Dedicated only	Yes	Yes	No (Alloy self-hosted)
Splunk	No	Yes	Yes	Yes
Honeycomb	No	Yes	No	No

Governance note: This table provides technical awareness for vendor selection. Grafana Cloud offers Tokyo region on Dedicated tier (not Free/Pro). Data residency alone does not constitute regulatory compliance. Evaluate your specific requirements (APPI, GDPR, FISC, ISMAP) with your compliance team. See the Retention Policy Matrix for regulation-to-vendor mapping.

Unique Strengths

Vendor	Best For
Datadog	Full-stack APM correlation, broadest feature set
New Relic	Generous free tier (100 GB), NRQL power
Splunk	Existing Splunk shops, SPL expertise, Firehose native
Grafana Cloud	OTLP-native, LogQL, open-source ecosystem
Elastic	Data sovereignty (self-hosted), ECS/SIEM, Kibana
Dynatrace	Davis AI root cause analysis, APM correlation
Sumo Logic	JP region data residency, generous free tier
Honeycomb	High-cardinality analysis (BubbleUp, Heatmaps)
OTel Collector	Multi-backend, vendor portability, redaction

Note on Grafana ecosystem: Grafana Alloy (formerly Grafana Agent) provides a Grafana-native alternative to the OpenTelemetry Collector with the same OTLP compatibility. Grafana Cloud's OTLP Gateway is available on all tiers including Free (US/EU regions only). For Tokyo data residency, Grafana Cloud Dedicated is required.

7 Patterns That Survived All 9 Integrations

1. Polling > Event-Driven (for FSx for ONTAP S3 AP)

FSx for ONTAP S3 Access Points don't support S3 Event Notifications. We evaluated CloudTrail data events as an alternative — however, CloudTrail data events for FSx for ONTAP S3 AP access are not consistently available across all configurations. The 5-minute EventBridge Scheduler poll is simpler, cheaper, and sufficient for audit log use cases where near-real-time (not real-time) delivery is acceptable.

2. Checkpoint-After-Delivery

Never advance the checkpoint before confirming vendor delivery. This single rule prevents data loss across all failure modes:

# CORRECT: checkpoint after confirmed delivery
ship_to_vendor(payload)  # Raises on failure
update_checkpoint(key)   # Only reached on success

# WRONG: checkpoint before delivery
update_checkpoint(key)   # What if ship_to_vendor fails next?
ship_to_vendor(payload)  # Data loss if this fails

3. Credential Caching with Reload-on-401

Every vendor integration uses the same SecretBackedAuth pattern: cache credentials at cold start, reload on TTL expiry or 401/403. This handles credential rotation without Lambda redeployment.

4. Reserved Concurrency = 1

The audit poller must not run concurrently (checkpoint race condition). ReservedConcurrentExecutions: 1 is the simplest guard. For higher throughput, move to DynamoDB-based per-object locking.

5. DLQ for Every Async Path

Every template includes a KMS-encrypted DLQ. In 9 integrations, the DLQ caught: vendor outages, credential expiry, malformed files, and Lambda timeouts. Without it, these failures would be silent data loss.

6. Vendor-Specific Batch Limits Matter

The biggest implementation difference across vendors is batch size handling:

Vendor	Limit	Lambda Behavior
Honeycomb	100 events	Split into chunks of 100
Dynatrace / Sumo Logic	1 MB	Measure payload size, split at boundary
Datadog	5 MB / 1000 items	Dual limit check
Elastic	~10 MB	Rarely hit with audit logs

7. OTLP as the Universal Format

If you're unsure which vendor you'll use long-term, start with OTLP. The OTel Collector integration (Part 5) proved that a single Lambda producing OTLP can feed Datadog, Grafana, and Honeycomb simultaneously — with zero code changes when adding or removing backends.

Beyond multi-backend delivery, the OTel Collector provides:

Enrichment: Resource detection, Kubernetes attributes, custom metadata injection
Sampling: Tail-based sampling for high-volume environments
Redaction: PII field removal/masking before data leaves your account (see PII Redaction Cookbook)
Format conversion: OTLP ↔ vendor-native format translation

Verified version: All OTel Collector configurations in this series were tested with OpenTelemetry Collector Contrib v0.152.0. OTel Collector has frequent releases with potential breaking changes — pin your version in production and test before upgrading.

What We'd Do Differently

Start with OTel Collector for Multi-Vendor Evaluation

If evaluating multiple vendors, deploy the OTel Collector path first. It lets you send the same data to 2-3 vendors simultaneously for comparison, without deploying separate Lambda stacks per vendor.

Define SLOs Before Building

We defined Pipeline SLOs after building all 9 integrations. In hindsight, defining "< 10 min delivery latency" and "< 0.01% data loss" upfront would have guided design decisions earlier (e.g., checkpoint granularity, retry policy).

Data Classification First

Audit logs contain PII (usernames, file paths). We documented this in the Data Classification Guide after implementation. For regulated environments, classify fields before choosing a vendor — it may eliminate options that don't support your data residency requirements.

Production Readiness Framework

After 9 integrations, we formalized a 4-level production readiness model:

Level	What	Go/No-Go to Next
Level 1: Quickstart	Audit poller + DLQ	Logs arrive, checkpoint advances, DLQ empty 24h
Level 2: Operational PoC	+ Dashboard + alerts	SLOs met 7 days, security review done
Level 3: Production	+ DynamoDB ledger + poison-pill	SLOs met 30 days, compliance pack
Level 4: Enterprise	+ OTel Collector + redaction	Multi-backend, PII redaction, DR tested

Most PoCs should target Level 2. Production deployments need Level 3. Enterprise pipelines with compliance requirements need Level 4.

Recommended transition timeline:

Level 1 → Level 2: ~1 week (add dashboards, define SLOs, validate 7-day stability)
Level 2 → Level 3: ~2-4 weeks (deploy DynamoDB ledger, implement poison-pill handling, complete security review)
Level 3 → Level 4: ~1-2 months (deploy OTel Collector, implement PII redaction, test DR failover, complete compliance evidence pack)

Full criteria: Pipeline SLO Definitions

Vendor Selection Decision Tree

Start
  |
  +-- Need JP data residency?
  |   +-- Yes -> Sumo Logic (JP) or Elastic (self-hosted in Tokyo VPC)
  |   +-- No  |
  |           v
  +-- Need self-hosted (air-gapped)?
  |   +-- Yes -> Elastic or Splunk
  |   +-- No  |
  |           v
  +-- Already have an observability platform?
  |   +-- Yes -> Use that vendor (all 9 are supported)
  |   +-- No  |
  |           v
  +-- Budget constraint (free tier needed)?
  |   +-- Yes -> Sumo Logic (500 MB/day) or Honeycomb (20M events) or New Relic (100 GB)
  |   +-- No  |
  |           v
  +-- Need AI-powered root cause analysis?
  |   +-- Yes -> Dynatrace (Davis AI)
  |   +-- No  |
  |           v
  +-- Need high-cardinality analysis?
  |   +-- Yes -> Honeycomb (BubbleUp)
  |   +-- No  |
  |           v
  +-- Need multi-backend / vendor portability?
  |   +-- Yes -> OTel Collector
  |   +-- No  |
  |           v
  +-- Default -> Datadog (broadest) or Grafana (OTLP-native, open ecosystem)

The FSx for ONTAP S3 AP Constraint That Shaped Everything

The single most impactful technical constraint: FSx for ONTAP S3 Access Points do not support S3 Event Notifications.

This one fact drove:

EventBridge Scheduler polling pattern (not event-driven)
SSM Parameter Store checkpointing (track what's been processed)
Reserved concurrency = 1 (prevent checkpoint races)
Safety threshold (stop before Lambda timeout)
MAX_KEYS_PER_RUN (bound processing per invocation)

If FSx for ONTAP S3 APs add event notification support in the future, the architecture could simplify significantly. As of May 2026, this feature is not supported, and the polling pattern is battle-tested across 9 vendors.

Cost Reality: EC2 vs Serverless

The original motivation: replace the EC2-based Splunk pattern (2x EC2 instances) with serverless.

Metric	EC2 Pattern	Serverless Pattern
Monthly AWS cost	~$66	~$5-8
OS patching	Required	None
Scaling	Manual	Automatic
Vendor support	Splunk only	9 vendors
Deploy time	Hours	30 minutes
Recovery from failure	Manual restart	Automatic (DLQ + retry)

90% cost reduction with zero operational burden. The serverless pattern wins on every dimension except one: real-time latency (EC2 syslog can be sub-second; our poller is 5-minute intervals). For audit logs, 5 minutes is acceptable. For real-time needs, use the FPolicy path (< 30 seconds).

What's Next

This series covered the foundation. The project continues with:

Phase 3 (delivered): Multi-account deployment (AWS Organizations + StackSets)
Phase 3 (delivered): DynamoDB object ledger for per-object processing state
Phase 3 (delivered): SQS buffering pattern for backpressure handling
Phase 3 (delivered): Cross-region DR with Active-Passive failover
Phase 3 (delivered): OTel Collector PII redaction cookbook (7 recipes for APPI/GDPR)
Phase 4: Terraform module equivalents
Phase 4: CDK construct library

See the full ROADMAP.

Resources

GitHub: github.com/Yoshiki0705/fsxn-observability-integrations
Pipeline SLO: docs/en/pipeline-slo.md
Data Classification: docs/en/data-classification.md
S3 AP Throughput Benchmark: docs/en/s3ap-throughput-benchmark.md
Vendor Comparison: docs/en/vendor-comparison.md
Partner FAQ: docs/en/partner-faq.md
Workshop Guide: docs/en/workshop-hands-on-half-day.md
Compliance Evidence Pack: docs/en/compliance-evidence-pack.md

Series Navigation

Part 1: Why Your FSx for ONTAP Audit Logs Deserve Better Than EC2
Part 2: Shipping FSx for ONTAP Logs to Datadog — The Serverless Way
Part 3: Event-Driven Ransomware Detection with ONTAP ARP + Datadog
Part 4: FPolicy File Activity Pipeline — ONTAP to Datadog via ECS Fargate
Part 5: Escape Vendor Lock-in: Multi-Backend Log Delivery with OTel Collector for FSx for ONTAP.
Part 6: Direct-to-Grafana: Shipping FSx for ONTAP Logs to Grafana Cloud Loki via OTLP Gateway
Part 7: Ship FSx for ONTAP Audit Logs to New Relic via Serverless Lambda Pipeline
Part 8: EC2 to Serverless: Modernizing FSx for ONTAP Splunk Integration
Part 9: Data Sovereignty: FSx for ONTAP Logs in Your VPC with Elastic
Part 10: High-Cardinality File Access Analysis with Honeycomb + OTel
Part 11: AI-Powered Root Cause: Correlating File Access with APM via Dynatrace
Part 12: FSx for ONTAP Audit Logs with Data Residency in your region with Sumo Logic
Part 13: 9 Vendors, One Architecture (this post)

Thank you for following this series. If you've deployed any of these integrations, I'd love to hear about your experience — drop a comment or open a GitHub issue.

GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

FSx for ONTAP Audit Logs with Data Residency in your region with Sumo Logic

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 31 May 2026 00:37:19 +0000

TL;DR

We built a serverless Lambda pipeline that ships FSx for ONTAP audit logs to Sumo Logic's JP (Tokyo) region deployment. For Japanese enterprises with data residency requirements under APPI (Act on the Protection of Personal Information), this means audit logs never leave Japan.

FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda → Sumo Logic HTTP Source (JP)
                                                                         │
                                                                         ▼
                                                              ┌───────────────────┐
                                                              │ Sumo Logic JP     │
                                                              │ (Tokyo)           │
                                                              │                   │
                                                              │ • 500 MB/day FREE │
                                                              │ • Data stays in   │
                                                              │   Japan           │
                                                              │ • 7-day retention │
                                                              │   (free tier)     │
                                                              └───────────────────┘

Key advantages:

500 MB/day free tier (~15 GB/month) — covers most FSx for ONTAP deployments at zero vendor cost
JP region deployment — data residency in Tokyo
Simplest auth model — URL-embedded token, no header management
30-minute end-to-end — HTTP Source URL is the only credential needed

Verified on Sumo Logic JP region. Logs searchable via _sourceCategory=aws/fsxn/audit.

This is Part 12 of the Serverless Observability for FSx for ONTAP series.

Why Sumo Logic for Japanese Enterprises?

For organizations operating under Japanese data protection regulations, the choice of observability platform often comes down to one question: where does the data physically reside?

Requirement	Sumo Logic JP	Other Options
Data residency in Japan	✅ Tokyo deployment	Varies by vendor
APPI compliance consideration	✅ Data stays in JP	May require cross-border assessment
Free tier for validation	✅ 500 MB/day	Most offer 14-day trials only
No agent installation	✅ HTTP Source (agentless)	Some require collectors

Sumo Logic's JP deployment (service.jp.sumologic.com) processes and stores all data within Japan, making it a straightforward choice for organizations that need to demonstrate data residency compliance.

Compliance note: This integration provides a technical path for data residency. Evaluate your specific regulatory requirements with your compliance team — data residency alone does not constitute full regulatory compliance.

Architecture

┌─────────────────────────────────────────────────────────┐
│ Event Sources                                           │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  EventBridge Scheduler                                  │
│  rate(5 minutes) ──→ Lambda                             │
│                       │ lists new files via             │
│                       │ S3 Access Point                 │
│                       │ (checkpoint in SSM)             │
│                       ▼                                 │
│           Sumo Logic HTTP Source                        │
│           (URL-embedded auth)                           │
│                       │                                 │
│  EMS Webhook          │                                 │
│  ──→ API GW ──→ Lambda ─────────────┤                   │
│     (ems_handler)                   │                   │
│                                     ▼                   │
│  FPolicy                       Sumo Logic               │
│  ──→ ECS Fargate ──→ SQS      (Log Search,              │
│  ──→ Bridge Lambda              Dashboards,             │
│  ──→ EventBridge                Alerts)                 │
│  ──→ Lambda (fpolicy_handler) ──────────────────────────┤
└─────────────────────────────────────────────────────────┘

The Simplest Auth Model

Sumo Logic's HTTP Source embeds the authentication token directly in the URL. No separate auth headers, no API key management, no token rotation complexity:

https://collectors.jp.sumologic.com/receiver/v1/http/<TOKEN>

Lambda just POSTs JSON to this URL. That's it. The metadata (source category, name, host) is sent via X-Sumo-* headers:

headers = {
    "Content-Type": "application/json",
    "X-Sumo-Category": "aws/fsxn/audit",
    "X-Sumo-Name": "fsxn-ontap-audit",
    "X-Sumo-Host": "fsxn-ontap"
}

⚠️ Security: The HTTP Source URL contains the auth token. Store it in Secrets Manager, never log it, and never expose it in environment variables or source control.

Quick Start (30 Minutes)

1. Create Sumo Logic Account (JP Region)

Sign up at service.jp.sumologic.com
Select APAC: Tokyo (JP) as your deployment region
Free tier: 500 MB/day, 7-day retention, full feature access for 30 days

2. Create Hosted Collector + HTTP Source

Go to Manage Data → Collection → Add Collector
Select Hosted Collector (name: fsxn-audit-collector)
Add HTTP Logs & Metrics Source:
- Name: fsxn-ontap-audit
- Source Category: aws/fsxn/audit
- Timestamp Parsing: Auto-detect
Copy the generated HTTP Source URL

3. Store HTTP Source URL

aws secretsmanager create-secret \
  --name "sumo-logic/fsxn-http-source" \
  --secret-string '{"url":"https://collectors.jp.sumologic.com/receiver/v1/http/<YOUR_TOKEN>"}' \
  --region ap-northeast-1

4. Deploy CloudFormation Stack

aws cloudformation deploy \
  --template-file integrations/sumo-logic/template.yaml \
  --stack-name fsxn-sumo-logic-integration \
  --parameter-overrides \
    S3AccessPointArn=arn:aws:s3:ap-northeast-1:123456789012:accesspoint/fsxn-audit-ap \
    SumoLogicHttpSourceSecretArn=arn:aws:secretsmanager:ap-northeast-1:123456789012:secret:sumo-logic/fsxn-http-source-XXXXXX \
    S3BucketName=my-fsxn-audit-bucket \
  --capabilities CAPABILITY_NAMED_IAM \
  --region ap-northeast-1

5. Verify in Sumo Logic

Run this search query:

_sourceCategory=aws/fsxn/audit

Note: First-time indexing on a new JP region account takes ~10 minutes. Subsequent ingestion is near-instant.

Query Examples

Sumo Logic uses a pipe-based query language:

Basic Investigation

-- All failed access attempts with user and path
_sourceCategory=aws/fsxn/audit
| json "Result", "UserName", "ObjectName"
| where Result = "Failure"
| count by UserName, ObjectName
| sort by _count desc

-- Top operations by volume
_sourceCategory=aws/fsxn/audit
| json "Operation"
| count by Operation
| sort by _count desc

-- Access pattern timeline (5-minute buckets)
_sourceCategory=aws/fsxn/audit
| json "Operation", "UserName"
| timeslice 5m
| count by _timeslice, Operation

Security Investigation

-- After-hours access (outside 9am-6pm JST)
_sourceCategory=aws/fsxn/audit
| json "UserName", "Operation", "ObjectName"
| where _messagetime < today() + 9h OR _messagetime > today() + 18h
| count by UserName, Operation

-- Bulk delete detection (potential data exfiltration)
_sourceCategory=aws/fsxn/audit
| json "Operation", "UserName"
| where Operation = "Delete"
| timeslice 5m
| count by _timeslice, UserName
| where _count > 50

-- Sensitive path access
_sourceCategory=aws/fsxn/audit
| json "ObjectName", "UserName", "Result"
| where ObjectName matches "*confidential*" OR ObjectName matches "*restricted*"
| count by UserName, Result

Operational Monitoring

-- Log volume trend (capacity planning)
_sourceCategory=aws/fsxn/audit
| timeslice 1h
| count by _timeslice
| outlier _count

-- SVM activity comparison
_sourceCategory=aws/fsxn/audit
| json "SVMName"
| timeslice 15m
| count by _timeslice, SVMName

Sumo Logic Metadata Headers

Lambda sends structured metadata with each request:

Header	Value	Purpose
`X-Sumo-Category`	`aws/fsxn/audit`	Primary search dimension
`X-Sumo-Name`	`fsxn-ontap-audit`	Source identification
`X-Sumo-Host`	`fsxn-ontap`	Host-level grouping

These headers enable efficient searching without parsing the log body:

-- Search by metadata (fast, no JSON parsing)
_sourceCategory=aws/fsxn/audit _sourceHost=fsxn-ontap

Cost Analysis: The Free Tier Advantage

Sumo Logic's free tier is the most generous for small-to-medium FSx for ONTAP deployments:

Monthly Log Volume	Daily Average	Sumo Logic Tier	Monthly Cost
1 GB	~33 MB/day	Free	$0
5 GB	~167 MB/day	Free	$0
10 GB	~333 MB/day	Free	$0
15 GB	~500 MB/day	Free (at limit)	$0
30 GB	~1 GB/day	Professional	~$108/month

Component	Monthly Cost (10 GB/month)
Lambda (5-min polling)	~$3
EventBridge Scheduler	~$1
Secrets Manager	~$1
Sumo Logic	$0 (within free tier)
Total	~$5

For most FSx for ONTAP deployments generating < 15 GB/month of audit logs, the total cost is just the AWS infrastructure (~$5/month). The observability platform itself is free.

Sumo Logic Deployment Regions

Region	URL	Data Residency
JP (Tokyo)	`service.jp.sumologic.com`	Japan
US1	`service.sumologic.com`	US
US2	`service.us2.sumologic.com`	US
EU (Ireland)	`service.eu.sumologic.com`	EU
AU (Sydney)	`service.au.sumologic.com`	Australia
IN (Mumbai)	`service.in.sumologic.com`	India
CA (Montreal)	`service.ca.sumologic.com`	Canada

Select the deployment matching your data residency requirements at account creation time. Region cannot be changed after account creation.

Gotchas & Lessons Learned

#	Discovery	Impact
1	JP region new accounts have ~10 minute initial indexing lag	First search returns empty; wait and retry
2	Search queries use `_sourceCategory` (underscore prefix)	Common mistake: `sourceCategory` without underscore returns nothing
3	HTTP Source URL contains embedded auth token	Rotate by creating a new HTTP Source, updating the Secrets Manager secret, then deleting the old Source
4	Free tier has 7-day retention only	Sufficient for real-time monitoring; archive to S3 for long-term
5	No built-in Firehose support	Lambda direct delivery only
6	Max 1MB per request (newline-delimited JSON)	Lambda batches accordingly
7	Region is permanent — cannot migrate data between deployments	Choose JP at signup for data residency

Free Tier vs Professional

Feature	Free (500 MB/day)	Professional
Daily ingestion	500 MB	1+ GB (configurable)
Retention	7 days	30-365 days
Users	3	Unlimited
Alerts	✅	✅
Dashboards	✅	✅
API access	✅	✅
Data residency	✅ (region-specific)	✅

Tip: Enable Field Extraction Rules (FER): After first ingestion, create a FER for _sourceCategory=aws/fsxn/audit with "Auto-parse JSON" enabled. This automatically extracts all JSON fields (UserName, Operation, ObjectName, etc.) as searchable metadata — no manual field definition needed. Go to Manage Data → Logs → Field Extraction Rules → Add Rule.

For most PoC and small production deployments, the free tier is sufficient. Upgrade when you need longer retention or higher volume.

Data Residency & Classification

Sumo Logic JP deployment keeps all data in Japan, but audit logs still contain PII fields:

Field	Classification	Sumo Logic Handling
`UserName`	PII	Use RBAC to restrict search access; consider field extraction rules for masking
`ObjectName`	Sensitive	Path may reveal business context; restrict dashboard sharing
`ClientIP`	Internal	Generally acceptable

For APPI compliance considerations:

Data stays in JP region (no cross-border transfer for log data)
Configure retention policies matching your regulatory requirements (7 days free tier vs 30-365 days paid)
Implement Sumo Logic RBAC to restrict PII field access by role

See the Data Classification Guide for full field classification and regulatory mapping.

Production Readiness

This integration follows the project's Production Readiness Levels:

Level	What You Get	Go/No-Go to Next
Level 1 (this Quick Start)	Audit poller + DLQ	Logs arrive, checkpoint advances, DLQ empty 24h
Level 2	+ Sumo dashboards + alerts	SLOs met 7 days, security review done
Level 3	+ DynamoDB ledger + poison-pill	SLOs met 30 days, compliance pack
Level 4	+ OTel Collector + redaction	Multi-backend, PII redaction, DR tested

Full criteria: Pipeline SLO Definitions | DLQ Replay Runbook

Enterprise scale: For multi-account deployments across your Organization, see the StackSets deployment guide. For compliance evidence collection (ISMAP, FISC, SOC 2), see the Compliance Evidence Pack. For regulation-to-vendor retention mapping, see the Retention Policy Matrix.

CloudFormation Templates

Template	Purpose	Key Parameters
`template.yaml`	FSx audit log poller	S3AccessPointArn, SumoLogicHttpSourceSecretArn
`template-ems.yaml`	EMS webhook handler	SumoLogicHttpSourceSecretArn
`template-fpolicy.yaml`	FPolicy EventBridge handler	SumoLogicHttpSourceSecretArn, EventBusName

Resources

GitHub: integrations/sumo-logic/
Sumo Logic JP: service.jp.sumologic.com
HTTP Source Docs: Configure HTTP Source
Search Query Language: Search Operators
Series GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

Series Navigation

Part 1: Why Your FSx for ONTAP Logs Deserve Better
Part 2: Shipping FSx for ONTAP Logs to Datadog — The Serverless Way
Part 3: Event-Driven Ransomware Detection with ONTAP ARP + Datadog
Part 4: FPolicy File Activity Pipeline — ONTAP to Datadog via ECS Fargate
Part 5: Escape Vendor Lock-in: Multi-Backend Log Delivery with OTel Collector for FSx for ONTAP.
Part 6: Direct-to-Grafana: Shipping FSx for ONTAP Logs to Grafana Cloud Loki via OTLP Gateway
Part 7: Ship FSx for ONTAP Audit Logs to New Relic via Serverless Lambda Pipeline
Part 8: EC2 to Serverless: Modernizing FSx for ONTAP Splunk Integration
Part 9: Data Sovereignty: FSx for ONTAP Logs in Your VPC with Elastic
Part 10: High-Cardinality File Access Analysis with Honeycomb + OTel
Part 11: AI-Powered Root Cause: Correlating File Access with APM via Dynatrace
Part 12: FSx for ONTAP Audit Logs with Data Residency in your region with Sumo Logic (this post)
Part 13: 9 Vendors, One Architecture

Questions about the Sumo Logic JP integration or data residency? Drop a comment below.

GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

AI-Powered Root Cause: Correlating File Access with APM via Dynatrace

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 31 May 2026 00:26:51 +0000

TL;DR

We built a serverless Lambda pipeline that ships FSx for ONTAP audit logs to Dynatrace via the Log Ingest API v2. The real value: Dynatrace's Davis AI can automatically correlate file access anomalies with application performance degradation — answering "why is the app slow?" with "because 500 users hit the same NFS share simultaneously."

FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda → Dynatrace Log Ingest API v2
                                                                         │
                                                                         ▼
                                                                    Davis AI
                                                              ┌───────────────────┐
                                                              │ Correlates:       │
                                                              │ • File access     │
                                                              │   anomalies       │
                                                              │ • APM metrics     │
                                                              │ • Infrastructure  │
                                                              │   health          │
                                                              │                   │
                                                              │ → Root cause      │
                                                              │   in seconds      │
                                                              └───────────────────┘

Verified on Dynatrace SaaS Trial (Tokyo-equivalent region). Logs visible in Logs Viewer within 1-2 minutes.

This is Part 11 of the Serverless Observability for FSx for ONTAP series.

Why Dynatrace for FSx for ONTAP?

Most observability tools treat storage logs as isolated data. Dynatrace is different — it builds a topology map of your entire stack and uses Davis AI to find causal relationships through time-window correlation and entity connectivity:

Scenario	Without Dynatrace	With Dynatrace
App latency spike	"Check the logs"	Davis AI detects temporal correlation: file access to /vol/data/ increased 10x within the same 5-minute window as app response time degradation, connected via topology (app → NFS mount → SVM)
Storage I/O anomaly	Manual investigation	Automatic correlation via shared topology entities — Davis identifies which services are affected based on entity relationships
User reports slow file access	Grep through audit logs	DQL query + topology view showing the full dependency path from user request to storage operation

The key differentiator: Davis AI correlates events across entities that share topology connections within overlapping time windows — not just keyword matching or manual dashboard correlation.

Architecture

┌─────────────────────────────────────────────────────────┐
│ Event Sources                                           │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  EventBridge Scheduler                                  │
│  rate(5 minutes) ──→ Lambda                             │
│                       │ lists new files via             │
│                       │ S3 Access Point                 │
│                       │ (checkpoint in SSM)             │
│                       ▼                                 │
│           Dynatrace Log Ingest API v2                   │
│           (Api-Token auth)                              │
│                       │                                 │
│  EMS Webhook          │                                 │
│  ──→ API GW ──→ Lambda ─────────────┤                   │
│     (ems_handler)                   │                   │
│                                     ▼                   │
│  FPolicy                       Dynatrace                │
│  ──→ ECS Fargate ──→ SQS      (Logs Viewer,             │
│  ──→ Bridge Lambda              Davis AI,               │
│  ──→ EventBridge                DQL,                    │
│  ──→ Lambda (fpolicy_handler)   Dashboards)             │
│  ──────────────────────────────────────────────────────┤│
└─────────────────────────────────────────────────────────┘

Davis AI: The Correlation Engine

When you ship FSx for ONTAP logs to Dynatrace alongside your APM data, Davis AI can detect patterns like:

Storage contention → App slowdown: Spike in file operations correlates with increased response times
Ransomware activity → Service impact: ARP (Anti-Ransomware Protection) EMS events correlate with unusual file encryption patterns
Quota exhaustion → Write failures: ONTAP quota warnings correlate with application write errors

This works because Dynatrace maps your FSx for ONTAP SVM as a custom device entity in its topology, connecting it to the applications that access it.

Quick Start (30 Minutes)

1. Create Dynatrace API Token

Log in to your Dynatrace environment
Go to Access Tokens (Settings → Integration → Access tokens)
Create a token with scope: logs.ingest
Token format: dt0c01.<TOKEN_ID>.<TOKEN_SECRET>

2. Store Credentials

aws secretsmanager create-secret \
  --name "dynatrace/fsxn-api-token" \
  --secret-string '{"api_token":"dt0c01.XXXXXXXX.YYYYYYYY"}' \
  --region ap-northeast-1

3. Deploy CloudFormation Stack

aws cloudformation deploy \
  --template-file integrations/dynatrace/template.yaml \
  --stack-name fsxn-dynatrace-integration \
  --parameter-overrides \
    S3AccessPointArn=arn:aws:s3:ap-northeast-1:123456789012:accesspoint/fsxn-audit-ap \
    DynatraceApiTokenSecretArn=arn:aws:secretsmanager:ap-northeast-1:123456789012:secret:dynatrace/fsxn-api-token-XXXXXX \
    DynatraceEnvUrl=https://abc12345.live.dynatrace.com \
    S3BucketName=my-fsxn-audit-bucket \
  --capabilities CAPABILITY_NAMED_IAM \
  --region ap-northeast-1

4. Verify in Dynatrace

Navigate to Logs → View logs → Run query:

fetch logs
| filter log.source == "fsxn-ontap"

Logs should appear within 1-2 minutes.

Log Entry Format

Each audit log event is shipped with structured attributes for DQL querying:

{
  "content": "{\"EventID\":\"4663\",\"UserName\":\"admin@corp.local\",...}",
  "log.source": "fsxn-ontap",
  "dt.source_entity": "CUSTOM_DEVICE-fsxn-svm-prod-01",
  "timestamp": "2026-01-15T12:00:00Z",
  "severity": "info",
  "fsxn.svm": "svm-prod-01",
  "fsxn.operation": "ReadData",
  "fsxn.user": "admin@corp.local",
  "fsxn.path": "/vol/data/file.txt",
  "fsxn.s3_key": "audit/2026/01/15/audit-001.json"
}

The dt.source_entity field links logs to a custom device in Dynatrace's topology, enabling Davis AI correlation.

DQL Query Examples

Dynatrace Query Language (DQL) provides powerful analytics:

Basic Investigation

// All failed file access attempts (using structured attributes)
fetch logs
| filter log.source == "fsxn-ontap"
| filter fsxn.result == "Failure"
| summarize count(), by: {fsxn.user, fsxn.path}

// Top operations by volume
fetch logs
| filter log.source == "fsxn-ontap"
| summarize count(), by: {fsxn.operation}
| sort count() desc

// Access timeline for a specific SVM
fetch logs
| filter fsxn.svm == "svm-prod-01"
| makeTimeseries count(), interval: 5m

APM Correlation Queries

// File access volume vs app response time (side-by-side)
fetch logs
| filter log.source == "fsxn-ontap"
| makeTimeseries file_ops = count(), interval: 5m

// Correlate with service metrics in a dashboard
// (Place this next to a service response time tile)

// Find users causing the most I/O during a performance incident
fetch logs
| filter log.source == "fsxn-ontap"
| filter timestamp >= now() - 1h
| summarize ops = count(), by: {fsxn.user}
| sort ops desc
| limit 10

Security Queries

// Detect potential ransomware (mass file modifications)
fetch logs
| filter log.source == "fsxn-ontap"
| filter fsxn.operation == "WriteData" OR fsxn.operation == "Delete"
| makeTimeseries write_ops = count(), interval: 1m
| filter write_ops > 100

// After-hours access
fetch logs
| filter log.source == "fsxn-ontap"
| filter hour(timestamp) < 7 OR hour(timestamp) > 19
| summarize count(), by: {fsxn.user, fsxn.path}

Deployment Options

Deployment	URL Format	Data Location
SaaS	`https://<env-id>.live.dynatrace.com`	Dynatrace-managed (region-specific)
Managed	`https://<your-domain>/e/<env-id>`	Your infrastructure
ActiveGate	`https://<host>:9999/e/<env-id>`	Your network (proxy)

For data sovereignty requirements, Dynatrace Managed or ActiveGate keeps all data within your infrastructure.

Cost Analysis

Dynatrace pricing is based on Davis Data Units (DDU):

Monthly Log Volume	DDU/day (est.)	Monthly DDU Cost
1 GB	~1 DDU	Minimal (within base allocation)
10 GB	~10 DDU	~$25/month (at $2.50/DDU)
100 GB	~100 DDU	~$250/month

Component	Monthly Cost (10 GB/month)
Lambda (5-min polling)	~$3
EventBridge Scheduler	~$1
Secrets Manager	~$1
Dynatrace DDU	~$25
Total	~$30

DDU pricing varies by contract. The 14-day trial includes generous DDU allocation for validation. Check your license terms for production estimates.

Gotchas & Lessons Learned

#	Discovery	Impact
1	API returns HTTP 204 on success (not 200)	Lambda must treat 204 as success
2	Trial environment has 1-2 minute ingestion lag	Wait before checking Logs Viewer
3	`logs.ingest` scope is required — `ReadConfig`/`WriteConfig` won't work	Token creation must select correct scope
4	`logs.read` scope needed separately for API-based queries	Create a second token for automation
5	Log entries older than 24 hours may be rejected	Use current timestamps in test data
6	Max 1MB per request (smallest batch limit in this series)	Lambda splits large batches
7	Firehose delivery requires ActiveGate (not direct to SaaS)	Use Lambda direct for simplicity

Davis AI Integration Pattern

To get the most from Davis AI correlation, all three prerequisites must be in place:

Ship FSx for ONTAP logs (this integration) — with dt.source_entity field set
Deploy OneAgent on application hosts that access FSx for ONTAP via NFS/SMB — this creates the application-side topology
Create custom device for each SVM (dt.source_entity) — this creates the storage-side topology node. Use the Entity API (POST /api/v2/entities/custom) or Settings API to pre-create the device entity before first log ingestion

Prerequisites for correlation: Davis AI correlation only activates when all three components are connected in the topology. Without OneAgent on the application hosts, Davis AI cannot establish the causal link between file access patterns and application performance. The custom device entity must use a consistent naming convention (e.g., CUSTOM_DEVICE-fsxn-{svm-name}) across all log entries.

Application (OneAgent) ──→ NFS/SMB ──→ FSx for ONTAP (SVM)
       │                                      │
       │ APM metrics                          │ Audit logs
       ▼                                      ▼
                    Dynatrace Davis AI
                    (automatic correlation)

Production Readiness

This integration follows the project's Production Readiness Levels:

Level	What You Get	Go/No-Go to Next
Level 1 (this Quick Start)	Audit poller + DLQ	Logs arrive, checkpoint advances, DLQ empty 24h
Level 2	+ DQL dashboards + alerts	SLOs met 7 days, security review done
Level 3	+ DynamoDB ledger + Davis AI correlation	SLOs met 30 days, compliance pack
Level 4	+ OTel Collector + redaction + OneAgent	Multi-backend, PII redaction, full topology

Data classification: Dynatrace receives fsxn.user and fsxn.path fields (PII/sensitive). Dynatrace SaaS environments are region-specific — select a region matching your data residency requirements. For Managed/ActiveGate deployments, data stays in your infrastructure. See Data Classification Guide.

Full criteria: Pipeline SLO Definitions | DLQ Replay Runbook

CloudFormation Templates

Template	Purpose	Key Parameters
`template.yaml`	FSx audit log poller	S3AccessPointArn, DynatraceApiTokenSecretArn, DynatraceEnvUrl
`template-ems.yaml`	EMS webhook handler	DynatraceApiTokenSecretArn, DynatraceEnvUrl
`template-fpolicy.yaml`	FPolicy EventBridge handler	DynatraceApiTokenSecretArn, DynatraceEnvUrl, EventBusName

Resources

GitHub: integrations/dynatrace/
Dynatrace Log Ingest API: API v2 Documentation
Davis AI: Davis AI Overview
DQL Reference: Dynatrace Query Language
Series GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

Series Navigation

Part 1: Why Your FSx for ONTAP Logs Deserve Better
Part 2: Shipping FSx for ONTAP Logs to Datadog — The Serverless Way
Part 3: Event-Driven Ransomware Detection with ONTAP ARP + Datadog
Part 4: FPolicy File Activity Pipeline — ONTAP to Datadog via ECS Fargate
Part 5: Escape Vendor Lock-in: Multi-Backend Log Delivery with OTel Collector for FSx for ONTAP.
Part 6: Direct-to-Grafana: Shipping FSx for ONTAP Logs to Grafana Cloud Loki via OTLP Gateway
Part 7: Ship FSx for ONTAP Audit Logs to New Relic via Serverless Lambda Pipeline
Part 8: EC2 to Serverless: Modernizing FSx for ONTAP Splunk Integration
Part 9: Data Sovereignty: FSx for ONTAP Logs in Your VPC with Elastic
Part 10: High-Cardinality File Access Analysis with Honeycomb + OTel
Part 11: AI-Powered Root Cause with Dynatrace (this post)
Part 12: FSx for ONTAP Audit Logs with Data Residency in your region with Sumo Logic
Part 13: 9 Vendors, One Architecture

Questions about the Dynatrace integration or Davis AI correlation? Drop a comment below.

GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

High-Cardinality File Access Analysis with Honeycomb + OTel

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 31 May 2026 00:15:53 +0000

TL;DR

We built a serverless pipeline that ships FSx for ONTAP audit logs to Honeycomb, where its high-cardinality query engine turns file access data into actionable insights. Two delivery paths verified:

[Path A: Direct]
FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda → Honeycomb Events Batch API

[Path B: OTel Collector]
FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda → OTel Collector → OTLP → Honeycomb

Why Honeycomb for file access logs? Because file access data is inherently high-cardinality: thousands of users × millions of file paths × dozens of operations × multiple SVMs. Traditional log tools force you to pre-aggregate or sample. Honeycomb lets you query the raw events at full resolution.

┌──────────────────────────────────────────────────────┐
│  Honeycomb Query Engine                              │
│                                                      │
│  "Show me which users accessed /vol/finance/*        │
│   between 2am-4am last Tuesday"                      │
│                                                      │
│  → BubbleUp: auto-detect anomalous dimensions        │
│  → Heatmap: visualize access density over time       │
│  → GROUP BY user, path, operation — no pre-indexing  │
│                                                      │
│  20M events/month FREE                               │
└──────────────────────────────────────────────────────┘

This is Part 10 of the Serverless Observability for FSx for ONTAP series.

Why Honeycomb for File Access Logs?

Most observability tools index a fixed set of fields. When you have high-cardinality dimensions — like file paths (/vol/data/project-alpha/2026/Q1/report-final-v3.docx) or Active Directory usernames — you hit index bloat, slow queries, or forced sampling.

Honeycomb's columnar storage handles this natively:

Capability	Traditional Logs	Honeycomb
Query by arbitrary field	Pre-index or full scan	Instant (columnar)
GROUP BY high-cardinality field	Expensive / limited	Native
BubbleUp (anomaly detection)	Manual investigation	Semi-automatic (select time range, BubbleUp identifies differing dimensions)
Heatmap visualization	Requires pre-aggregation	Raw events

For FSx for ONTAP audit logs, this means you can ask questions like:

"Which users accessed the most files in the last hour?" (GROUP BY user)
"What's different about the spike at 3am?" (BubbleUp)
"Show me the access pattern heatmap for /vol/finance/" (Heatmap)

Architecture

┌─────────────────────────────────────────────────────────┐
│ Event Sources                                           │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  EventBridge Scheduler                                  │
│  rate(5 minutes) ──→ Lambda                             │
│                       │ lists new files via             │
│                       │ S3 Access Point                 │
│                       │ (checkpoint in SSM)             │
│                       ▼                                 │
│              Honeycomb Events Batch API                 │
│              (x-honeycomb-team header)                  │
│                       │                                 │
│  EMS Webhook          │                                 │
│  ──→ API GW ──→ Lambda ─────────────┤                   │
│     (ems_handler)                   │                   │
│                                     ▼                   │
│  FPolicy                        Honeycomb               │
│  ──→ ECS Fargate ──→ SQS       (BubbleUp,               │
│  ──→ Bridge Lambda               Heatmap,               │
│  ──→ EventBridge                 Explore)               │
│  ──→ Lambda (fpolicy_handler) ──────────────────────────┤
└─────────────────────────────────────────────────────────┘

Two Verified Delivery Paths

Path A: Direct Events Batch API

Simplest path. Lambda sends events directly to Honeycomb's Events Batch API.

# Batch format
[
  {
    "time": "2026-01-15T12:00:00Z",
    "data": {
      "source": "fsxn-ontap",
      "service": "ontap-audit",
      "event_type": "4663",
      "svm": "svm-prod-01",
      "user": "admin@corp.local",
      "operation": "ReadData",
      "path": "/vol/data/file.txt",
      "result": "Success",
      "client_ip": "10.0.x.x"
    }
  }
]

Path B: OTel Collector (OTLP)

For multi-backend delivery or when you want enrichment/redaction in the pipeline. Verified in Part 5 with Honeycomb as one of the backends.

The OTel Collector uses the otlp_http exporter with x-honeycomb-dataset header:

exporters:
  otlphttp/honeycomb:
    endpoint: https://api.honeycomb.io
    headers:
      x-honeycomb-team: ${HONEYCOMB_API_KEY}
      x-honeycomb-dataset: fsxn-audit

Quick Start (30 Minutes)

1. Get a Honeycomb Ingest Key

Sign up at honeycomb.io (free tier: 20M events/month)
Go to Account → Team Settings → API Keys
Create an Ingest Key (starts with hcaik_)

⚠️ Critical: You MUST use an Ingest Key (hcaik_*). Environment Keys (hcxik_*) will be rejected.

2. Store Credentials

aws secretsmanager create-secret \
  --name "honeycomb/fsxn-api-key" \
  --secret-string '{"api_key":"hcaik_01abc..."}' \
  --region ap-northeast-1

3. Deploy CloudFormation Stack

aws cloudformation deploy \
  --template-file integrations/honeycomb/template.yaml \
  --stack-name fsxn-honeycomb-integration \
  --parameter-overrides \
    S3AccessPointArn=arn:aws:s3:ap-northeast-1:123456789012:accesspoint/fsxn-audit-ap \
    HoneycombApiKeySecretArn=arn:aws:secretsmanager:ap-northeast-1:123456789012:secret:honeycomb/fsxn-api-key-XXXXXX \
    HoneycombDataset=fsxn-audit \
    S3BucketName=my-fsxn-audit-bucket \
  --capabilities CAPABILITY_NAMED_IAM \
  --region ap-northeast-1

4. Verify in Honeycomb

Navigate to your dataset → Explore Data:

WHERE service = "ontap-audit" | COUNT

Events should appear within seconds.

Honeycomb Query Examples

Basic Investigation

# All failed access attempts
WHERE result = "Failure" | GROUP BY user, path | COUNT

# Top 20 users by file access volume
GROUP BY user | COUNT | ORDER BY COUNT DESC | LIMIT 20

# Operations breakdown
GROUP BY operation | COUNT

High-Cardinality Analysis (Honeycomb's Strength)

# BubbleUp: What's different about the 3am spike?
# Select the spike in the time series → click BubbleUp
# Honeycomb auto-identifies which dimensions differ

# Heatmap: Access density by hour
WHERE operation = "ReadData" | HEATMAP(timestamp)

# Trace a specific user's activity
WHERE user = "admin@corp.local" | VISUALIZE COUNT | GROUP BY operation, path

# Find unusual path access patterns
GROUP BY path | COUNT | WHERE COUNT > 100

Security Investigation

# After-hours access to sensitive paths
WHERE path CONTAINS "confidential" AND hour(timestamp) NOT BETWEEN 9 AND 17
| GROUP BY user | COUNT

# Users accessing paths they haven't accessed before
# (Use Honeycomb's "compare to baseline" feature)

# Bulk file operations (potential exfiltration)
WHERE operation = "ReadData" | GROUP BY user | COUNT | WHERE COUNT > 1000

Event Schema (13 Fields)

All fields are queryable at full cardinality without pre-indexing:

Field	Example	Cardinality
`source`	fsxn-ontap	Low
`service`	ontap-audit	Low
`event_type`	4663	Low (~10 types)
`svm`	svm-prod-01	Low (~5-20)
`user`	admin@corp.local	High (thousands)
`operation`	ReadData	Low (~10 types)
`path`	/vol/data/report.pdf	Very High (millions)
`result`	Success / Failure	Low (2)
`client_ip`	10.0.x.x	Medium (hundreds)
`s3_key`	audit/svm-prod-01/2026/...	Very High

Cost Analysis

Honeycomb pricing is event-based, not volume-based:

Monthly Log Volume	Estimated Events	Honeycomb Cost
1 GB	~500K events	Free (20M/month included)
10 GB	~5M events	Free
30 GB	~15M events	Free
50 GB	~25M events	Paid tier (~$100/month)

Component	Monthly Cost (10 GB/month)
Lambda (5-min polling)	~$3
EventBridge Scheduler	~$1
Secrets Manager	~$1
Honeycomb	Free (5M events < 20M limit)
Total	~$5

The 20M events/month free tier covers most FSx for ONTAP deployments. Estimate ~500 events per MB of audit log data.

Gotchas & Lessons Learned

#	Discovery	Impact
1	*Must use Ingest Key (`hcaik_`)** — Environment Key (`hcxik_*`) is silently rejected	Events disappear without error if wrong key type
2	Events with timestamps older than ~4 hours are rejected	Test data must use current timestamps
3	5MB max request body size; our implementation batches in chunks of 100 events for reliability	Lambda splits large files into multiple requests
4	Honeycomb processes data in US regions only	Evaluate cross-border data transfer requirements
5	Dataset auto-created on first event if it doesn't exist	No pre-provisioning needed
6	OTel Collector path requires `x-honeycomb-dataset` header	Without it, events go to a default dataset

Direct vs OTel Collector: When to Use Which

Criteria	Direct (Path A)	OTel Collector (Path B)
Simplicity	✅ Fewer components	More infrastructure
Multi-backend	❌ Honeycomb only	✅ Any OTLP backend
Enrichment/redaction	❌ In Lambda only	✅ Collector processors
Cost	Lower (no Collector)	Collector compute cost
Recommendation	Single-backend PoC	Production multi-backend

Note from Honeycomb: Honeycomb recommends OTLP as the primary ingest path for new production deployments. The Events Batch API (Path A) remains fully supported and is simpler for single-backend PoCs. If you start with Path A, migrating to Path B (OTLP) requires no changes to your Honeycomb queries — only the delivery mechanism changes.

Production Readiness

This integration follows the project's Production Readiness Levels:

Level	What You Get	Go/No-Go to Next
Level 1 (this Quick Start)	Audit poller + DLQ	Logs arrive, checkpoint advances, DLQ empty 24h
Level 2	+ Honeycomb queries + alerts	SLOs met 7 days, security review done
Level 3	+ DynamoDB ledger + poison-pill	SLOs met 30 days, compliance pack
Level 4	+ OTel Collector + redaction	Multi-backend, PII redaction, DR tested

Data classification note: Honeycomb receives user and path fields which are classified as PII/sensitive. Since Honeycomb processes data in US regions only, evaluate cross-border transfer requirements. For PII-sensitive deployments, use the OTel Collector path (Path B) with redaction processors. See Data Classification Guide.

Full criteria: Pipeline SLO Definitions | DLQ Replay Runbook

CloudFormation Templates

Template	Purpose	Key Parameters
`template.yaml`	FSx audit log poller	S3AccessPointArn, HoneycombApiKeySecretArn, HoneycombDataset
`template-ems.yaml`	EMS webhook handler	HoneycombApiKeySecretArn, HoneycombDataset
`template-fpolicy.yaml`	FPolicy EventBridge handler	HoneycombApiKeySecretArn, HoneycombDataset, EventBusName

Resources

GitHub: integrations/honeycomb/
OTel Collector path: integrations/otel-collector/
Honeycomb Docs: docs.honeycomb.io
Honeycomb BubbleUp: BubbleUp Guide
Series GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

Series Navigation

Part 1: Why Your FSx for ONTAP Logs Deserve Better
Part 2: Shipping FSx for ONTAP Logs to Datadog — The Serverless Way
Part 3: Event-Driven Ransomware Detection with ONTAP ARP + Datadog
Part 4: FPolicy File Activity Pipeline — ONTAP to Datadog via ECS Fargate
Part 5: Escape Vendor Lock-in: Multi-Backend Log Delivery with OTel Collector for FSx for ONTAP.
Part 6: Direct-to-Grafana: Shipping FSx for ONTAP Logs to Grafana Cloud Loki via OTLP Gateway
Part 7: Ship FSx for ONTAP Audit Logs to New Relic via Serverless Lambda Pipeline
Part 8: EC2 to Serverless: Modernizing FSx for ONTAP Splunk Integration
Part 9: Data Sovereignty: FSx for ONTAP Logs in Your VPC with Elastic
Part 10: High-Cardinality File Access Analysis with Honeycomb (this post)
Part 11: AI-Powered Root Cause: Correlating File Access with APM via Dynatrace
Part 12: FSx for ONTAP Audit Logs with Data Residency in your region with Sumo Logic
Part 13: 9 Vendors, One Architecture

Questions about high-cardinality analysis or the Honeycomb integration? Drop a comment below.

GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

Data Sovereignty: FSx for ONTAP Logs in Your VPC with Elastic

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sun, 31 May 2026 00:08:55 +0000

TL;DR

We built a serverless Lambda pipeline that ships FSx for ONTAP audit logs to Elasticsearch via the Bulk API, with Elastic Common Schema (ECS) field mapping for native Kibana SIEM compatibility. Verified on Elastic Cloud Serverless (Tokyo, AWS).

FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda → Elasticsearch Bulk API
                                                                      │
                                                          ┌───────────┴───────────┐
                                                          │                       │
                                                   Elastic Cloud            Self-hosted
                                                   Serverless (Tokyo)       (VPC / EKS)
                                                          │                       │
                                                          ▼                       ▼
                                                       Kibana                  Kibana
                                                    (SIEM / Discover)      (full control)

Three deployment options — same Lambda, same ECS mapping, same KQL queries:

Option	Data Location	Ops Burden	Best For
Elastic Cloud Serverless	Elastic-managed (Tokyo)	Zero	Fast PoC, small teams
Elastic Cloud Dedicated	Elastic-managed (region choice)	Low	Production, compliance
Self-hosted (EC2/EKS in VPC)	Your VPC	High	Data sovereignty, air-gapped

This is Part 9 of the Serverless Observability for FSx for ONTAP series.

Why Elastic for FSx for ONTAP?

If your organization has data sovereignty requirements — public sector, financial services, healthcare — you need to control where your audit logs live. Elastic gives you that choice:

Self-hosted in your VPC: Zero data leaves your network boundary
Elastic Cloud in Tokyo: Managed service with regional data residency
ECS field mapping: Native compatibility with Kibana Security SIEM rules

The Elastic Common Schema mapping means your FSx for ONTAP audit logs work with Kibana's built-in detection rules, dashboards, and SIEM workflows without custom parsing.

Architecture

┌─────────────────────────────────────────────────────────┐
│ Event Sources                                           │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  EventBridge Scheduler                                  │
│  rate(5 minutes) ──→ Lambda                             │
│                       │ lists new files via             │
│                       │ S3 Access Point                 │
│                       │ (checkpoint in SSM)             │
│                       ▼                                 │
│              Elasticsearch Bulk API                     │
│              (ECS field mapping)                        │
│                       │                                 │
│  EMS Webhook          │                                 │
│  ──→ API GW ──→ Lambda ─────────────┤                   │
│     (ems_handler)                   │                   │
│                                     ▼                   │
│  FPolicy                         Kibana                 │
│  ──→ ECS Fargate ──→ SQS        (Discover,              │
│  ──→ Bridge Lambda                SIEM,                 │
│  ──→ EventBridge                  Dashboard)            │
│  ──→ Lambda (fpolicy_handler) ──────────────────────────┤
└─────────────────────────────────────────────────────────┘

ECS Field Mapping: Why It Matters

Our Lambda maps FSx for ONTAP audit fields to Elastic Common Schema:

{
  "@timestamp": "2026-01-15T12:00:00Z",
  "event": {"code": "4663", "type": ["access"], "category": ["file"]},
  "user": {"name": "admin@corp.local"},
  "source": {"ip": "10.0.x.x"},
  "fsxn": {
    "operation": "ReadData",
    "path": "/vol/data/confidential/report.pdf",
    "result": "Success",
    "svm": "svm-prod-01"
  },
  "cloud": {
    "provider": "aws",
    "service": {"name": "fsx-ontap"}
  }
}

This means:

event.code maps Windows Event IDs for SIEM correlation (ECS-compliant)
event.type and event.category enable Kibana Security's detection rules
user.name works with Kibana Security's user activity dashboards
source.ip integrates with threat intelligence feeds
Daily indices (fsxn-audit-YYYY.MM.DD) support ILM for retention policies

Data Streams (Elastic 8.x+): For new deployments, consider using Data Streams instead of daily indices. Data Streams handle rollover automatically and simplify ILM configuration. Our template supports both patterns.

Quick Start (30 Minutes)

1. Create Elastic Cloud Serverless Project

Sign up at cloud.elastic.co
Create an Elasticsearch Serverless project
Select AWS as cloud provider, ap-northeast-1 (Tokyo) as region
Create an API Key: Kibana → Stack Management → Security → API Keys
- Minimum required privilege: Grant only index privilege on fsxn-audit-* indices with write and create_index actions. Do not use a superuser key.

2. Store Credentials

# API Key is in Base64-encoded format (id:key)
# The key should have ONLY index:write scope on fsxn-audit-* indices
aws secretsmanager create-secret \
  --name "elastic/fsxn-api-key" \
  --secret-string '{"api_key":"<base64_encoded_id:api_key>"}' \
  --region ap-northeast-1

3. Deploy CloudFormation Stack

aws cloudformation deploy \
  --template-file integrations/elastic/template.yaml \
  --stack-name fsxn-elastic-integration \
  --parameter-overrides \
    S3AccessPointArn=arn:aws:s3:ap-northeast-1:123456789012:accesspoint/fsxn-audit-ap \
    ElasticApiKeySecretArn=arn:aws:secretsmanager:ap-northeast-1:123456789012:secret:elastic/fsxn-api-key-XXXXXX \
    ElasticEndpoint=https://my-project.es.ap-northeast-1.aws.elastic.cloud:443 \
    S3BucketName=my-fsxn-audit-bucket \
  --capabilities CAPABILITY_NAMED_IAM \
  --region ap-northeast-1

4. Verify in Kibana

Navigate to Kibana → Discover → select fsxn-audit-* index pattern:

fsxn.result: "Failure"

You should see audit log documents with ECS-mapped fields within minutes.

Index Template (Recommended)

Create this index template before first ingestion for optimal field types:

PUT _index_template/fsxn-audit
{
  "index_patterns": ["fsxn-audit-*"],
  "template": {
    "settings": {
      "number_of_replicas": 1,
      "index.lifecycle.name": "fsxn-audit-policy"
    },
    "mappings": {
      "properties": {
        "@timestamp": {"type": "date"},
        "event.code": {"type": "keyword"},
        "event.type": {"type": "keyword"},
        "event.category": {"type": "keyword"},
        "user.name": {"type": "keyword"},
        "source.ip": {"type": "ip"},
        "fsxn.operation": {"type": "keyword"},
        "fsxn.path": {"type": "text", "fields": {"keyword": {"type": "keyword"}}},
        "fsxn.result": {"type": "keyword"},
        "fsxn.svm": {"type": "keyword"},
        "cloud.provider": {"type": "keyword"},
        "cloud.service.name": {"type": "keyword"}
      }
    }
  }
}

ILM Policy (Recommended)

Create this ILM policy to manage index lifecycle automatically:

PUT _ilm/policy/fsxn-audit-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {"max_age": "1d", "max_primary_shard_size": "50gb"}
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": {"number_of_shards": 1},
          "forcemerge": {"max_num_segments": 1}
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {}
      },
      "delete": {
        "min_age": "90d",
        "actions": {"delete": {}}
      }
    }
  }
}

Adjust retention periods based on your regulatory requirements. See the Retention Policy Matrix for regulation-specific guidance (FISC: 7 years, ISMAP: 1 year, SOC 2: 1 year).

KQL Query Examples

# All failed access attempts
fsxn.result: "Failure"

# Specific user activity
user.name: "admin@corp.local" AND fsxn.operation: "WriteData"

# File access by path pattern (sensitive directories)
fsxn.path: *confidential* OR fsxn.path: *restricted*

# Operations by SVM (using event.code for Windows Event ID)
fsxn.svm: "svm-prod-01" AND event.code: "4663"

# Delete operations (potential data exfiltration)
fsxn.operation: "Delete" AND NOT user.name: "backup-svc@corp.local"

Kibana SIEM Integration

With ECS mapping, you can create detection rules in Kibana Security:

Rule	Condition	Severity
Mass file deletion	`fsxn.operation: "Delete"` count > 50 in 5m	Critical
After-hours access	`@timestamp` outside business hours + `fsxn.result: "Success"`	Medium
Unauthorized path access	`fsxn.path: restricted` AND `fsxn.result: "Failure"`	High
New user first access	`user.name` not seen in last 30 days	Low

Self-Hosted Option: Data Never Leaves Your VPC

For maximum data sovereignty, deploy Elasticsearch in your VPC:

Lambda (VPC) ──→ Elasticsearch (VPC, private subnet)
                        │
                        ▼
                 Kibana (VPC, private subnet + ALB)

Deploy Lambda in the same VPC with security group rules allowing port 9200 to the Elasticsearch cluster.

⚠️ Network path consideration: If this Lambda also reads from the FSx for ONTAP S3 Access Point, a NAT Gateway is required (S3 AP access from VPC requires internet-routed path — see S3 AP Specification). For the self-hosted pattern, consider splitting into two Lambdas: one outside VPC for S3 AP reads, and one inside VPC for Elasticsearch writes, connected via SQS.

Cost Analysis

Component	Monthly Cost (10 GB/month logs)
Lambda (5-min polling)	~$3
EventBridge Scheduler	~$1
Secrets Manager	~$1
Elastic Cloud Serverless	~$95 (Standard tier)
Self-hosted (3× r6g.large)	~$300 (EC2 + EBS)

Monthly Log Volume	Elastic Cloud	Self-Hosted
1 GB	Free trial	EC2 cost only
10 GB	~$95/month	~$300/month
100 GB	~$500/month	~$900/month

Self-hosted has no per-GB ingestion cost but requires infrastructure management. Elastic Cloud Serverless eliminates ops burden at the cost of per-GB pricing.

Gotchas & Lessons Learned

#	Discovery	Impact
1	API Key must be in Encoded (Base64) format, not the raw id:key pair	Auth failures if stored incorrectly
2	Elastic Cloud Serverless returns data instantly (no indexing lag)	Faster validation than some competitors
3	Daily indices (`fsxn-audit-YYYY.MM.DD`) require ILM policy for cleanup	Unbounded storage growth without ILM
4	`fsxn.path` needs both `text` and `keyword` sub-field for full-text + exact match	Missing queries if only one type
5	Bulk API recommended max ~10MB per request	Lambda batches accordingly
6	No built-in Firehose destination — Lambda direct delivery only	Simpler architecture, but no Firehose buffering

Data Sovereignty & Classification

Audit logs contain fields classified as PII (user.name) and sensitive business data (fsxn.path). For data sovereignty deployments:

Self-hosted: Full control — implement field-level security in Elasticsearch to restrict PII access by role
Elastic Cloud: Use Elastic's field-level security + ILM retention policies
Redaction path: Deploy OTel Collector (Part 5) with attributes/redact processor before Elasticsearch

See the Data Classification Guide for field-by-field PII classification and recommended handling patterns per regulatory framework (APPI, GDPR, FISC, ISMAP).

Production Readiness

This integration follows the project's Production Readiness Levels:

Level	What You Get	Go/No-Go to Next
Level 1 (this Quick Start)	Audit poller + DLQ	Logs arrive, checkpoint advances, DLQ empty 24h
Level 2	+ Kibana dashboard + alerts	SLOs met 7 days, security review done
Level 3	+ DynamoDB ledger + poison-pill	SLOs met 30 days, compliance pack
Level 4	+ OTel Collector + redaction	Multi-backend, PII redaction, DR tested

Full criteria: Pipeline SLO Definitions | DLQ Replay Runbook

Enterprise scale: For multi-account deployments, see the StackSets deployment guide. For DR, see Cross-Region Replication. For compliance evidence collection, see the Compliance Evidence Pack.

CloudFormation Templates

Template	Purpose	Key Parameters
`template.yaml`	FSx audit log poller	S3AccessPointArn, ElasticApiKeySecretArn, ElasticEndpoint
`template-ems.yaml`	EMS webhook handler	ElasticApiKeySecretArn, ElasticEndpoint
`template-fpolicy.yaml`	FPolicy EventBridge handler	ElasticApiKeySecretArn, ElasticEndpoint, EventBusName

Resources

GitHub: integrations/elastic/
Elastic Cloud: cloud.elastic.co
Elastic Common Schema: ECS Reference
Kibana SIEM: Security Solution
Series GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

Series Navigation

Part 1: Why Your FSx for ONTAP Logs Deserve Better
Part 2: Shipping FSx for ONTAP Logs to Datadog — The Serverless Way
Part 3: Event-Driven Ransomware Detection with ONTAP ARP + Datadog
Part 4: FPolicy File Activity Pipeline — ONTAP to Datadog via ECS Fargate
Part 5: Escape Vendor Lock-in: Multi-Backend Log Delivery with OTel Collector for FSx for ONTAP.
Part 6: Direct-to-Grafana: Shipping FSx for ONTAP Logs to Grafana Cloud Loki via OTLP Gateway
Part 7: Ship FSx for ONTAP Audit Logs to New Relic via Serverless Lambda Pipeline
Part 8: EC2 to Serverless: Modernizing FSx for ONTAP Splunk Integration
Part 9: Data Sovereignty with Elastic (this post)
Part 10: High-Cardinality File Access Analysis with Honeycomb + OTel
Part 11: AI-Powered Root Cause: Correlating File Access with APM via Dynatrace
Part 12: FSx for ONTAP Audit Logs with Data Residency in your region with Sumo Logic
Part 13: 9 Vendors, One Architecture

Questions about the Elastic integration or data sovereignty patterns? Drop a comment below.

GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

EC2 to Serverless: Modernizing FSx for ONTAP Splunk Integration

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sat, 30 May 2026 23:56:44 +0000

TL;DR

The existing AWS Blog approach ships FSx for ONTAP audit logs to Splunk via two EC2 instances (syslog-ng + Universal Forwarder). We replaced it with a single Lambda function — same Splunk index, same SPL queries, 90% AWS infrastructure cost reduction.

[Before] FSx for ONTAP → syslog-ng (EC2) → Splunk UF (EC2) → Splunk
         Monthly AWS infra cost: ~$66 (2× t3.medium + EBS)
         Ops burden: OS patching, agent updates, scaling

[After]  FSx for ONTAP → S3 Access Point → Lambda → Splunk HEC
         Monthly AWS infra cost: ~$6 (Lambda + EventBridge)
         Ops burden: Zero (managed services only)

Important: The 90% cost reduction refers to AWS infrastructure costs only (EC2/Lambda/EventBridge). Splunk platform licensing costs remain unchanged regardless of the delivery method.

This is Part 8 of the Serverless Observability for FSx for ONTAP series.

The Problem with EC2-Based Splunk Integration

The AWS Blog's architecture works, but it comes with operational overhead:

Concern	EC2-Based	Serverless
Monthly cost	~$66 fixed	~$6 pay-per-use
OS patching	Monthly	None
Agent updates	Manual (UF + syslog-ng)	None
Scaling	Manual instance resize	Automatic (Lambda concurrency)
Availability	Single AZ (unless you add redundancy)	Multi-AZ by default
Time to deploy	Hours (provision + configure)	30 minutes (CloudFormation)

If you're already running this EC2 pattern and want to modernize, this article shows you how — with a parallel deployment strategy that ensures zero data loss during cutover.

Architecture

┌──────────────────────────────────────────────────────────┐
│ FSx for ONTAP                                            │
│                                                          │
│  Audit Volume ──→ S3 Access Point                        │
│                        │                                 │
│                        ▼                                 │
│  EventBridge Scheduler (rate: 5 min)                     │
│                        │                                 │
│                        ▼                                 │
│  Lambda (Python 3.12)                                    │
│    • Reads audit logs via S3 AP                          │
│    • Parses JSON/EVTX                                    │
│    • Formats as Splunk HEC events                        │
│    • Sends with Authorization: Splunk <token>            │
│    • Checkpoints in SSM Parameter Store                  │
│                        │                                 │
│                        ▼                                 │
│  Splunk HEC                                              │
│  https://<splunk>:8088/services/collector/event          │
│  Response: {"text":"Success","code":0}                   │
│                                                          │
│  SPL: index=fsxn_audit sourcetype=fsxn:ontap:audit       │
└──────────────────────────────────────────────────────────┘

High-Volume Alternative: Firehose Path

For sustained >1000 events/sec, use Kinesis Data Firehose with its built-in Splunk destination:

FSx for ONTAP → S3 AP → Lambda (transform) → Kinesis Data Firehose → Splunk HEC

A separate template-firehose.yaml is provided for this path.

Migration Strategy (Zero Data Loss)

Phase 1: Parallel Deployment (Day 1-3)

Deploy the serverless stack alongside the existing EC2 pipeline. Use a separate Splunk index for validation:

aws cloudformation deploy \
  --template-file integrations/splunk-serverless/template.yaml \
  --stack-name fsxn-splunk-integration \
  --parameter-overrides \
    S3AccessPointArn=<S3_AP_ARN> \
    SplunkHecTokenSecretArn=<SECRET_ARN> \
    SplunkHecEndpoint=https://splunk.example.com:8088 \
    S3BucketName=<BUCKET> \
    SplunkIndex=fsxn_audit_serverless \
  --capabilities CAPABILITY_IAM

Compare events between old and new pipelines for 48 hours:

| stats count by index
| where index IN ("fsxn_audit", "fsxn_audit_serverless")

Phase 2: Cutover (Day 4-5)

Once event parity is confirmed:

Update the stack to use the production index (fsxn_audit)
Stop the syslog-ng and UF services on EC2 (don't terminate yet)
Monitor for 24 hours

Phase 3: Cleanup (Day 7+)

# Terminate EC2 instances
# Remove security groups, IAM roles, EBS volumes
# Delete old CloudFormation/Terraform resources

What Changes for Splunk Users

Unchanged ✅

Index name and sourcetype (configurable)
SPL queries — same field names
Dashboards and saved searches
Alert rules

Changed ⚠️

host field: EC2 hostname → SVM name
source field: syslog path → fsxn-observability
Delivery latency: near-real-time (syslog) → polling interval (default 5 min)

HEC Event Format

{
  "time": 1716508800,
  "host": "svm-prod-01",
  "source": "fsxn-observability",
  "sourcetype": "fsxn:ontap:audit",
  "index": "fsxn_audit",
  "event": {
    "event_type": "4663",
    "user": "admin@corp.local",
    "operation": "ReadData",
    "path": "/vol/data/report.pdf",
    "result": "Success",
    "client_ip": "10.0.1.50"
  }
}

SPL Query Examples

# Failed access attempts
index=fsxn_audit sourcetype=fsxn:ontap:audit result=Failure
| stats count by user, path
| sort -count

# Operations timeline
index=fsxn_audit sourcetype=fsxn:ontap:audit
| timechart span=5m count by operation

# Top users
index=fsxn_audit sourcetype=fsxn:ontap:audit
| stats count by user
| sort -count
| head 20

# Specific user investigation
index=fsxn_audit sourcetype=fsxn:ontap:audit user="admin@corp.local"
| table _time, operation, path, result, client_ip

Cost Comparison

Component	EC2-Based (monthly)	Serverless (monthly)	Savings
EC2 instances (2× t3.medium)	$60	$0	100%
EBS volumes (2× 20GB)	$6	$0	100%
Lambda	$0	~$5	—
EventBridge Scheduler	$0	~$0.01	—
Secrets Manager	$0	~$0.40	—
Total	$66	$6	91%

Note: EC2 cost assumes 2× t3.medium (as per the AWS Blog reference architecture). Actual EC2 costs vary by instance type and region. Splunk Cloud licensing costs are contract-dependent and may differ significantly from list pricing.

Network Considerations

Splunk Deployment	Lambda Config	Notes
Splunk Cloud (public HEC)	Lambda outside VPC	Simplest
Splunk Enterprise (private VPC)	Lambda in VPC + NAT	Same VPC as Splunk
Splunk Cloud (PrivateLink)	Lambda in VPC + VPC Endpoint	Most secure

⚠️ VerifySSL: Set to true in production. Only use false for self-signed certs in dev environments.

Rollback Plan

If issues are discovered after cutover:

Start the stopped EC2 instances (syslog-ng + UF)
Verify syslog-ng is receiving events
Delete the serverless CloudFormation stack
Investigate and resolve before re-attempting

The serverless Lambda uses checkpointing — no events are lost during the overlap period (brief duplicates are possible).

What's Next

Firehose path: For high-volume logs (>1000 events/sec), use template-firehose.yaml
HEC Acknowledgment (useACK): For Level 2+, enable HEC indexer acknowledgment to guarantee at-least-once delivery. Lambda waits for ack before advancing checkpoint
CIM compliance: Map fields to Splunk's Common Information Model (Authentication or Change data model) for compatibility with Splunk Enterprise Security correlation searches
Index pre-creation: The fsxn_audit index must be created before first ingestion (Splunk Cloud: Admin Console; Enterprise: indexes.conf)
EMS webhooks: Real-time ARP ransomware detection alerts
FPolicy: Sub-second file operation streaming
Production Readiness: Progress from Level 1 (this Quick Start) to Level 4 (Enterprise) — see the Pipeline SLO Definitions

Production Readiness

This integration follows the project's Production Readiness Levels:

Level	What You Get	Go/No-Go to Next
Level 1 (this Quick Start)	Audit poller + DLQ	Logs arrive, checkpoint advances, DLQ empty 24h
Level 2	+ Splunk dashboards + alerts	SLOs met 7 days, security review done
Level 3	+ DynamoDB ledger + poison-pill	SLOs met 30 days, compliance pack
Level 4	+ OTel Collector + redaction	Multi-backend, PII redaction, DR tested

Data classification: Splunk receives user and path fields (PII/sensitive). For Splunk Cloud, data is processed in the vendor's infrastructure. For self-hosted Splunk Enterprise, data stays in your VPC. See Data Classification Guide for field-by-field PII classification and handling patterns.

Full criteria: Pipeline SLO Definitions | DLQ Replay Runbook

Resources

Series Navigation

Part 1: Why Your FSx for ONTAP Audit Logs Deserve Better Than EC2
Part 2: Shipping FSx for ONTAP Logs to Datadog — The Serverless Way
Part 3: Event-Driven Ransomware Detection with ONTAP ARP + Datadog
Part 4: FPolicy File Activity Pipeline — ONTAP to Datadog via ECS Fargate
Part 5: Escape Vendor Lock-in: Multi-Backend Log Delivery with OTel Collector for FSx for ONTAP.
Part 6: Direct-to-Grafana: Shipping FSx for ONTAP Logs to Grafana Cloud Loki via OTLP Gateway
Part 7: Ship FSx for ONTAP Audit Logs to New Relic via Serverless Lambda Pipeline
Part 8: EC2 to Serverless: Modernizing Splunk Integration (this post)
Part 9: Data Sovereignty: FSx for ONTAP Logs in Your VPC with Elastic
Part 10: High-Cardinality File Access Analysis with Honeycomb + OTel
Part 11: AI-Powered Root Cause: Correlating File Access with APM via Dynatrace
Part 12: FSx for ONTAP Audit Logs with Data Residency in your region with Sumo Logic
Part 13: 9 Vendors, One Architecture

Questions about the Splunk migration or serverless HEC delivery? Drop a comment below.

GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

Ship FSx for ONTAP Audit Logs to New Relic via Serverless Lambda Pipeline

Yoshiki Fujiwara(藤原善基)@AWS Community Builder — Sat, 30 May 2026 23:38:34 +0000

TL;DR

New Relic's 100 GB/month free tier makes it the lowest-barrier entry point for FSx for ONTAP audit log observability. Deploy a single CloudFormation stack, and your file access events are queryable in NRQL within 30 minutes — no credit card required.

FSx for ONTAP audit volume
  → S3 Access Point
    → EventBridge Scheduler (every 5 min)
      → Lambda (reads + formats)
        → New Relic Log API v1 (HTTP 202)
          → NRQL queryable in ~30 seconds

This is Part 7 of the Serverless Observability for FSx for ONTAP series. Previous parts covered Datadog, OTel Collector, and Grafana Cloud.

Why New Relic for FSx for ONTAP Audit Logs?

Criteria	New Relic	Comparison
Free tier	100 GB/month (permanent, no expiry)	Grafana: 50 GB/month (14-day retention), Sumo Logic: 500 MB/day (~15 GB/month), Datadog: none
Retention (free)	30 days	Grafana: 14 days
Query language	NRQL (SQL-like)	Familiar to anyone who knows SQL
Alert conditions	NRQL-based, free	Included in free tier
Credit card required	No	Some vendors require CC for trial

For a typical FSx for ONTAP file server generating 1-10 GB/month of audit logs, New Relic's free tier covers the entire workload at zero cost.

Architecture

┌──────────────────────────────────────────────────────────┐
│ FSx for ONTAP                                            │
│                                                          │
│  Audit Volume ──→ S3 Access Point                        │
│                        │                                 │
│                        ▼                                 │
│  EventBridge Scheduler (rate: 5 min)                     │
│                        │                                 │
│                        ▼                                 │
│  Lambda (Python 3.12)                                    │
│    • Reads audit logs via S3 AP                          │
│    • Parses JSON/EVTX                                    │
│    • Formats for New Relic Log API                       │
│    • Sends gzip-compressed payload                       │
│    • Checkpoints in SSM Parameter Store                  │
│                        │                                 │
│                        ▼                                 │
│  New Relic Log API v1                                    │
│  https://log-api.newrelic.com/log/v1                     │
│  Header: Api-Key: <license-key>                          │
│  Response: HTTP 202 + requestId                          │
│                                                          │
│                        ▼                                 │
│  NRQL: SELECT * FROM Log WHERE source='fsxn-ontap'       │
└──────────────────────────────────────────────────────────┘

Quick Start (30 Minutes)

Prerequisites

AWS account with FSx for ONTAP + S3 Access Point configured (see Prerequisites Guide if not yet set up — allow 1-2 hours for initial FSx for ONTAP + S3 AP configuration)
New Relic free account (sign up here — no credit card)

Step 1: Get Your License Key

New Relic removed the "Copy key" button from the UI in September 2024. Use the NerdGraph API:

# 1. Create a User API Key in New Relic UI (API Keys page)
# 2. Store it in an environment variable (never pass keys inline):
export NR_USER_API_KEY="NRAK-YOUR-USER-KEY"
export NR_ACCOUNT_ID="YOUR_ACCOUNT_ID"

# 3. Use it to query the License Key via NerdGraph:
curl -s -X POST "https://api.newrelic.com/graphql" \
  -H "Content-Type: application/json" \
  -H "API-Key: ${NR_USER_API_KEY}" \
  -d "{\"query\":\"{ actor { apiAccess { keySearch(query: {types: INGEST, scope: {accountIds: ${NR_ACCOUNT_ID}}}) { keys { ... on ApiAccessIngestKey { key name type } } } } } }\"}" \
  | jq -r '.data.actor.apiAccess.keySearch.keys[0].key'

# The output is your License Key — store it securely, do not log it.

Step 2: Store in Secrets Manager

aws secretsmanager create-secret \
  --name "new-relic/fsxn-license-key" \
  --secret-string '{"license_key":"YOUR_LICENSE_KEY_HERE"}' \
  --region ap-northeast-1

For automated credential rotation, see the Secrets Rotation Sample Template which provides a Lambda-based rotation pattern for all vendor integrations.

Step 3: Deploy

aws cloudformation deploy \
  --template-file integrations/new-relic/template.yaml \
  --stack-name fsxn-new-relic-integration \
  --parameter-overrides \
    S3AccessPointArn=arn:aws:s3:ap-northeast-1:123456789012:accesspoint/fsxn-audit \
    NewRelicLicenseKeySecretArn=<SECRET_ARN> \
    NewRelicRegion=US \
    S3BucketName=<YOUR_BUCKET> \
  --capabilities CAPABILITY_NAMED_IAM \
  --region ap-northeast-1

Step 4: Verify

-- In New Relic Query Builder:
SELECT count(*) FROM Log WHERE source = 'fsxn-ontap' SINCE 15 minutes ago

If you see results, you're done. The pipeline is working.

What Gets Shipped

Each audit log event is formatted as a New Relic log entry with structured attributes:

{
  "message": "{\"EventID\":\"4663\",\"UserName\":\"admin@corp.local\",...}",
  "timestamp": 1716508800000,
  "attributes": {
    "source": "fsxn-ontap",
    "service": "ontap-audit",
    "event_type": "4663",
    "svm": "svm-prod-01",
    "user": "admin@corp.local",
    "client_ip": "10.0.1.50",
    "operation": "ReadData",
    "path": "/vol/data/report.pdf",
    "result": "Success",
    "s3_key": "audit/svm-prod-01/2026/05/24/audit-001.json"
  }
}

Important: New Relic requires timestamp as Unix epoch in milliseconds (not seconds, not ISO 8601 strings). Our Lambda handles this conversion automatically.

NRQL Query Examples

-- Failed access attempts (security investigation)
SELECT count(*) FROM Log
WHERE source = 'fsxn-ontap' AND result = 'Failure'
FACET user, path
SINCE 1 hour ago

-- Top users by file access volume
SELECT count(*) FROM Log
WHERE source = 'fsxn-ontap'
FACET user
SINCE 1 day ago

-- Operations breakdown (pie chart)
SELECT count(*) FROM Log
WHERE source = 'fsxn-ontap'
FACET operation
SINCE 1 hour ago

-- Time series for anomaly detection
SELECT count(*) FROM Log
WHERE source = 'fsxn-ontap'
TIMESERIES 5 minutes
SINCE 6 hours ago

-- Specific user investigation
SELECT * FROM Log
WHERE source = 'fsxn-ontap' AND user = 'admin@corp.local'
SINCE 1 hour ago
LIMIT 100

Alert Configuration

Create a NRQL alert condition to detect failed access spikes:

mutation {
  alertsNrqlConditionStaticCreate(
    accountId: YOUR_ACCOUNT_ID,
    policyId: YOUR_POLICY_ID,
    condition: {
      name: "FSxN Failed Access Spike",
      nrql: {
        query: "SELECT count(*) FROM Log WHERE source = 'fsxn-ontap' AND result = 'Failure'"
      },
      terms: [{
        threshold: 10,
        thresholdOccurrences: AT_LEAST_ONCE,
        thresholdDuration: 300,
        operator: ABOVE,
        priority: CRITICAL
      }]
    }
  ) { id }
}

This fires when more than 10 failed access attempts occur in a 5-minute window.

Threshold rationale: In typical FSx for ONTAP environments, failed access attempts are 0-2 per 5-minute window during normal operations (e.g., stale NFS handles, permission misconfigurations). A threshold of 10 indicates a potential brute-force attempt or systematic misconfiguration. Adjust based on your baseline — monitor for 1 week before setting production thresholds.

Cost Analysis

Monthly Log Volume	AWS Cost	New Relic Cost	Total
1 GB	~$2	$0 (free tier)	$2
10 GB	~$8	$0 (free tier)	$8
50 GB	~$25	$0 (free tier)	$25
100 GB	~$41	$0 (at limit)	$41
200 GB	~$80	$35 (overage)	$115

For comparison, the EC2-based approach (syslog-ng + Universal Forwarder) costs ~$66/month in fixed infrastructure regardless of log volume.

Gotchas We Discovered

1. License Key Retrieval (Post-September 2024)

New Relic removed the "Copy key" button from the API Keys UI. You must use NerdGraph to retrieve the full License Key value using the Key ID.

2. Timestamp Format

New Relic's Log API rejects ISO 8601 strings with Error unmarshalling message payload. Timestamps must be Unix epoch in milliseconds (integer).

# Wrong: "2026-01-15T12:00:00Z" → rejected
# Right: 1768478400000 → accepted

3. New Account Initialization Lag

First-time data ingestion on a new account may take 5-10 minutes to appear in the UI. Subsequent deliveries appear within 30 seconds.

4. US vs EU Region

Choose your region at signup. It cannot be changed later. For Japan-based workloads, US is recommended until the Tokyo data center launches in July 2026 (announced March 2026).

What's Next

EMS webhooks: Receive ONTAP system events (ARP ransomware detection, quota alerts) in real-time
FPolicy events: Stream file operations with sub-second latency
OTel Collector path: If you later need multi-backend delivery, the same log format works with the Part 5 Collector
OTLP ingest: New Relic supports OTLP natively — if you adopt OTel Collector, you can switch from Log API v1 to OTLP without changing the Lambda code
logtype attribute: Add "logtype": "fsxn-ontap-audit" to enable New Relic's automatic Parsing Rules for structured field extraction
Production Readiness: Progress from Level 1 (this Quick Start) to Level 4 (Enterprise) — see the Pipeline SLO Definitions

Production Readiness

This integration follows the project's Production Readiness Levels:

Level	What You Get	Go/No-Go to Next
Level 1 (this Quick Start)	Audit poller + DLQ	Logs arrive, checkpoint advances, DLQ empty 24h
Level 2	+ NRQL dashboards + alerts	SLOs met 7 days, security review done
Level 3	+ DynamoDB ledger + poison-pill	SLOs met 30 days, compliance pack
Level 4	+ OTel Collector + redaction	Multi-backend, PII redaction, DR tested

Data classification: New Relic receives user and path fields (PII/sensitive). New Relic currently processes data in US and EU regions. For Japan-based workloads requiring data residency, evaluate cross-border transfer requirements with your compliance team. See Data Classification Guide.

Full criteria: Pipeline SLO Definitions | DLQ Replay Runbook

Note on S3 AP network path: The Lambda in this integration is deployed outside VPC (no VPC configuration) for simplest S3 Access Point access. If your Lambda needs VPC access for other reasons, add a NAT Gateway — see the S3 AP Specification for network constraints.

Resources

Series Navigation

Part 1: Why Your FSx for ONTAP Audit Logs Deserve Better Than EC2
Part 2: Shipping FSx for ONTAP Logs to Datadog — The Serverless Way
Part 3: Event-Driven Ransomware Detection with ONTAP ARP + Datadog
Part 4: FPolicy File Activity Pipeline — ONTAP to Datadog via ECS Fargate
Part 5: Escape Vendor Lock-in: Multi-Backend Log Delivery with OTel Collector for FSx for ONTAP.
Part 6: Direct-to-Grafana: Shipping FSx for ONTAP Logs to Grafana Cloud Loki via OTLP Gateway
Part 7: Ship FSx for ONTAP Audit Logs to New Relic via Serverless Lambda Pipeline (this post)
Part 8: EC2 to Serverless: Modernizing FSx for ONTAP Splunk Integration
Part 9: Data Sovereignty: FSx for ONTAP Logs in Your VPC with Elastic
Part 10: High-Cardinality File Access Analysis with Honeycomb + OTel
Part 11: AI-Powered Root Cause: Correlating File Access with APM via Dynatrace
Part 12: FSx for ONTAP Audit Logs with Data Residency in your region with Sumo Logic
Part 13: 9 Vendors, One Architecture

Questions about the New Relic integration or the 100GB free tier? Drop a comment below.

GitHub: github.com/Yoshiki0705/fsxn-observability-integrations

DEV Community: Yoshiki Fujiwara(藤原 善基)@AWS Community Builder

Governance & Cross-Platform Access: Lake Formation, PII Anonymization, and Multi-Engine Reality for S3 Tables

Previously...

Lake Formation: Governance on Unstructured Data

The Problem

The Solution

Verified: Access Control in Action

Lake Formation Governance Status

Observed Limitation: Column-Level Grants on This S3 Tables Federated Catalog Path

PII Detection: English + Japanese

The Challenge

Dual-Engine Architecture

Japanese PII Detection (Verified)

Anonymization Pipeline

Data Clean Room Pattern

Encryption and Data Residency

Audit Log Retention

Path Sensitivity Model

Raw Data Access Boundary

S3 Access Point Identity Boundary

Permission Identity Strategy

Retention and Deletion Semantics

Approval Evidence Template (for Regulated Industries)

Regulated Workload Readiness

Cross-Platform Access: The Current Reality

Fully Verified ✅

Expected / Requires Validation ⚠️

What Doesn't Work (Yet) ⚠️

Databricks: Three Integration Paths to Validate

Confirmed Limitations (2026-06-01)

Federation Directionality

Databricks UC Audit Logging for External Engines (Confirmed 2026-06-01)

Snowflake: S3 Tables via Glue REST + VENDED_CREDENTIALS ✅

Working Configuration (Verified 2026-06-05)

Verified Capabilities (2026-06-08)

Key Insight: Why Previous Attempts Failed

Governance Limitation: Lake Formation Column-Level (2026-06-08)

Snowflake Iceberg Access Modes (Summary)

Snowflake Metadata Activation Pattern

Snowflake Cortex Search Activation Pattern

Snowflake Metadata Product Contract

Databricks Metadata Activation Pattern

Other Lakehouse Engines to Validate

Catalog Authority Rule

The Bigger Picture

Catalog Decision Guide

When to Consider Snowflake Open Catalog / Polaris

Format Decision for Databricks Environments

Summary: What We Built

The Numbers

What's Next for This Project

Get Involved

AI Enrichment Pipeline: From Sample Classification to 100K-File Metadata Search with Bedrock and OpenSearch NextGen

Quick Recap: What We Built in Part 1

What We're Building

AI Classification: Bedrock Claude Vision

How It Works

Results (Measured)

For Non-Image Files

Vector Embeddings: Titan Embeddings V2

Why 1024 Dimensions?

Embedding Storage in Iceberg

Important: Append-Only Writes and Deduplication

Vector Search: OpenSearch Serverless NextGen

The Scale-to-Zero Revolution

kNN Search Implementation

Search Results (Measured)

The Complete Pipeline

Processing Flow

Error Handling

Monitoring the Pipeline

Cost at Scale

Search Index Consistency

FPolicy Event Design

Hybrid Search Pattern

Storage Tier Impact During Backfill

Backfill vs Incremental Cost Model

Try It Yourself

AI Safety and Human Review Boundary

Evaluation Plan

DEV Community: Yoshiki Fujiwara(藤原善基)@AWS Community Builder