Yoshiki Fujiwara(藤原善基)@AWS Community Builder for AWS Community Builders

Posted on May 31

Data Sovereignty: FSx for ONTAP Logs in Your VPC with Elastic

#aws #elastic #observability #amazonfsxfornetappontap

TL;DR

We built a serverless Lambda pipeline that ships FSx for ONTAP audit logs to Elasticsearch via the Bulk API, with Elastic Common Schema (ECS) field mapping for native Kibana SIEM compatibility. Verified on Elastic Cloud Serverless (Tokyo, AWS).

FSx for ONTAP → S3 Access Point → EventBridge Scheduler → Lambda → Elasticsearch Bulk API
                                                                      │
                                                          ┌───────────┴───────────┐
                                                          │                       │
                                                   Elastic Cloud            Self-hosted
                                                   Serverless (Tokyo)       (VPC / EKS)
                                                          │                       │
                                                          ▼                       ▼
                                                       Kibana                  Kibana
                                                    (SIEM / Discover)      (full control)

Three deployment options — same Lambda, same ECS mapping, same KQL queries:

Option	Data Location	Ops Burden	Best For
Elastic Cloud Serverless	Elastic-managed (Tokyo)	Zero	Fast PoC, small teams
Elastic Cloud Dedicated	Elastic-managed (region choice)	Low	Production, compliance
Self-hosted (EC2/EKS in VPC)	Your VPC	High	Data sovereignty, air-gapped

This is Part 9 of the Serverless Observability for FSx for ONTAP series.

Why Elastic for FSx for ONTAP?

If your organization has data sovereignty requirements — public sector, financial services, healthcare — you need to control where your audit logs live. Elastic gives you that choice:

Self-hosted in your VPC: Zero data leaves your network boundary
Elastic Cloud in Tokyo: Managed service with regional data residency
ECS field mapping: Native compatibility with Kibana Security SIEM rules

The Elastic Common Schema mapping means your FSx for ONTAP audit logs work with Kibana's built-in detection rules, dashboards, and SIEM workflows without custom parsing.

Architecture

┌─────────────────────────────────────────────────────────┐
│ Event Sources                                           │
├─────────────────────────────────────────────────────────┤
│                                                         │
│  EventBridge Scheduler                                  │
│  rate(5 minutes) ──→ Lambda                             │
│                       │ lists new files via             │
│                       │ S3 Access Point                 │
│                       │ (checkpoint in SSM)             │
│                       ▼                                 │
│              Elasticsearch Bulk API                     │
│              (ECS field mapping)                        │
│                       │                                 │
│  EMS Webhook          │                                 │
│  ──→ API GW ──→ Lambda ─────────────┤                   │
│     (ems_handler)                   │                   │
│                                     ▼                   │
│  FPolicy                         Kibana                 │
│  ──→ ECS Fargate ──→ SQS        (Discover,              │
│  ──→ Bridge Lambda                SIEM,                 │
│  ──→ EventBridge                  Dashboard)            │
│  ──→ Lambda (fpolicy_handler) ──────────────────────────┤
└─────────────────────────────────────────────────────────┘

ECS Field Mapping: Why It Matters

Our Lambda maps FSx for ONTAP audit fields to Elastic Common Schema:

{
  "@timestamp": "2026-01-15T12:00:00Z",
  "event": {"code": "4663", "type": ["access"], "category": ["file"]},
  "user": {"name": "admin@corp.local"},
  "source": {"ip": "10.0.x.x"},
  "fsxn": {
    "operation": "ReadData",
    "path": "/vol/data/confidential/report.pdf",
    "result": "Success",
    "svm": "svm-prod-01"
  },
  "cloud": {
    "provider": "aws",
    "service": {"name": "fsx-ontap"}
  }
}

This means:

event.code maps Windows Event IDs for SIEM correlation (ECS-compliant)
event.type and event.category enable Kibana Security's detection rules
user.name works with Kibana Security's user activity dashboards
source.ip integrates with threat intelligence feeds
Daily indices (fsxn-audit-YYYY.MM.DD) support ILM for retention policies

Data Streams (Elastic 8.x+): For new deployments, consider using Data Streams instead of daily indices. Data Streams handle rollover automatically and simplify ILM configuration. Our template supports both patterns.

Quick Start (30 Minutes)

1. Create Elastic Cloud Serverless Project

Sign up at cloud.elastic.co
Create an Elasticsearch Serverless project
Select AWS as cloud provider, ap-northeast-1 (Tokyo) as region
Create an API Key: Kibana → Stack Management → Security → API Keys
- Minimum required privilege: Grant only index privilege on fsxn-audit-* indices with write and create_index actions. Do not use a superuser key.

2. Store Credentials

# API Key is in Base64-encoded format (id:key)
# The key should have ONLY index:write scope on fsxn-audit-* indices
aws secretsmanager create-secret \
  --name "elastic/fsxn-api-key" \
  --secret-string '{"api_key":"<base64_encoded_id:api_key>"}' \
  --region ap-northeast-1

3. Deploy CloudFormation Stack

aws cloudformation deploy \
  --template-file integrations/elastic/template.yaml \
  --stack-name fsxn-elastic-integration \
  --parameter-overrides \
    S3AccessPointArn=arn:aws:s3:ap-northeast-1:123456789012:accesspoint/fsxn-audit-ap \
    ElasticApiKeySecretArn=arn:aws:secretsmanager:ap-northeast-1:123456789012:secret:elastic/fsxn-api-key-XXXXXX \
    ElasticEndpoint=https://my-project.es.ap-northeast-1.aws.elastic.cloud:443 \
    S3BucketName=my-fsxn-audit-bucket \
  --capabilities CAPABILITY_NAMED_IAM \
  --region ap-northeast-1

4. Verify in Kibana

Navigate to Kibana → Discover → select fsxn-audit-* index pattern:

fsxn.result: "Failure"

You should see audit log documents with ECS-mapped fields within minutes.

Index Template (Recommended)

Create this index template before first ingestion for optimal field types:

PUT _index_template/fsxn-audit
{
  "index_patterns": ["fsxn-audit-*"],
  "template": {
    "settings": {
      "number_of_replicas": 1,
      "index.lifecycle.name": "fsxn-audit-policy"
    },
    "mappings": {
      "properties": {
        "@timestamp": {"type": "date"},
        "event.code": {"type": "keyword"},
        "event.type": {"type": "keyword"},
        "event.category": {"type": "keyword"},
        "user.name": {"type": "keyword"},
        "source.ip": {"type": "ip"},
        "fsxn.operation": {"type": "keyword"},
        "fsxn.path": {"type": "text", "fields": {"keyword": {"type": "keyword"}}},
        "fsxn.result": {"type": "keyword"},
        "fsxn.svm": {"type": "keyword"},
        "cloud.provider": {"type": "keyword"},
        "cloud.service.name": {"type": "keyword"}
      }
    }
  }
}

ILM Policy (Recommended)

Create this ILM policy to manage index lifecycle automatically:

PUT _ilm/policy/fsxn-audit-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {"max_age": "1d", "max_primary_shard_size": "50gb"}
        }
      },
      "warm": {
        "min_age": "7d",
        "actions": {
          "shrink": {"number_of_shards": 1},
          "forcemerge": {"max_num_segments": 1}
        }
      },
      "cold": {
        "min_age": "30d",
        "actions": {}
      },
      "delete": {
        "min_age": "90d",
        "actions": {"delete": {}}
      }
    }
  }
}

Adjust retention periods based on your regulatory requirements. See the Retention Policy Matrix for regulation-specific guidance (FISC: 7 years, ISMAP: 1 year, SOC 2: 1 year).

KQL Query Examples

# All failed access attempts
fsxn.result: "Failure"

# Specific user activity
user.name: "admin@corp.local" AND fsxn.operation: "WriteData"

# File access by path pattern (sensitive directories)
fsxn.path: *confidential* OR fsxn.path: *restricted*

# Operations by SVM (using event.code for Windows Event ID)
fsxn.svm: "svm-prod-01" AND event.code: "4663"

# Delete operations (potential data exfiltration)
fsxn.operation: "Delete" AND NOT user.name: "backup-svc@corp.local"

Kibana SIEM Integration

With ECS mapping, you can create detection rules in Kibana Security:

Rule	Condition	Severity
Mass file deletion	`fsxn.operation: "Delete"` count > 50 in 5m	Critical
After-hours access	`@timestamp` outside business hours + `fsxn.result: "Success"`	Medium
Unauthorized path access	`fsxn.path: restricted` AND `fsxn.result: "Failure"`	High
New user first access	`user.name` not seen in last 30 days	Low

Self-Hosted Option: Data Never Leaves Your VPC

For maximum data sovereignty, deploy Elasticsearch in your VPC:

Lambda (VPC) ──→ Elasticsearch (VPC, private subnet)
                        │
                        ▼
                 Kibana (VPC, private subnet + ALB)

Deploy Lambda in the same VPC with security group rules allowing port 9200 to the Elasticsearch cluster.

⚠️ Network path consideration: If this Lambda also reads from the FSx for ONTAP S3 Access Point, a NAT Gateway is required (S3 AP access from VPC requires internet-routed path — see S3 AP Specification). For the self-hosted pattern, consider splitting into two Lambdas: one outside VPC for S3 AP reads, and one inside VPC for Elasticsearch writes, connected via SQS.

Cost Analysis

Component	Monthly Cost (10 GB/month logs)
Lambda (5-min polling)	~$3
EventBridge Scheduler	~$1
Secrets Manager	~$1
Elastic Cloud Serverless	~$95 (Standard tier)
Self-hosted (3× r6g.large)	~$300 (EC2 + EBS)

Monthly Log Volume	Elastic Cloud	Self-Hosted
1 GB	Free trial	EC2 cost only
10 GB	~$95/month	~$300/month
100 GB	~$500/month	~$900/month

Self-hosted has no per-GB ingestion cost but requires infrastructure management. Elastic Cloud Serverless eliminates ops burden at the cost of per-GB pricing.

Gotchas & Lessons Learned

#	Discovery	Impact
1	API Key must be in Encoded (Base64) format, not the raw id:key pair	Auth failures if stored incorrectly
2	Elastic Cloud Serverless returns data instantly (no indexing lag)	Faster validation than some competitors
3	Daily indices (`fsxn-audit-YYYY.MM.DD`) require ILM policy for cleanup	Unbounded storage growth without ILM
4	`fsxn.path` needs both `text` and `keyword` sub-field for full-text + exact match	Missing queries if only one type
5	Bulk API recommended max ~10MB per request	Lambda batches accordingly
6	No built-in Firehose destination — Lambda direct delivery only	Simpler architecture, but no Firehose buffering

Data Sovereignty & Classification

Audit logs contain fields classified as PII (user.name) and sensitive business data (fsxn.path). For data sovereignty deployments:

Self-hosted: Full control — implement field-level security in Elasticsearch to restrict PII access by role
Elastic Cloud: Use Elastic's field-level security + ILM retention policies
Redaction path: Deploy OTel Collector (Part 5) with attributes/redact processor before Elasticsearch

See the Data Classification Guide for field-by-field PII classification and recommended handling patterns per regulatory framework (APPI, GDPR, FISC, ISMAP).

Production Readiness

This integration follows the project's Production Readiness Levels:

Level	What You Get	Go/No-Go to Next
Level 1 (this Quick Start)	Audit poller + DLQ	Logs arrive, checkpoint advances, DLQ empty 24h
Level 2	+ Kibana dashboard + alerts	SLOs met 7 days, security review done
Level 3	+ DynamoDB ledger + poison-pill	SLOs met 30 days, compliance pack
Level 4	+ OTel Collector + redaction	Multi-backend, PII redaction, DR tested

Full criteria: Pipeline SLO Definitions | DLQ Replay Runbook

Enterprise scale: For multi-account deployments, see the StackSets deployment guide. For DR, see Cross-Region Replication. For compliance evidence collection, see the Compliance Evidence Pack.

CloudFormation Templates

Template	Purpose	Key Parameters
`template.yaml`	FSx audit log poller	S3AccessPointArn, ElasticApiKeySecretArn, ElasticEndpoint
`template-ems.yaml`	EMS webhook handler	ElasticApiKeySecretArn, ElasticEndpoint
`template-fpolicy.yaml`	FPolicy EventBridge handler	ElasticApiKeySecretArn, ElasticEndpoint, EventBusName