Yoshiki Fujiwara(藤原善基)@AWS Community Builder for AWS Community Builders

Posted on Jul 20

Mackerel's Log Feature Just Opened in Beta — Here's What It Takes to Wire It Into an OTLP Pipeline

#aws #observability #opentelemetry #serverless

TL;DR

Mackerel — Hatena's Japan-origin observability platform — opened its log feature as public beta on July 16, 2026. In response, this repository added Mackerel integration for shipping FSx for ONTAP file access audit logs, EMS events, and FPolicy events via the existing Lambda/OTel Collector pipeline.

The OTel Collector path worked immediately — just add an exporter config entry. But the direct-send path (Lambda → vendor OTLP endpoint, no Collector) exposed two compatibility gaps:

Mackerel requires a custom Mackerel-Api-Key header, not Bearer/Basic
Mackerel's OTLP endpoint only accepts Protobuf, rejecting OTLP/JSON outright

This article covers the generic auth header primitive and limited Protobuf encoder added to close those gaps. Both delivery paths are confirmed E2E against a live Mackerel organization, but since the log feature is still in beta, production deployments should use the Collector path and should not treat Mackerel as the sole audit trail destination.

Scope: This article covers shipping FSx for ONTAP file access audit logs, EMS events, and FPolicy events to Mackerel via the existing Lambda/OTel Collector integration. It does not cover Collector high availability, Lambda retry/DLQ design, NAT Gateway routing, or log-loss recovery — see the base article (Part 5) and repository operations docs for those topics.

FSx for ONTAP
├─ File access audit (EVTX/XML)
├─ EMS events
└─ FPolicy events
         │
         ▼
Lambda: builds OTLP Logs Data Model
         │
         ├──▶ OTel Collector ──▶ Mackerel [recommended]
         │
         └──▶ Direct to Mackerel [verification path]
              - Mackerel-Api-Key header
              - Accept: */*
              - OTLP/Protobuf required

GitHub: Yoshiki0705/fsxn-observability-integrations
New files: integrations/mackerel/, integrations/otel-collector/otel-collector-config-mackerel.yaml
Modified: integrations/otel-collector/lambda/{handler,ems_handler,fpolicy_handler}.py

This is a standalone entry in the Serverless Observability for FSx for ONTAP series — it's not a continuation of the incident-response article, and doesn't assume you've read it. It builds on Part 5 (the vendor-neutral OTel Collector pattern), since that's exactly the mechanism a new backend plugs into.

Why Mackerel, and Why Now

This series has already shipped FSx for ONTAP audit logs to 9 observability platforms — Datadog, Splunk, Grafana Cloud, Honeycomb, and others — all confirmed E2E against real accounts. A few readers, mostly from Japan, have asked why Mackerel wasn't on that list. Fair question, and one I'd been wanting to answer for a while.

The short answer: it wasn't this repository's gap to close. Mackerel simply didn't have a log-sending feature yet. It's offered OpenTelemetry-based tracing (APM) for a while, but logs were a gap in Mackerel's own product.

There's another direct trigger behind this article. At SRE NEXT 2026, I had a chance to casually catch up with some longtime acquaintances on the Mackerel team and trade notes on Observability for Amazon FSx for NetApp ONTAP. That particular conversation didn't lead anywhere specific on its own, but the conference as a whole gave me useful ideas and motivation for infrastructure-focused development, which fed directly into picking this integration up.

That changed on July 16, 2026, when Mackerel opened its log feature as a public beta. A feature I'd been waiting on finally showed up, which is the actual reason this article exists.

That said, "just opened in beta" is worth taking seriously. The feature itself is new and explicitly non-GA — a genuinely different situation from adding an eighth or ninth already-mature vendor without much thought. This article walks through what actually broke while wiring it up, and why it still sits in a separate section instead of being quietly folded into the main vendor table.

Vendor-neutral note: this article evaluates Mackerel's OTLP log ingestion on its technical merits — the same way this repository's vendor-comparison.md treats Datadog, Grafana, Splunk, and the other 9. No platform in this comparison is framed as superior to another; each fits a different context, and the goal here is documenting what's actually supported, not ranking who's "better."

When this integration is a good fit:

You already use Mackerel for monitoring/APM
You want to integrate FSx for ONTAP events into your existing incident investigation workflow
You use an OTel Collector as a shared telemetry router
You can evaluate a beta feature in a non-production or supplementary capacity

When to consider other options for now:

Regulatory/audit requirements demand guaranteed data retention
This would be your sole security log destination
Log loss is unacceptable and formal support terms are required
Strict data residency or contract-path conditions apply

What Mackerel's Beta Actually Supports

Before writing any integration code, the first step was reading Mackerel's own documentation rather than assuming logs would work like traces. As of July 2026, the log-sending method documented by Mackerel's official help is OpenTelemetry-based. This article uses OTLP/HTTP accordingly. Some things turned out to be shared, some didn't:

	Mackerel Tracing (APM)	Mackerel Logs (beta, 2026-07-16)
Protocol	OTLP/HTTP	OTLP/HTTP (same)
Endpoint	`https://otlp-vaxila.mackerelio.com`	Same endpoint
Auth	`Mackerel-Api-Key` header (Write scope)	Same header, same scope
Required extra header	`Accept: /` (documented as required by Mackerel; the reason isn't publicly documented)	Same requirement
Grouping key	—	`service.namespace` + `service.name` resource attributes
Retention	(APM's own terms)	30 days planned at GA; beta operates under the same window but with no guarantee
Collector batch config	—	~3.5MB (`sending_queue.batch.max_size: 3500000` bytes). This is a Collector-side setting matching Mackerel's official config example, not an absolute API max request size

The practical upshot: if you already have an OTel Collector sending traces to Mackerel, adding logs is a second exporter pipeline entry, not a new integration pattern.

The Architecture Question: Collector or Direct-Send?

The Collector path is recommended for most deployments. Direct-send is an option for minimal setups without a Collector, or for verifying protocol compatibility. Mackerel's own documentation also recommends placing a Collector in between for filtering and masking.

Criteria	Recommended Path
Production use, masking, multiple destinations, retry control	Collector
Minimize components in a small environment	Direct-send
Just want to try Mackerel integration first	Local Collector test
Handling security audit logs	Collector (filter sensitive attributes)

This repository's OTel Collector integration already ships FSx for ONTAP audit/EMS/FPolicy logs to Datadog, Grafana Cloud, and Honeycomb simultaneously from one Lambda codebase — the Lambda builds a backend-neutral OTLP payload, and the Collector's exporters config decides where it goes. Adding Mackerel as a fourth backend to that pattern needed zero Lambda changes:

# otel-collector-config-mackerel.yaml
# NOTE: This repo's verified Collector version (0.152.0) uses otlp_http. Older
# versions may require otlphttp (no underscore). Check your version's docs.
exporters:
  otlp_http/mackerel:
    endpoint: https://otlp-vaxila.mackerelio.com
    headers:
      Accept: "*/*"
      Mackerel-Api-Key: ${env:MACKEREL_APIKEY}
    sending_queue:
      batch:
        max_size: 3500000
        sizer: bytes
service:
  pipelines:
    logs:
      receivers: [otlp]
      exporters: [otlp_http/mackerel]

That's the whole integration for the Collector-mediated path. But this repository also supports a direct-send path — Lambda posts straight to a vendor's OTLP endpoint, skipping the Collector entirely — used today for Grafana Cloud via AUTH_MODE=basic. Checking whether that path also worked for Mackerel is where things got more interesting.

The Gap: A Header Mackerel Needs That the Direct-Send Path Couldn't Send

"Add an exporter and you're done" turned out not to cover every path. The direct-send auth logic in handler.py, ems_handler.py, and fpolicy_handler.py only supported two modes:

if AUTH_MODE == "basic":
    encoded = base64.b64encode(token.encode("utf-8")).decode("utf-8")
    auth_headers = {"Authorization": f"Basic {encoded}"}
else:
    auth_headers = {"Authorization": f"Bearer {token}"}

Mackerel's auth is neither. It's a bare custom header, Mackerel-Api-Key: <token>, with no Bearer or Basic wrapping. At that point, direct-send to Mackerel without a Collector simply wasn't possible.

The fix is a generic primitive rather than a Mackerel-specific branch:

AUTH_MODE = os.environ.get("AUTH_MODE", "bearer")  # "bearer", "basic", or "header"
AUTH_HEADER_NAME = os.environ.get("AUTH_HEADER_NAME", "Authorization")
EXTRA_HEADERS_JSON = os.environ.get("EXTRA_HEADERS_JSON", "")

# ...
elif AUTH_MODE == "header":
    auth_headers = {AUTH_HEADER_NAME: token}
# ...
if EXTRA_HEADERS_JSON:
    extra_headers = json.loads(EXTRA_HEADERS_JSON)
    auth_headers = {**(auth_headers or {}), **extra_headers}

AUTH_MODE=header sends the secret verbatim under any header name you specify. Mackerel happens to be the first consumer, but nothing about the option itself references Mackerel. EXTRA_HEADERS_JSON covers Mackerel's required Accept: */*, which isn't authentication at all — just a static header their backend depends on.

EXTRA_HEADERS_JSON constraint: This is for static, non-secret headers like Accept only. Do not specify auth headers (Authorization, Mackerel-Api-Key, etc.), Content-Type, Content-Length, or Host — these are controlled by AUTH_MODE/OTLP_CONTENT_TYPE settings or set automatically by the HTTP library.

The same fix went into ems_handler.py and fpolicy_handler.py too. The CloudFormation template passes these env vars to all three Lambda functions, so patching only one would have left the other two silently ignoring AUTH_MODE=header if someone set it.

12 new unit tests cover this across the three handlers (4 each in handler.py, ems_handler.py, and fpolicy_handler.py): custom-header auth, EXTRA_HEADERS_JSON merging, the case where no API key secret is configured but extra headers still apply, and invalid EXTRA_HEADERS_JSON producing a startup warning with extra headers disabled. cfn-lint and gitleaks run clean on the modified template and new files.

The Second Gap: Auth Was Fixed, But the Payload Format Wasn't

Fixing auth wasn't the finish line, though. The unit tests mock the HTTP layer entirely, so they passed without catching what a real API key call turned up next:

OTLP endpoint error 400: {"code":400,"message":"json is not supported yet"}

_send_otlp_payload always sends OTLP/JSON with Content-Type: application/json, but Mackerel's OTLP endpoint accepts Protobuf only and rejects JSON outright. The Collector-mediated path never surfaced this: the Collector decodes received OTLP data into its internal data model, then re-serializes via the otlphttp exporter whose default encoding is Protobuf (proto). So even though the Lambda sends OTLP/JSON to the Collector, the Collector sends OTLP/Protobuf to Mackerel. The direct-send Lambda code simply never had a Protobuf encoder.

This project's policy is to include no additional Python dependency packages in the Lambda deployment package. Only boto3 and urllib3 — both available in the Lambda Python runtime environment — are used; protobuf/opentelemetry-proto are not bundled. Adding those packages for one vendor felt disproportionate, so the fix is a small hand-rolled encoder (otlp_protobuf.py). This encoder is not a general-purpose OTLP implementation. It covers only the limited data structures this repository's FSx for ONTAP log builders produce: timeUnixNano, severityNumber, severityText, body.stringValue, attributes (stringValue only), resource attributes (stringValue only), and InstrumentationScope. It does not support the full OTLP Logs field set or all AnyValue types, and is not intended for reuse outside this project. Field numbers came straight from the official OTLP proto definitions (Apache License 2.0); the encoder's byte output was cross-checked against the official opentelemetry-proto generated Python classes in a throwaway virtualenv before it touched a real account. We monitor official proto definition updates and maintain compatibility tests against the generated classes.

OTLP_CONTENT_TYPE = os.environ.get("OTLP_CONTENT_TYPE", "json")  # "json" or "protobuf"

if content_type == "protobuf":
    headers = {"Content-Type": "application/x-protobuf"}
    body = encode_logs_data(payload)
else:
    headers = {"Content-Type": "application/json"}
    body = json.dumps(payload).encode("utf-8")

After adding OTLP_CONTENT_TYPE=protobuf, calling handler.py's actual build_otlp_payload and _send_otlp_payload functions — the same code the Lambda runs, not a reimplementation — against the real Mackerel API key succeeded. For the two sample records sent (ReadData/Success and Delete/Access Denied), this repository's generated audit attributes — operation type, result, file path, SVM name, user information, event timestamp — were confirmed searchable on Mackerel's log search UI.

Unit tests verify auth headers, Content-Type branching, and encoding output. However, they cannot detect vendor-side acceptance behavior, rate limits, network path issues, or search-indexing delays — those require E2E tests against the real endpoint, which we run separately. 20 more unit tests were added: 3 per handler for the new content-type path, plus 11 in a dedicated test_otlp_protobuf.py, bringing the OTel Collector integration's test suite to 110 passing tests.

Verifying one delivery path for a vendor doesn't verify another. Auth headers, payload wire format, and network reachability can each fail independently, and a unit-test suite that mocks the HTTP layer won't catch a payload-format mismatch that only a real vendor endpoint enforces.

Why This Still Isn't in the "9 Vendors, All Verified" List

Every other vendor in this repository has a checkmark for a reason: an actual payload was sent to a live account and confirmed to arrive. Mackerel now has that checkmark too. Sample FSx audit log payloads showed up in Mackerel's log search UI with this repository's generated audit attributes confirmed searchable, through both the Collector-mediated path and the direct-send path once OTLP_CONTENT_TYPE=protobuf was in place. This repo's README.md reflects that:

✅ E2E verified (open beta) — Confirmed end-to-end against a live account, but the backend platform's own feature is itself in open beta (no data retention guarantee, unscheduled maintenance possible) — see the linked integration README for beta constraints.

Verification Detail	Value
E2E verified	2026-07-18
Collector version	0.152.0 (OpenTelemetry Collector Contrib)
Region	ap-northeast-1
Repository commit	`main` branch HEAD at time of publication

So the code-side gate is cleared. There's a second reason the integration still doesn't sit in the main table, though, and it's not about this repository's code at all — it's about what Mackerel itself has published about its own feature:

No data-retention guarantee during the beta period
Unscheduled maintenance is possible
GA is planned for fall 2026, but the exact date isn't fixed yet

Cost note: As of July 2026, the beta period is free. GA pricing is planned as ingest-volume-based billing. Pricing details, measurement units, and free-tier terms may change after this article's publication — check Mackerel's official pricing page for current terms.

Folding this integration quietly into the "supported vendors" table with the same checkmark as the other 9 would let someone deploying it for security alerting (this repository's automated incident-response module, for instance) reasonably assume the same confidence level as Datadog or Splunk. That assumption would be wrong, not because of anything this repo's code does, but because the platform underneath it is explicitly pre-GA. So the integration stays in a separate "Emerging / Beta Vendors Under Evaluation" section of vendor-comparison.md, marked as E2E-verified-but-beta, until Mackerel's own log feature reaches GA.

One thing worth flagging for anyone reproducing this kind of verification: during investigation we also inspected GraphQL requests used internally by the browser UI, but we have not confirmed these are public/stable APIs. This repository's permanent E2E verification does not depend on them — final confirmation uses the official UI's log search screen combined with the Collector's own send metrics (otelcol_exporter_sent_log_records). Internal implementation details on Mackerel's side may change without notice.

Security Considerations

FSx for ONTAP audit logs may contain usernames, file paths, share names, and client IP addresses — content that may constitute sensitive or personally-identifiable information depending on your organization's policies. Before sending production data to any external observability service:

Confirm your organization's data classification, storage location, access control, and retention requirements
Use OTel Collector filter/transform processors to remove or mask sensitive attributes as needed (see this repository's PII redaction cookbook)
Do not use beta-period Mackerel as the sole storage destination for audit trails

On EXTRA_HEADERS_JSON specifically:

Never put secret values in it — API keys belong in Secrets Manager
Never put API keys directly in CloudFormation parameters or Lambda environment variables
Never log full header contents at runtime

Troubleshooting: 4-Layer Isolation

When issues occur, isolate across these four layers:

Layer	Target	What to Check
1. Generation	Lambda	OTLP record count, required attributes, timestamp, severity, resource attributes. If enabling debug logging, avoid outputting sensitive info like file paths or usernames
2. Transmission	Lambda / Collector exporter	HTTP response code, timeouts, retry results (400, 401, 403, 429, 5xx)
3. Acceptance	Collector metrics / OTLP response	`sent`/`failed` log records, partial success, drops
4. Consumption	Mackerel UI	service attribute match, search time range, filter conditions, indexing delay

Distinguishing Collector/HTTP transmission failures (layers 2-3) from post-acceptance search/display issues (layer 4) is critical. If Mackerel rejects with HTTP 400, that's a layer 2-3 problem, not layer 4. See the troubleshooting table in integrations/mackerel/docs/ for detailed guidance.

Recommended Adoption Steps

For evaluating this beta integration in stages:

E2E verification with sample logs using a local Collector (test-local-mackerel.sh)
Send non-production FSx for ONTAP audit logs in limited scope
Evaluate send volume, data loss, searchability, and sensitive-information handling
In production, implement dual-send to your existing log platform (CloudWatch Logs, S3, existing vendor)
During the beta period, do not use Mackerel as your sole audit trail destination
Re-evaluate for primary-path promotion once data retention terms, support conditions, and pricing are finalized at GA

What's Actually Usable Today

Both paths below are now confirmed working end-to-end:

# Collector-mediated (recommended, zero Lambda changes)
cp integrations/otel-collector/.env.mackerel.example .env.mackerel
# edit .env.mackerel with a Write-scoped Mackerel API key
bash integrations/otel-collector/scripts/test-local-mackerel.sh

# Direct-send (skips the Collector; needs AUTH_MODE=header AND OtlpContentType=protobuf)
aws cloudformation deploy \
  --template-file integrations/otel-collector/template.yaml \
  --stack-name fsxn-otel-integration \
  --parameter-overrides \
    OtlpEndpoint=https://otlp-vaxila.mackerelio.com \
    AuthMode=header \
    AuthHeaderName=Mackerel-Api-Key \
    ExtraHeadersJson='{"Accept":"*/*"}' \
    OtlpContentType=protobuf \
    ApiKeySecretArn=<secret-arn> \
    ... \
  --capabilities CAPABILITY_NAMED_IAM

OtlpContentType=protobuf is not optional for Mackerel — without it, the send fails with the JSON-rejection error described above, even with correct auth.

Full setup guides (bilingual) are in integrations/mackerel/docs/, including a troubleshooting table that specifically tells you to isolate Collector-side failures from Mackerel-side rejections before debugging the wrong layer.

Wrapping Up, and What's Next

So that's where Mackerel's log beta stands today. Getting there took two fixes that aren't visible when only the Collector-mediated path is tested — a custom auth header format, and a JSON-vs-Protobuf mismatch — and neither one surfaced until real credentials hit the direct-send path. A green unit-test suite doesn't tell you that.

The AUTH_MODE=header, EXTRA_HEADERS_JSON, and OTLP_CONTENT_TYPE=protobuf options added here are not Mackerel-specific code — they're reusable primitives for any future OTLP/HTTP backend that requires a custom auth header or Protobuf-only ingestion.

A few things to watch going forward:

Watch for Mackerel's GA announcement for its log feature (data retention terms, pricing) — this article and the linked docs will be updated, not silently left stale, once that happens.
Once GA lands with confirmed data retention conditions, support terms, and pricing, move Mackerel from "Emerging / Beta Vendors" into the main "Supported Vendors" comparison table.
If you're running Mackerel already, the setup guide's sample OTLP payload is a good starting point to validate your own organization's setup independently.

Mackerel's log feature still has some growing to do before GA, and I'm looking forward to seeing where it lands. Hope this was useful to someone wiring up the same thing.

DEV Community