Fernando Azevedo

Posted on Oct 20 Originally published at fernando.moretes.com

Retro: EMR Serverless on IPv6-only and the subnet left without DNS64

#awscloud #emrserverless #ipv6 #vpc

The dangerous question about IPv6 on EMR Serverless is not "do my jobs run on IPv6?", it is "what does each subnet in the application know how to do when a job asks for an address that only exists in IPv4?". The 10/16/2026 announcement is short: Spark and Hive can now run on dual-stack or IPv6-only subnets, at no additional cost, in every Region and release version of the service. The documentation is more honest than the announcement: the addressing mode is derived from the subnets, and getting IPv6 traffic to your data is your responsibility. This piece is a pre-emptive retro, an incident that did not happen, built from the documentation and the failure modes it lists itself. I would rather write the postmortem before than after.

The scenario: /22 subnets that no longer fit three applications

The starting point is what AWS itself uses as a planning example. Each EMR Serverless worker consumes an IP address in the subnet where it launches, through an ENI. An application with 4 vCPU workers and a 4,000 vCPU ceiling reaches 1,000 workers, meaning 1,000 ENIs. A /22 subnet has 1,024 addresses and 1,019 usable ones, because AWS reserves five. With three applications sharing the same subnets and a 12,000 vCPU account quota, the documentation's recommendation becomes 3,000 free addresses per subnet, in at least two Availability Zones.

In a bank's data platform this usually hits a wall: the VPC IPv4 block was negotiated with corporate networking years ago, does not overlap with on-premises and does not grow without an approval round. That is exactly the case IPv6 solves. A VPC IPv6 subnet is a /64, and the address conversation is over.

In the scenario, the team goes straight to IPv6-only on three subnets, one per AZ, created by an IaC module. Two were built by hand during the pilot and got DNS64. The third came from the module, which did not know DNS64 existed. No integration test caught it, because the test job ran twice and landed on the two good subnets both times.

The numbers that drive the decision

1.000: ENIs per application. 4 vCPU workers with a 4,000 vCPU ceiling: one IP per worker
1.019: Usable IPs in a /22. Barely fits one application; three need 3,000 per subnet
US$ 0,045/GB: NAT gateway processing. Pricing page example (US East Ohio): every byte through NAT64 pays

Timeline of the simulated incident

D-7: subnet swap: The team stops the application, runs update-application with the three IPv6-only subnets in --network-configuration and starts it again. The mode is evaluated at every start, so the change only applies after the restart. Two test jobs pass.
02:00: the nightly batch starts: About a third of the jobs fail with UnknownHostException when calling endpoints that only publish an A record. Each job runs in a single subnet picked from the set, and the ones that landed on the subnet without DNS64 cannot resolve anything that is not IPv6.
02:40: retries hide the pattern: The orchestrator's retry policy resubmits the jobs. Some pass because they land on another subnet. The failure looks intermittent, and intermittent becomes "flaky network" in the on-call's first hypothesis.
04:10: correlation by subnet: Someone opens each job run detail, which shows the subnet used, and crosses it with success and failure. 100% of the failures sit on the same subnet. modify-subnet-attribute --enable-dns64 fixes name resolution.
05:00: the second failure: An ingestion job with an old JDBC driver fails with UnresolvedAddressException. It carried -Djava.net.preferIPv4Stack=true in spark.driver.extraJavaOptions, a workaround from years ago. The documentation forbids forcing the IPv4 stack in any mode.
D+5: the bill arrives: Cost Explorer shows a jump in NAT gateway data processing. S3 reads, which used to leave through the gateway endpoint, now resolve through DNS64 to 64:ff9b::/96 and cross NAT64.

Root cause: EMR Serverless derives the addressing mode from the subnets and does not validate whether the network is ready for it. Each subnet's readiness (IPv6 CIDR, auto-assign, DNS64, a 64:ff9b::/96 route to the NAT gateway, a ::/0 route to the egress-only gateway) was not encoded in the IaC module, and one-subnet-per-job selection turned a deterministic defect into an apparently random failure.

Why the failure looked random: the addressing mode rules

It is worth understanding the rules before blaming the network. An IPv4-only subnet runs as it always did. A dual-stack subnet runs IPv4 until you turn on spark.emr-serverless.networking.ipv6.enabled=true in the application's spark-defaults (for Hive, hive.emr-serverless.networking.ipv6.enabled in hive-site). An IPv6-only subnet runs IPv6 with no property at all.

When the set mixes types, the whole application runs in the one mode every subnet supports. IPv4-only with IPv6-only is rejected at validation. IPv6-only with dual-stack runs everything IPv6-only, including the dual-stack subnets, which must then meet the same prerequisites. IPv4-only with dual-stack runs everything on IPv4.

Three details produce incidents:

The mode applies at start: a running application keeps the mode it started with. You fix the subnet and nothing changes until the next stop/start.
One job, one subnet: a job runs in a single subnet picked from the list. One bad subnet out of three fails a third of the time, and retries mask it.
Dual-stack does not prefer IPv6 by default: even with the opt-in on, the engine uses IPv4 until you set spark.emr-serverless.driverEnv.SPARK_PREFER_IPV6 and spark.executorEnv.SPARK_PREFER_IPV6. Setting those on an IPv4-only subnet kills the job.

The lesson: the addressing mode is derived state, not configured state. The API does not let you declare the mode; it lets you declare subnets and infers the rest. Derived state needs a test that proves the derivation on every subnet, not on average.

Where each byte of a job on an IPv6-only subnet leaves

The path depends on the destination's DNS record: AAAA leaves directly, A becomes 64:ff9b::/96 and pays NAT64.

🟧 AWS: subnet IPv6-only (uma por AZ)

Spark driver SPARK_PREFER_IPV6 automático (compute)
Executors 1 ENI IPv6 por worker (compute)

📡 VPC: resolução e rotas

Route 53 Resolver DNS64 na subnet (network)
NAT gateway rota 64:ff9b::/96 (NAT64) (network)
Egress-only IGW rota ::/0 (network)
Interface endpoints Glue, STS, KMS, Logs (security)

🌐 Destinos

S3 dual-stack s3.dualstack. (storage)
Endpoint só IPv4 JDBC legado, API pública (external)
Endpoint IPv6 AAAA publicado (external)

Flows

drv -> r53: DNS query
exe -> r53: DNS query
r53 -> nat: A record only: synthesizes 64:ff9b::
nat -> v4: translates to IPv4, billed per GB
exe -> eigw: destination with AAAA
eigw -> s3ds: data reads and writes
eigw -> v6: egress without ingress
drv -> vpce: catalog, credentials, keys

Network remediation: turning prerequisites into invariants

The DNS64 fix took one command. The remediation that prevents recurrence took more work, and it is the one that matters.

Subnet module with a contract: the IaC module now requires, for every subnet tagged as an EMR Serverless target, assign_ipv6_address_on_creation = true, enable_dns64 = true, a 64:ff9b::/96 route to the NAT gateway and a ::/0 route to the egress-only internet gateway. Auto-assign only applies to resources created after the change, so turning it on late does not fix an ENI that already exists.

Validation before apply: a pipeline check (policy as code, or a test that reads describe-subnets and describe-route-tables) fails the deploy if any subnet in the set breaks the contract. Validation that runs after the incident is a postmortem, not a control.

Per-subnet canary: since the job picks the subnet and you do not, I create a canary application with a single subnet, one per AZ. A minimal Spark job resolves and connects to the endpoints the batch depends on (catalog, STS, KMS, the data bucket, the source database) and fails loudly. It runs before the batch window and after any network change.

Clean JVM: a sweep across every sparkSubmitParameters and runtimeConfiguration looking for preferIPv4Stack. A 2019 workaround cannot decide the network stack in 2026.

None of this is sophisticated. Its cost is not writing it, it is keeping the contract when someone creates the fourth subnet eighteen months from now without reading this document.

The cost the announcement does not mention: NAT64 and S3

"No additional cost" is true for the feature. It is not true for the path the traffic now takes.

On an IPv6-only subnet, every destination that only publishes an A record gets a synthesized address in 64:ff9b::/96, and that route points to the NAT gateway. The NAT gateway bills per hour and per GB processed; the VPC pricing page uses US$ 0.045 per GB as its example in US East (Ohio). A platform reading 8 TB a night from S3 and sending it through NAT64 pays around US$ 360 a night in processing alone, roughly US$ 10,800 over 30 nights. The S3 gateway endpoint on the workload subnet does not catch this traffic, because the destination is no longer an IPv4 prefix.

S3 only serves IPv6 through its dual-stack endpoints (s3.dualstack.<region>.amazonaws.com), and the client has to ask for them explicitly. Do not trust it before measuring: compare the NAT gateway's BytesOutToDestination before and after the switch.

Other services need the same care. In the AWS IPv6 support table, Glue has dual-stack and a private endpoint with IPv6, but not the public endpoint; Lake Formation also only has IPv6 through its private endpoint. On IPv6-only, that means interface endpoints for catalog and permissions, or paid NAT64 on every call.

And security: the S3 documentation warns that policies with IPv4-only aws:SourceIp make clients incorrectly lose or gain access once they start using IPv6. Review every bucket policy before the switch, not after.

The three modes side by side

Criterion	IPv4-only	Dual-stack with opt-in	IPv6-only
IPv4 consumption	1 per worker	1 per worker, still consumed	Zero
Activation	-	`ipv6.enabled` on the application + `SPARK_PREFER_IPV6` on the job	Automatic from the subnets
IPv4-only destinations	Direct	Direct over IPv4	Only with DNS64 + NAT64, billed per GB
Incident risk	Exhausting the subnet at peak	Low: failure falls back to IPv4	High if any subnet or dependency is not ready
When to use	Address headroom and no IPv6 mandate	Transition and dependency validation	Compliance mandate or real exhaustion, with dependencies mapped

Well-Architected reading

security: An IPv6 address is globally unique: egress only through an egress-only gateway, never a ::/0 route to the internet gateway on worker subnets. EMR Serverless already refuses security groups with ports open to 0.0.0.0/0 or ::/0. Review aws:SourceIp in bucket policies and prefer aws:SourceVpce.
reliability: Spread subnets across at least two AZs, as the documentation recommends, but treat each one as its own failure unit: a canary per subnet and an alarm per subnet. Automatic retries without per-subnet correlation turn a deterministic defect into noise.

Anti-patterns this retro tries to prevent

Validating with two jobs: with three subnets and one picked per job, two green jobs prove nothing about the third.
Going straight to IPv6-only without a dependency inventory: every driver, SDK and endpoint the job touches must support IPv6 or go through NAT64; EMR Serverless does not change your code or third-party endpoints.
Turning on SPARK_PREFER_IPV6 as a global default: on an IPv4-only subnet the preference kills the job.
Fixing the subnet without restarting the application: the mode was fixed at start, and on-call concludes the fix did not work.

What I would do: I would start with dual-stack with the opt-in, on a new application, and keep the IPv6 preference on only for jobs whose dependencies I have already measured. IPv6-only waits for a mandate or real exhaustion, and only after two weeks of green per-subnet canaries and a NAT gateway bytes chart I can explain line by line. The lesson I took from other network migrations is that the defect is almost never in the new service, it is in the subnet someone created with the old module. That is why the contract lives in IaC, not in the wiki.

Lessons and recommendation

Adopt IPv6 on EMR Serverless when the IPv4 space in shared subnets already limits your vCPU ceiling or when there is a formal IPv6 requirement. Use dual-stack with the opt-in while there is any IPv4-only dependency that matters in volume; use IPv6-only when every dependency publishes AAAA or sits behind an interface endpoint, and NAT64 is limited to control calls. Either way, encode DNS64, routes and auto-assign in the subnet module, run a per-subnet canary and measure the NAT gateway before declaring the migration done. The feature is free; an unprepared network bills per GB.

Rating: Adote com dual-stack primeiro / Adopt wi

References

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

DEV Community