Fernando Azevedo

Posted on Oct 8 Edited on Oct 12 Originally published at fernando.moretes.com

Notebook off the cluster: migrating to Spark Connect on EMR on EKS

#awscloud #emroneks #sparkconnect #kubernetes

The 24 September 2026 announcement fits in one line: Amazon EMR on EKS now offers interactive Spark sessions through Spark Connect, from emr-7.14.0 onward. The migration it enables does not fit in one line. Pulling the kernel out of the cluster and leaving only the driver there is a change of protocol, of network surface and of authorization model at the same time, and each of those three has its own way of failing at 2 AM.

The starting point: the notebook that lived inside the cluster

The previous pattern for interactive exploration on EMR on EKS was a managed endpoint of type JUPYTER_ENTERPRISE_GATEWAY. The kernel runs on the cluster, Livy mediates, and session state lives in the same pod that interprets the code. It works, and it charges three prices that only show up in operations.

The tooling is imposed: whoever writes PySpark gives up VS Code with linting, type checking and the same pyproject.toml as the production repository, and starts writing in a browser tab. Debugging is print, not a breakpoint.

The deploy cycle is fake: the code explored in the notebook is not the code that ships. It gets copied and adapted, and the adaptation is where the defect nobody reviewed is born.

The maintenance window contaminates analysis: an EKS add-on upgrade, a node group rotation, an AMI change, all of it kills the kernel of whoever was mid-investigation, because the kernel is a pod in your cluster.

None of those three problems is about Spark. They are coupling problems between the development environment and the execution plane. That coupling is exactly what Spark Connect cuts: the client is a thin PySpark library speaking gRPC to a server in the driver. What crosses the network is a logical plan, not bytecode.

The journey, in the order it actually happens

1. Inventory what the session will touch: Before any create-security-configuration, list the S3 buckets, Glue Data Catalog databases and subnets the exploration needs. The endpoint execution role is the access ceiling for everyone who connects to it, deciding it later means deciding it wrong.
2. Provision the AWS Load Balancer Controller and a private subnet: Spark Connect routes gRPC through an internal Network Load Balancer, and the controller is what creates it. A cluster without the controller installed, or without a private subnet in the VPC, never gets to create the endpoint, a prerequisite failure, not a Spark misconfiguration.
3. Create the security configuration with a system namespace: Spark Connect infrastructure runs in a system namespace, separate from the virtual cluster user namespace. And the relationship is 1:1, one security configuration per virtual cluster, no reuse. That reshapes your multi-tenant design before you write the first cell.
4. Recreate the virtual cluster with --session-enabled true: Without that flag and without --security-configuration-id, the virtual cluster will not accept a Spark Connect endpoint. For anyone already running StartJobRun in production, this is the step with real risk: it is a new resource beside the old one, not an in-place change.
5. Create the first endpoint and wait for the NLB: create-managed-endpoint --type SPARK_CONNECT with --session-idle-timeout-in-minutes. The first endpoint on each EKS cluster takes minutes longer: it provisions the internal NLB and the VPC interface endpoint (PrivateLink) that every later endpoint reuses.
6. Pin the client version in the repository, not in the README: pyspark[connect]==3.5.8 for emr-7.14.0; the emr-spark-8.1.0 line pins the client at pyspark[connect]==4.0.2. A version mismatch yields connection errors or odd behaviour, and a Python UDF with a different Python minor fails with PYTHON_VERSION_MISMATCH.
7. Automate token renewal: GetManagedEndpointSessionCredentials returns a short-lived token used as x-aws-proxy-auth in the auth proxy URL. When it expires, gRPC calls fail with an authentication error and you need a fresh SparkSession, a helper in the repository, not a wiki instruction.

What changes when the protocol becomes gRPC

Spark Connect's client-server architecture looks like an implementation detail and is the most operational change of all. The PySpark client builds an unresolved plan, serializes it and sends it over gRPC on TLS; the driver on EKS resolves, optimizes and executes. Three concrete consequences:

The session outlives your local process. The Spark context is persistent and spans cells and scripts, you blend local Python with remote operations. Ctrl+C on the client does not kill the driver, which is why sessionIdleTimeoutInMinutes (default 60) is the only brake between a distracted developer and a driver pod alive all night.

The network surface is new and internal. gRPC arrives through an internal NLB in a private subnet, with a VPC interface endpoint in front. Nothing exposed to the internet, but there is one more network component in the threat diagram, with its own Flow Logs, health checks and security group.

Anything that is not a DataFrame does not get through. Spark Connect supports DataFrame and SQL APIs in PySpark; RDD-based APIs are not supported. That is not a footnote asterisk: it is the triage criterion for which legacy job can be explored interactively and which must be rewritten first. Finding that out mid-migration costs two sprints.

From cell to executor: the path of one DataFrame operation

The client never talks to the driver directly: it goes through the control plane for URL and token, then through the auth proxy inside the cluster.

👤 Cliente, estação de trabalho ou notebook gerenciado

pyspark[connect] 3.5.8 (emr-7.14.0) (frontend)
SageMaker Unified Studio notebook gerenciado (frontend)

🔐 AWS, plano de controle emr-containers

CreateManagedEndpoint type=SPARK_CONNECT (security)
GetManagedEndpointSession Credentials, token (security)

📡 VPC, entrada privada de gRPC

VPC interface endpoint AWS PrivateLink (network)
NLB interno sub-rede privada (network)

🟧 AWS, seu cluster EKS (virtual cluster session-enabled)

Auth proxy x-aws-proxy-auth (security)
Spark driver servidor gRPC (pod) (compute)
Executores dynamicAllocation 3→5 (compute)

💾 Dados e evidência

Amazon S3 dados + logs do Spark (storage)
Glue Data Catalog metadados de tabela (data)

Flows

dev -> client: writes and debugs with breakpoints
dev -> studio: managed alternative
client -> api: describe: fetch authProxyUrl
client -> creds: short-lived token (SigV4)
client -> pl: gRPC/TLS, logical plan
studio -> pl: same session, another client
pl -> nlb: routes without leaving the VPC
nlb -> proxy: validates the session token
proxy -> driver: persistent Spark context
driver -> exec: schedules tasks as pods
exec -> s3: reads via the IAM execution role
driver -> glue: resolves table and schema
driver -> s3: s3MonitoringConfiguration

Three numbers to bring to the approval meeting

60 min: Default idle timeout. sessionIdleTimeoutInMinutes. It is the only automatic brake against a driver and executors alive after hours.
US$ 16,43: Per month of idle NLB. US$ 0.0225/h × 730h in us-east-1, before any NLCU. The NLB is only removed when the last session-enabled virtual cluster is deleted.
0 RDD: Legacy APIs that get through. DataFrame and SQL pass; RDD is not supported. That is the triage criterion for your job backlog before promising a migration.

IAM is the only access control, and that reshapes the namespace

Here is the decision that separates whoever migrated carefully from whoever will find the problem during an audit. Spark Connect endpoints on EMR on EKS do not support Lake Formation fine-grained access control and do not support Trusted Identity Propagation. The documentation is explicit: to enforce access control, use the IAM execution role associated with the endpoint.

In an environment with BACEN, PCI-DSS or LGPD in scope, that reads directly: the endpoint is the unit of authorization, not the user. If ten people from two different products share one endpoint, they share data permissions, and the CloudTrail record will show the execution role, not who typed the query. Individual traceability comes back through the session (each has its own id and runs as pods tagged by project and user), not through the identity of the principal that read S3.

The design consequence is resource multiplication. Since each security configuration has a 1:1 relationship with a virtual cluster, isolating by data domain means one virtual cluster per domain, each with its security configuration, its system namespace and its execution role. And the policy for whoever creates endpoints needs iam:PassRole with an iam:PassedToService condition equal to emr-containers.amazonaws.com, without that condition, you just handed data engineering a privilege-escalation primitive.

Before and after, with the third column nobody puts in the table

Criterion	Livy / JEG (`JUPYTER_ENTERPRISE_GATEWAY`)	Spark Connect (`SPARK_CONNECT`)	Local Spark on the laptop
Where the code is interpreted	Kernel inside the cluster; the IDE is the gateway's	Local client; only the logical plan goes over gRPC	All local, on sample data that does not represent production
Network entry point you now operate	EMR managed endpoint, no NLB of its own	Internal NLB + VPC interface endpoint, one per EKS cluster	None, and no auditable path to the real data
Fine-grained data access control	Paths with Lake Formation and identity propagation	IAM execution role only: no FGAC, no TIP	Personal credentials in `~/.aws`, the worst of both
What breaks first	EKS maintenance kills the kernel mid-analysis	Token expires and the `SparkSession` must be rebuilt	The job passes locally and fails at scale, on a Friday night

The cost that never shows up in the POC spreadsheet

Every cost is a maintenance cost, and interactive sessions are the workload category that most rewards silent waste. Do the math up front.

The base already existed: EMR on EKS charges an uplift on actual consumption: US$ 0.01012 per vCPU-hour and US$ 0.00111125 per GB-hour in us-east-1, on top of the EKS cluster (US$ 0.10/hour) and the EC2 underneath.

The endpoint in the documentation example: a 4 GB driver, three executors of 2 vCPU and 4 GB, dynamicAllocation up to five, consumes roughly 7 vCPU and 16 GB while open. That is about US$ 0.089/hour in uplift alone. Over 176 business hours a month, around US$ 15.60 per live session, before EC2.

The new component is the floor: the internal NLB is born with the first endpoint and is only removed when the last session-enabled virtual cluster is deleted. US$ 0.0225/hour is US$ 16.43/month running with zero sessions connected, plus NLCU.

The design that avoids the surprise is mundane and almost nobody does it: sessionIdleTimeoutInMinutes well below the default 60 for exploration endpoints, mandatory project and user tags so spend can be attributed in Cost Explorer, and an alarm on the count of ACTIVE endpoints, not on the invoice, which arrives too late.

The four failure modes I have seen in the wrong order: An expired token is an authentication failure, not a network one: gRPC calls drop and the team opens a VPC ticket. A divergent client version produces connection errors or odd behaviour, and a Python UDF with a different minor fails with PYTHON_VERSION_MISMATCH, neither shouts its cause. A security configuration cannot be deleted before every endpoint using it is deleted; a teardown script that ignores the order hangs. The API limit is low where it hurts: GetManagedEndpointSessionCredentials has a bucket of 25 refilling at 1 per second, and the aggregate ceiling across all EMR on EKS APIs is 200 at 20/s. A wrapper that requests a token per cell, with twenty people online, meets throttling before it meets scale.

Anti-patterns

One endpoint for all of data engineering: since authorization is the endpoint execution role, a shared endpoint means shared permissions, and CloudTrail records the role, not the person.
Treating Spark Connect as the production path: interactive sessions are for exploration and incremental development. Scheduled pipelines stay on StartJobRun, with retry, idempotency and job observability, not a session someone left open.
Keeping the 60-minute idle default in a cost-controlled environment: that is the value that makes sense for the service, not for your spend ceiling.
Promising a migration of RDD jobs: RDD-based APIs do not pass through Spark Connect. The inventory comes before the dated commitment.
Installing the AWS Load Balancer Controller in a hurry to unblock the endpoint: it now governs NLBs in your VPC. That is a platform decision, not a tutorial step.

Reading it through the Well-Architected pillars

security: The endpoint is the authorization boundary: with no Lake Formation FGAC and no Trusted Identity Propagation, the execution role defines everything. Scope the policy to arn:aws:emr-containers:...:/virtualclusters/<id>/endpoints/* and require iam:PassedToService equal to emr-containers.amazonaws.com on iam:PassRole. gRPC never leaves the VPC, internal NLB in a private subnet with PrivateLink in front.
reliability: The first endpoint on each EKS cluster takes minutes longer because it provisions the NLB and VPC endpoint; treat that as a bootstrap step, not as usage latency. An expired token requires a new SparkSession, so the client needs automatic renewal before it gets users.

Curation note: If I were running this migration on a financial platform, I would start with a single data domain, a new session-enabled virtual cluster, with an execution role that only reads that domain's bucket, and keep the Livy path alive in parallel for a quarter. It is not fear of new technology; it is that the absence of Lake Formation FGAC changes the isolation design, and that kind of change does not surface in a two-week POC. The lesson I already paid for: new exploration capacity always becomes permanent consumption, so I ship the endpoint together with the short timeout and the per-tag spend dashboard on the same day, adding controls after the team gets used to the comfort is a far worse conversation. And I would keep production on StartJobRun, no exceptions, because an interactive session is excellent at finding the answer and terrible at repeating it every day.

References

Verdict

Migrate, for exploration, with narrow scope and a dated review. Spark Connect on EMR on EKS solves a real and old problem: the developer goes back to their own IDE, with breakpoints, over production-scale data, while the driver stays inside the cluster the platform already operates and already monitors. It is worth the effort when three conditions hold together: the backlog is DataFrame and SQL in PySpark, execution-role access control is sufficient for your regulatory regime, and someone owns deleting idle endpoints. If any of the three fails, wait: with no FGAC and no identity propagation, an endpoint shared across data domains is an audit finding waiting for its date. And in no scenario does this replace StartJobRun in production, the gain here is in the discovery cycle, not the execution one.

Rating: Adotar com escopo controlado

Originally published at fernando.moretes.com. By Fernando F. Azevedo: Senior Solutions Architect.

DEV Community