Stop moving data to Spark when your warehouse is already Snowflake

Six months ago, our Friday deployments involved babysitting a 40-node EMR cluster. We were dumping 2TB of Parquet files from Snowflake to S3, spinning up a massive Spark job to perform window functions, and then wrestling with COPY INTO commands to shove the results back into Snowflake for the BI team. If the network flickered or the spark.executor.memoryOverhead wasn’t tuned perfectly, we’d wake up to a PagerDuty alert at 3:00 AM. Total runtime: 45 minutes.

Last week, I deleted the entire Spark infrastructure. We refactored those pipelines into Snowpark Python stored procedures running directly inside the Snowflake warehouse. The same transformation now executes in 12 minutes. We stopped paying AWS for the cluster, stopped managing IAM roles for S3 buckets, and most importantly, we stopped debugging serialization errors between PySpark and the Snowflake connector.

The real problem

The industry narrative for years has been "Snowflake for storage, Spark for transformation." It was sound advice when Snowflake's Python support was a glorified UDF wrapper. But the paradigm has shifted. Data gravity is a real, measurable cost. Every byte you move out of your warehouse is a tax you pay in latency, egress fees, and maintenance overhead.

The "real" problem isn't performance—Spark can technically be faster if you have a massive, highly-tuned cluster. The real problem is the operational tax of managing a secondary compute engine. If your data lives in Snowflake, moving it to Spark is an admission of failure. You aren't just writing code; you’re managing a distributed system, a network layer, and a security boundary. Snowpark isn't about replacing Spark in every scenario; it’s about recognizing that for 95% of financial reporting and ETL workloads, the overhead of "external compute" is a liability, not an asset.

Photo by Yue Ma on Unsplash

Step 1: Rethinking the execution model

In Spark, you are responsible for the JVM, memory management, and shuffling. In Snowpark, you are writing Python code that translates into Snowflake's optimized SQL execution plan. The first step is stop thinking about RDDs and DataFrames as objects that live in memory. You are building a lazy evaluation graph that Snowflake compiles into a single, massive query plan.

If you try to write Snowpark exactly like PySpark, you will hit a wall. You cannot collect() a 50GB dataframe to your local machine to inspect it. You have to embrace the session.sql() and dataframe.show() patterns. Here is how we define a pipeline that replaces an old Spark job:

# The Snowpark approach: Keep it inside the warehouse
from snowflake.snowpark import Session

def main(session: Session):
    # Instead of reading from S3, we reference the table directly
    df = session.table("RAW.TRANSACTIONS")

    # Transformations stay in the engine
    result = df.filter(df["STATUS"] == "COMPLETED") \
               .group_by("USER_ID") \
               .agg(sum("AMOUNT").alias("TOTAL_SPEND"))

    result.write.mode("overwrite").save_as_table("ANALYTICS.USER_SPEND")

The difference here is that no data leaves the Snowflake boundary. You aren't serializing objects; you are building an expression tree that the query optimizer handles.

Step 2: Configuring the environment without the hell of JARs

One of the biggest time-sinks in Spark is dependency management. You’ve been there: java.lang.NoClassDefFoundError because a library version in your requirements.txt didn't match the one on the worker nodes. In Snowpark, you handle dependencies via packages in your stored procedure definition.

We pin our environment using a specific Snowflake package set, which ensures that the library versions are consistent across the warehouse nodes. You don't need to build a custom Docker image or manage a private PyPI mirror.

session.sproc.create_from_function(
    func=my_transformation_logic,
    name="PROCESS_SALES",
    is_permanent=True,
    stage_location="@MY_STAGE",
    packages=["pandas", "scikit-learn==1.2.2", "snowflake-snowpark-python"],
    replace=True
)

By explicitly pinning scikit-learn==1.2.2, we avoid the "it worked on my machine" nightmare that plagues Spark clusters. The Snowflake environment is isolated, immutable, and versioned.

Step 3: Handling the failure modes of the engine

Spark fails with OutOfMemoryError or ShuffleFetchFailed exceptions. Snowpark fails with standard SQL errors, which are significantly easier to debug. When a Snowpark job fails, you don't dig through YARN logs or Spark UI task histories. You look at the QUERY_HISTORY view in Snowflake.

If your Python code hits a limit, the error message tells you exactly which query failed, which line of SQL caused it, and why. Here is the configuration I use to prevent "runaway query" costs, which is a different kind of failure mode compared to Spark:

-- Set a warehouse-level limit to prevent runaway Snowpark code
ALTER WAREHOUSE COMPUTE_WH SET MAX_CONCURRENCY_LEVEL = 8;
ALTER WAREHOUSE COMPUTE_WH SET STATEMENT_TIMEOUT_IN_SECONDS = 3600;

In Spark, you manage the "cluster" size. In Snowpark, you manage the "warehouse" budget. If you want to scale, you don't add more nodes to a cluster; you scale the warehouse size (e.g., X-Small to Medium), and Snowflake handles the parallelism automatically.

Photo by Pankaj Patel on Unsplash

Lessons learned from production

After six months of running mission-critical financial pipelines, these are the cold, hard truths:

• Avoid UDFs if possible: User-Defined Functions in Snowpark are powerful, but they trigger a serialization layer that can slow down performance. If you can express your logic in Snowpark Dataframe API methods (which translate to SQL), do it. Use UDFs only for complex Python-native logic that cannot be vectorized.
• The "Small File" problem is gone: Spark struggles with small files because of metadata overhead. Snowflake doesn't care. You don't need to run a compaction job after every write. Snowflake’s micro-partitioning handles this natively.
• Debug via SQL: If your Snowpark job is hanging, don't try to look at the Python trace first. Run SELECT * FROM TABLE(INFORMATION_SCHEMA.QUERY_HISTORY()) WHERE ... and look for the underlying SQL queries being generated. 9 times out of 10, the "Python issue" is actually a SQL join that is missing a partition key.
• Memory is not infinite: Even though you aren't managing Spark executors, Python memory limits still exist in Snowpark. If you are doing to_pandas() on a massive table, you will crash the warehouse node. Always filter your data as much as possible before pulling it into memory.

Conclusion

Is Spark dead? No. If you are doing heavy-duty machine learning with iterative training on petabytes of unstructured text, you might still need the flexibility of a dedicated Spark cluster. But for 90% of data engineering—filtering, joining, aggregating, and transforming—Spark is overkill that introduces unnecessary complexity.

You are likely already paying for Snowflake, which is the most sophisticated distributed query engine on the planet. Why spend 40% of your engineering time building a bridge to a second, inferior engine?

Try it: Take one of your low-impact Spark jobs—the one that triggers the most PagerDuty alerts—and rewrite it in Snowpark. Don't worry about "performance tuning" initially. Just get the logic moved over. You’ll find that the time you lose in initial refactoring is dwarfed by the time you save in operational maintenance. Stop being a cluster administrator and start being a data engineer.

Cover photo by Tyler on Unsplash.