DEV Community

Developer213
Developer213

Posted on

Snowflake



def main():
    # Initialize Spark session
    spark = SparkSession.builder \
        .appName("SnowflakeToS3") \
        .config("spark.hadoop.fs.s3a.connection.timeout", "3600000") \  # 1 hour
        .config("spark.hadoop.fs.s3a.socket.timeout", "3600000") \  # 1 hour
        .config("spark.sql.sources.commitProtocolClass", "org.apache.spark.sql.execution.datasources.SQLHadoopMapReduceCommitProtocol") \ 
        .config("spark.sql.parquet.output.committer.class", "org.apache.parquet.hadoop.ParquetOutputCommitter") \
        .getOrCreate()
Enter fullscreen mode Exit fullscreen mode

Top comments (0)

Sentry image

See why 4M developers consider Sentry, “not bad.”

Fixing code doesn’t have to be the worst part of your day. Learn how Sentry can help.

Learn more

👋 Kindness is contagious

Please leave a ❤️ or a friendly comment on this post if you found it helpful!

Okay