augusto kiniama rosa

Posted on Jan 1 • Originally published at blog.infostrux.com on Aug 2, 2024

The Unofficial Snowflake Monthly Release Notes: July 2024

#snowflake #datasuperhero #data #newreleases

Monthly Snowflake Unofficial Release Notes #New features #Previews #Clients #Behavior Changes

Welcome to the fantastic Unofficial Release Notes for Snowflake! You’ll find all the latest features, drivers, and more in one convenient place.

As an unofficial source, I am excited to share my insights and thoughts. Let’s dive in! You can also find all of Snowflake’s releases here.

This month, we provide coverage up to release 8.28 (General Availability — GA). I hope to extend this eventually to private previews notices as well.

I would appreciate your suggestions on continuing to combine these monthly release notes. Feel free to comment below or chat with me on LinkedIn.

Behavior change bundles 2024_03 active by Default, 2024_05 enabled by default but able to be disabled and 2024_06 is available to be enabled. I DO NOT USUALLY call something out, but in 2024_5 , Snowflake made a breaking change that broke most Snowflake Terraform RBAC modules, and older terraform RBAC needs to be reworked!!!

What’s New in Snowflake

New Features

Snowflake Connector for PostgreSQL (Preview), allows you to load data into Snowflake from a PostgreSQL database and configure replication so that changes in your PostgreSQL database are replicated to Snowflake
Snowflake Connector for MySQL (Preview), allows you to load data into Snowflake from a MySQL database and configure replication so that changes in your MySQL database are replicated to Snowflake
Estimate the cost of Automatic Clustering (Preview), new system function, SYSTEM$ESTIMATE_AUTOMATIC_CLUSTERING_COSTS, that estimates the cost of enabling Automatic Clustering for a table and maintaining the table in a well-clustered state
Access to external network locations on AWS in the Gov region (GA), access to external network locations from function and procedure handlers for code deployed in the AWS Gov region
Support for Snowpark Container Services in custom budgets (GA), monitor the cost of compute pools associated with Snowpark Container Services
Snowflake Copilot (GA), an LLM-powered assistant that simplifies data analysis while maintaining robust data governance and seamlessly integrates into your existing Snowflake workflow available only on us-east-1, us-west-2, and eu-central-1
Support for sending webhook notifications to Slack, Microsoft Teams, and PagerDuty (GA)
Triggered tasks (GA), run when the related stream has new data
Managing Listings using SQL (GA), can now create, alter, describe, show, and drop the contents of a listing using SQL commands

Snowsight Updates

New homepage for Snowsight (Preview), new navigation menu, Updated search, Quick actions, and recently viewed
Snowflake Notebooks external access (Preview), support for external access within Notebooks
Network rules and network policies (GA), streamlined the user experience to simplify managing network access to your Snowflake account via SnowUI
Default warehouse for Notebooks workloads (Preview), SYSTEM$STREAMLIT_NOTEBOOK_WH is a new, Snowflake-managed warehouse that is provisioned in each account for running Notebooks. This default warehouse helps in reducing cluster fragmentation and optimizing your overall costs on Notebooks

Snowflake Applications

Support for shared external table and Iceberg tables, allows providers to share external tables and iceberg tables with consumers

Data Lake Updates

Iceberg tables: Support for Polaris Catalog, using a catalog integration configure for Polaris, you can do the following: Query a table in Polaris Catalog using Snowflake, and sync a Snowflake-managed Iceberg table with Polaris Catalog.

Streamlit Updates

External network access in Streamlit, create secure access to specific network locations external to Snowflake, and you can use that access from within the Streamlit app code. You can enable access through an external access integration

Data governance updates

Custom Data Classification (GA), the CUSTOM_CLASSIFIER class in the SNOWFLAKE.DATA_PRIVACY schema to enable data engineers to extend Snowflake’s data classification capabilities based on their own knowledge of the data
Data Classification of tables in a schema with Snowsight (GA), using Snowsight to classify tables in a schema
Object Tagging: Support added for replication and failover groups,
Data Quality and data metric functions (DMFs) (GA), uses DMFs to continuously monitor data quality metrics such as completeness, accuracy, uniqueness, and validity

SQL Updates

Wildcards are now supported in OBJECT constants, use wildcards in OBJECT constants. When you construct OBJECT data, you can use a wildcard by specifying {*}
New TYPE property for USER (Preview), TYPE property lets you differentiate between service and human users
GET_DDL function: Support for warehouses, GET_DDL function to get the DDL statement for recreating a warehouse
New function, ARRAYS_ZIP, returns an array of objects, each of which contains key-value pairs for an nth element in the input arrays
CREATE and ALTER commands for replication and failover groups: Support added for tags, support to set a tag on replication and failover groups
Account Usage: New SEARCH_OPTIMIZATION_BENEFITS view, view provides information about the number of partitions pruned specifically due to search optimization

Hybrid Tables

Hybrid tables: Changes to capacity quotas (Preview), default capacity quotas for hybrid table storage and requests are now enforced at the database level for all Snowflake accounts, also limits the number of databases within a single Snowflake account that can contain hybrid table

Machine Learning Updates (Cortex, ML, DocumentAI)

CORTEX_FUNCTIONS_USAGE_HISTORY view (GA), ability to query the usage history for the Cortex AI Large Language Model (LLM) Functions, while allows you to track usage for all Cortex AI LLM Functions and individual models
New Meta AI models available in Snowflake Cortex AI, added Llama 3.1 collection of multilingual large language models (LLMs), llama3.1–405b, llama3.1–8b and llama3.1–70b
Cortex Guard for Snowflake Cortex AI (GA), enables enterprises to easily implement safeguards that filter out potentially inappropriate or unsafe large language model (LLM) responses
Document AI delete documents, can now delete documents that were not used for training from a Document AI model build
Cortex Search (Preview), an LLM-powered assistant that simplifies data analysis while maintaining robust data governance and seamlessly integrating into your existing Snowflake workflow in us-east-1 and us-west-2
AI21 model available, AI21’s foundational model, jamba-instruct, is now available for serverless inference in Snowflake Cortex AI
Cortex Fine-tuning (Preview), offers a way to customize large language models for your specific task. Now you can fine-tune models through Snowsight without writing any SQL

Data Clean Rooms Updates

Sequenced template execution, use a template chain to define a sequence of templates to be executed in a particular order
Multi-factor authentication, for increased security, users are required to use multi-factor authentication (MFA) when signing in to the web app
Register objects in a managed access schema, collaborators can now register individual tables and views in a managed access schema (that is, a schema created with the WITH MANAGE ACCESS clause) without registering other objects in the schema
Support for additional region, eu-central-2
Acxiom Real ID integration, A new third-party connector integrates Acxiom Real ID, an identity solution, with the clean room environment, allowing multiple entities to collaborate without needing to directly expose or share sensitive data
Using developer APIs for provider activation, Collaborators can now use the developer APIs to send a consumer’s analysis results back to the provider for activation
User interface for custom templates enhancement, create a drop-down list of tables in the clean room without having to individually specify them
SQL Query template enhancement, allows the consumer to select a warehouse when running an analysis

Data Pipelines/Data Loading/Unloading Updates

Data pipelines: Support for Iceberg tables with dynamic tables and streams (Preview), ability to use Snowflake-managed Iceberg table as the source in dynamic tables
Dynamic Iceberg table (Preview), a new dynamic table type that stores query results as a Snowflake-managed Iceberg table
Snowpipe: New output in SYSTEM$PIPE_STATUS, when a pipe fails over, load history entries might continue to be replicated for the pipe, ensuring that changes from the last refresh operation are up to date
Dynamic tables: Support for incremental lateral flatten, use lateral flatten with incremental refresh by setting the refresh mode to INCREMENTAL

Security Updates

Trust Center: CIS Benchmarks scanner package (GA), CIS Benchmarks scanner package in the Trust Center
Authentication policies: New multi-factor authentication parameters, MFA_AUTHENTICATION_METHODS and MFA_ENROLLMENT determine options to MFA and if a user must be enrolled

Open-Source Updates

terraform-snowflake-provider 0.94.0 (user SDK, session parameters, roles and few others)
terraform-snowflake-provider 0.93.0 (focused in security changes and a few new goods)
Streamlit 1.37.0 (a few new features and bug fixes)
Modin (nothing released)
Snowflake VS Code Extension (Edit Snowflake connections.toml files and support for Snowflake Native App Framework)

Client, Drivers, Libraries and Connectors Updates

New features:

Go Snowflake Driver 1.11.0 (Go 1.22, driver configuration for China, bind structured types in queries, passcode with MFA token caching enabled., setting session variables in DSN, simpler solution to define structured objects using tags, mechanism to wrap each goroutine in custom code)
Ingest Java SDK 2.1.2 (Improved InsertRows performance, added or improved various logs for better observability,and fine-tuned channel and chunk sizes)
JDBC Driver 3.17.0 (improved logging, vector dimension in column metadata, getObject on vector columns, reading the connection information from a file, Java version 21, dynamic Max LOB size in metadata, JDBC connectivity diagnostics mode)
JDBC Driver 3.18.0 (Updated the netty library to version 4.1.111.Final, missing property setters in SnowflakeBasicDataSource, connection parameters to support backward compatibility for handling timezones, Picked a top-level domain for Snowflake hosts, Set the last query ID for all failed statements)
Snowflake Connector for Google Analytics Raw Data 1.5.2 (REFRESH_VIEWS procedure, which allows recreating flattened data view)
Snowflake Connector for Google Analytics Raw Data 1.4.1 (Reloads are now scheduled automatically for all properties, UPDATE_INGESTION_OPTIONS procedure which allows customizing the ingestion settings for certain properties)
ODBC Driver 3.4.0 (passing private key file content through SQLSetConnectAttr when using key-pair authentication, different top-level domains, such as .cn in China, maximum allowable LOB (large object) size)
PHP PDO Driver for Snowflake 3.0.1 (removed hardcoded domains)
.NET Driver 4.0.0 (BCR change with connection, changed connection pool behavior with multiple pools, connection string driven configuration, targeting to .netstandard 2.0, strong name signature to the driver assembly, ability to set the QueryTag parameter in the connection string and on the SnowflakeDbCommand object to mark the connection and command queries with a tag, new DISABLE_SAML_URL_CHECK parameter that disables checking if the SAML postback URL matches the host URL when authenticating with Okta)
Snowflake CLI 2.6.0 (Added the snow object create command, support for a title field in Streamlit definition in the snowflake.yml project file, --auto-compress flag to the snow stage copy command to enable gzip compression files during upload,native_app.application.post_deploy section to snowflake.yml schema to execute actions after the application has been deployed via snow app run, sql_script hook type to run SQL scripts with template support, support for --env command line arguments for templating, snow sql command now show query text before executing it)
Snowflake Connector for Kafka 2.3.0 (close Snowpipe Streaming channels in parallel, which significantly reduces time for rebalancing, new SnowflakeConnectorPushTime property in the metadata that represents the time when the message was pushed by the connector)
Snowflake Connector for Python 3.12.0 (default connection timeout to 10 seconds and the socket read time to 10 minutes for HTTP calls in file transfers, ability to connect to multiple domains, Optimized to_pandas() performance by using fully-parallel downloading logic)
Snowflake Connector for Spark 3.0.0 (Breaking changes with removal Advanced Query Pushdown, single artifact works with multiple Spark versions, JDBC to 3.17.0 to Support LOB, support for Spark 3.5.0)
Snowflake API for Python 0.11.0 (requires 8.27, client logging to the library to enhance debug ability, undrop support to the DynamicTable, Schema, and Table object types, enhanced support for the Grant object type with a few limitations, a few commands deprecated and have been renamed create_or_update to create_or_alter, delete to drop, undelete to undrop)
Snowpark Library for Python 1.20.0 (distributed tracing using open telemetry APIs, arrays_zip function, performance for binary column expressions and df._in, Improved error messages for write_pandas, Snowpark local testing updates, Snowpark pandas API Updates, support for Index.copy(), support for Index APIs: dtype, values, item(), tolist(), to_series() and to_frame(), support for DataFrames with no rows in pd.pivot_table and DataFrame.pivot_table, support for inplace parameter in DataFrame.sort_index and Series.sort_index)
Snowpark ML 1.6.0 (breaking changes, New Feature Store features, New Model Registry features, New modeling features, lots of changes)
SQLAlchemy 1.6.0 (SQLAlchemy 2.0 syntax)
Snowflake Connector for Google Analytics Aggregate Data 1.5.0 (Reports can now be configured to avoid sampling by shortening ingestion interval length)
Snowflake Connector for ServiceNow® V2 5.8.0 (The row_filter field in ENABLE_TABLE procedure now accepts arbitrary number of whitespace characters in filtering expression rather than allowing only single space between expression elements)
Snowflake Connector for ServiceNow® V2 5.7.0 (CHECK_ROW_COUNT, ENABLE_TABLE (without custom configuration parameters) and SHOW_REFERENCES_OF_TABLE can now be called in a user-owned task)
Snowflake Connector for ServiceNow® V2 5.6.0 (Row filtering)
Snowflake Connector for MySQL 6.3.0 (initial release)
Snowflake Connector for PostgreSQL 6.3.0 (initial release)

Bug fixes:

.NET Driver 4.0.0 (handling of date and time, sending SQL queries to the server by sending original queries, more reliable way of providing hosts in NONPROXYHOSTS parameter, support of double quotes in DB, SCHEMA, WAREHOUSE, ROLE connection string parameters, S3 clients by adding “https://” to ServiceUrl if it is missing, secret detector to better mask secrets when logging, SnowflakeDbParameter.DbType value, increasing maxHttpRetries to the default value for HTTP clients)
Go Snowflake Driver 1.11.0 (handling session expiration when executing long-running queries, issue OCSP failures when OCSP cache is disabled, reading arrow batches that contained integer columns whose size is smaller than 64b)
JDBC Driver 3.18.0 (retry backoff time could fall outside the minimum and maximum range, converting nested fields metadata in OBJECT columns, date files returned the wrong day when using the getString or getDate method, user permission check for a token file)
JDBC Driver 3.17.0 (nserting and reading timestamps assymetrically if a batch inserts a large number of columns, returning inconsistent timestamps_ltz between JSON and ARROW result sets, driver failed file pattern expansion on file not found in a different pattern)
Ingest Java SDK 2.1.2 (failover across deployments)
ODBC Driver 3.4.0 (caused the driver to become unresponsive in some cases when logging, error is returned when calling a SQL procedure that returns NULL, multi-statement queries returned incorrect query results when the multi-statement query had more than one USE command, memory leaks when reading environment variables, proxy settings in environment variables is not honored on Windows)
PHP PDO Driver for Snowflake 3.0.1 (Microsoft Windows not honoring proxy settings in environment variables)
Snowflake CLI 2.6.1 (clarified the error message returned when executing snow object create if a database is not defined for the connection, crash when save_logs is false and the log directory does not exist)
Snowflake CLI 2.6.1 (many bug fixes and message improvements)
Snowflake Connector for Kafka 2.3.0 (dependencies with known vulnerabilities)
Snowflake Connector for Python 3.12.0 (client_session_keep_alive_heartbeat_frequency in snowflake-sqlalchemy could make the connector unresponsive, incorrect private_key connection parameter type hint)
Snowflake Connector for Spark 3.0.0 (Removed the requirement of the SFUSER parameter when using OAUTH)
Snowflake API for Python 0.11.0 (stored procedure generated code)
Snowpark Library for Python 1.20.0 (bug where the SQL generated for lag(x, 0), many other bugs)
Snowpark ML 1.6.0 (Modeling bug fixes, Model Registry bug fixes)
SQLAlchemy 1.6.1 (internal project workflow with pypi publishing)
Snowflake Connector for Google Analytics Aggregate Data 1.5.0 (quoted identifiers in the GET_TROUBLESHOOTING_DATA procedure)
Snowflake Connector for ServiceNow® V2 5.8.0 (table reload row filter and column filtering now taken into account, row filter now works as expected for tables without a sys_updated_on column
Snowflake Connector for ServiceNow® V2 5.6.0 (performance of the migration script)
Snowflake Connector for MySQL 6.3.2 (Fixed handling of deleted columns in subsequent schema changes, compute warehouse not suspending in scheduled mode, validation of reserved column names to be case insensitive, MySQL tables failing replication on huge transactions)
Snowflake Connector for PostgreSQL 6.3.2 (Fixed handling of deleted columns in subsequent schema changes, compute warehouse not suspending in scheduled mode, validation of reserved column names to be case insensitive, MySQL tables failing replication on huge transactions)

Conclusion

Wow, there is some much in this release this month that its hard to even pick what to work on, from an ever increase in machine learning and genAI features like Cortex Search, Cortex Guard, Snowflake Copilot and new LLM models. New Snowflake connectors for PostGres and MySQL, and Snowflake Container Services are out in GA. The list is super long and hard to follow. Those though I found of note and I wanted to call attention to them. For us engineers, Copilot has a potential to make everyone more efficient but the connector for MySQL and PostGres has a potential as those are high used databases across the many companies.

I hope you continue to enjoy these articles. Thanks for reading my blog post.

I am Augusto Rosa, VP of Engineering for Infostrux Solutions. Snowflake Data Super Hero and Snowflake SME. You can follow me on LinkedIn.

Subscribe to Infostrux Medium Blogs https://medium.com/infostrux-solutions for the most interesting Data Engineering and Snowflake news.

Sources:

How I Cut 22.3 Seconds Off an API Call with Sentry 🕒

Struggling with slow API calls? Dan Mindru walks through how he used Sentry's new Trace View feature to shave off 22.3 seconds from an API call.

Get a practical walkthrough of how to identify bottlenecks, split tasks into multiple parallel tasks, identify slow AI model calls, and more.

Top comments (0)

Try REST API Generation for Snowflake

DevOps for Private APIs. Automate the building, securing, and documenting of internal/private REST APIs with built-in enterprise security on bare-metal, VMs, or containers.

Auto-generated live APIs mapped from Snowflake database schema
Interactive Swagger API documentation
Scripting engine to customize your API
Built-in role-based access control

Learn more

DEV Community