augusto kiniama rosa

Posted on Jan 1 • Originally published at blog.infostrux.com on Dec 31, 2024

The Unofficial Snowflake Monthly Release Notes: December 2024

#data #snowflake #snowflakedatacloud #newreleases

Monthly Snowflake Unofficial Release Notes #New features #Previews #Clients #Behavior Changes!

Welcome to the fantastic Unofficial Release Notes for Snowflake for December 2024! You’ll find all the latest features, drivers, and more in one convenient place.

As an unofficial source, I am excited to share my insights and thoughts. Let’s dive in! You can also find all of Snowflake’s releases here.

This month, we provide coverage up to release 8.47 (General Availability—GA). I hope to eventually extend this to private preview notices as well.

I would appreciate your suggestions on continuing to combine these monthly release notes. Feel free to comment below or chat with me on LinkedIn.

Behavior change bundles 2024_06 is Generally Enabled, 2024_07 is enabled by default but can be disabled, and 2024_08 can be enabled but disabled by default.

What’s New in Snowflake

New Features

Restricted caller’s rights(Preview), previously, when an executable like a stored procedure ran with the privileges of the caller of the executable (caller’s rights), the executable could run with all of the caller’s privileges. Restricted caller’s rights allow an executable to run with caller’s rights but restrict which of the caller’s privileges the executable runs with. Administrators use caller grants to specify which of the caller’s privileges the executable can use to operate.
Outbound private connectivity for Snowflake features (Preview), create private endpoints in Snowflake to access a cloud platform using the platform’s private connectivity solution rather than the Internet. This lets you access cloud platform services privately and securely from Snowflake.
External stages using Azure Private Link(Preview), configure an external stage and create a private endpoint so bulk loading from Azure storage occurs over Azure Private Link.
External volumes using Azure Private Link(Preview), configure an external volume and create a private endpoint so you can connect Snowflake to your external cloud storage for Iceberg tables using Azure Private Link instead of the public Internet.
Snowpipe automation using Azure Private Link(Preview), configure an external stage and notification integration, and create a private endpoint, so that automatic Snowpipe data loads that are triggered by Microsoft Azure Event Grid use Azure Private Link instead of the public Internet.
Visual Studio Code extension for Snowpark Python(GA), integrates with Snowpark Python to provide authoring and debugging features for Snowpark Python code with: Inline debugging of Snowpark Python functions, Syntax highlighting and autocomplete suggestions for Snowflake SQL in Python strings within Python files or notebook cells, and Syntax highlighting and bracket autocomplete of Jinja templates in Snowflake SQL.

Snowsight Updates

New login screen version, improved performance and enhanced security.
New homepage for Snowsight(GA), updated to include: New navigation menu — updated to include menu items for creating and managing data products, notebooks, worksheets, databases and all related artifacts, Updated search — updated to allow for easier discovery of content, Quick actions — quickly and easily perform operations specific to your current role. For example, examine tables or create worksheets to execute python code, and Recently viewed tab, which includes recent operations and associated content.
Private Notebooks in a Personal Database(Preview), create a private, user-owned notebook. This private notebook is stored in your personal database: a dedicated workspace where you can create, modify, and manage your private notebooks.

Snowflake Applications

Snowflake Native Apps with Azure Private Link support(GA), can be deployed and operated with Private Link connectivity in Azure, enabling secure network isolation.
Inbound private connectivity to Snowpark Container Services for accounts on AWS(Preview), support for inbound private connectivity to Snowpark Container Services for accounts on AWS.
Using block storage with Snowpark Container Services job services(Preview), support for using block storage volumes with Snowpark Container Services job services.
Snowflake Native Apps in Azure Government regions(GA), Providers publishing apps from government regions can only share listings within the same organization, while consumers in government regions can install apps from the public marketplace.

Data Lake Updates

Specify an external ID for SIGV4 REST catalog integrations, support for specifying an external ID when you create a catalog integration for Apache Iceberg™ REST that uses SIGV4 authentication. Specifying an external ID lets you use the same IAM role across multiple catalog integrations. This can be useful in testing scenarios when you need to recreate or replace a catalog integration many times.

Streamlit Updates

Support for Streamlit 1.39.0(Preview), support for version 1.39.0 of the Streamlit open-source library in Streamlit in Snowflake.
Azure Private Link in Streamlit in Snowflake(GA)
Azure Private Link in Streamlit in Snowflake(Preview)

SQL Updates

New function ICEBERG_TABLE_FILES, information schema table function that returns information about the data files registered to an externally managed Apache Iceberg™ table.
New function ICEBERG_TABLE_SNAPSHOT_REFRESH_HISTORY, information schema table function that returns metadata and snapshot information about the most recent refresh history for a specified externally managed Apache Iceberg™ table.
The new function, AVAILABLE_LISTING_REFRESH_HISTORY, returns the past 14 days of refresh history for a database mounted from a listing using cross-cloud auto-fulfillment.
Snowflake Scripting: Asynchronous child jobs(Preview), Snowflake Scripting (SQL) stored procedures can run queries concurrently as asynchronous child jobs. The query can be any valid SQL statement, including SELECT statements and DML statements, such as INSERT or UPDATE, add the ASYNC keyword to the query for a RESULTSET. When this keyword is omitted, the stored procedure runs child jobs sequentially, and each child job waits for the running child job to finish before it starts.

Extensibility Updates

Support for a wildcard character in network rule network identifiers(Preview), use an asterisk as a wildcard character when specifying a network identifier for a network rule in its VALUE_LIST parameter.
Profiling Python stored procedure handlers(Preview), discover how much time or memory was spent executing your stored procedure handler code by using the built-in code profiler. The profiler generates information describing how much time or memory was spent executing each line of Python code.
Java 17 support(GA) in Snowpark, create and run stored procedures and UDFs using Java 17. The Snowpark API and JDBC Driver have also been updated to support Java 17.
External network access for Azure Gov regions(GA), Access network locations external to Snowflake from within procedure and UDF handler code is now generally available in all regions.

AI Updates (Cortex, ML, DocumentAI)

Document AI improvements: Doubling the length limit of the answers provided by the model. The model can now return answers that are up to 512 tokens long (about 320 words) per question, Re-ordering values and answers in the Document AI user interface by dragging them is now possible. This makes the evaluation process easier, as the defined values can follow a certain order, and general performance improvements.
Snowflake Cortex Powered Descriptions(GA), ability to sign in to Snowsight and use Snowflake Cortex to generate descriptions for tables, views, and columns. Snowflake Cortex leverages large language models to generate the descriptions based on object metadata and, optionally, sample data.

Sharing and Listing Updates:

Organizational listings: Discovery and access(Preview), enhancements to Organizational listing: discovery and access. Listing creators can now configure both the access and discovery of organizational listings. Listing creators can now define how various accounts and roles in their organization can discover and access the organizational listings. Listing creator can also specify who can access the Listing directly in the Internal Marketplace.

Performance Improvements

Improved sharing of common or similar parts of a query, reduces query execution time for queries with multiple WITH clauses.
Improved scaling of document pre-processing and inference in Document AI, decreases processing time of documents.

Data Pipelines/Data Loading/Unloading Updates

Dynamic tables: maximum number of dynamic tables in an account increased to 10,000 (GA), hold a maximum of 10,000 dynamic tables (4,000 was the previous limit).
Dynamic tables: Unlimited inputs(GA), define dynamic tables that read from an unlimited number of tables or dynamic tables(the previous limit was 100 tables or dynamic tables).

Security & Governance Updates

OBJECT_DEPENDENCIES view: Support for dynamic tables, view in the ACCOUNT_USAGE schema now includes dependencies involving dynamic tables.

Open-Source Updates

terraform-snowflake-provider 1.0.1 (Misc: Add missing preview features (#3306), references #3302, Update docs and migration guide (#3313, #3288), references #3303, #3302, #3298, Add FAQ entry for account is empty error (#3312), references #3198, #3308, Bug fixes: Fixes in account parameters)
terraform-snowflake-provider 1.0.0 (Breaking changes: Implement a flag for enabling preview features (#3218), (#3225), Remove deprecated resources, data sources, and fields (#3215), (#3268), Adjust disclaimers in resources, data sources, and general documentation (#3277), Update feature lists in v1-preparations (#3268), New: Account v1 readiness (#3236) (5df33a8), Account v1 readiness generated files (#3242) (3df59dd), Account v1 readiness resource (#3252) (8f5698d), Add a new account roles data source (#3257) (b3d6b9e), Add account data source (#3261) (6087fc9), Add all other functions and procedures implementations (#3275) (7a6f68d), Basic functions implementation (#3269) (6d4a103), Basic procedures implementation (#3271) (933335f), Docs, test, and missing parameter (#3280) (10517f3), Functions and procedures schemas and generated sources (#3262) (9b70f87), Functions sdk update (#3254) (fc1eace), Handle missing fields in function and procedure (#3273) (53e7a0a), Procedures schemas and generated sources (#3263) (211ad46), Procedures sdk update (#3255) (682606a), Rework account parameter resource (#3264) (15aa9c2), Rework data types (#3244) (05ada91), support table data type (#3274) (13401d5), Tag association v1 readiness (#3210) (04f6d54), Test imports and small fixes (#3276) (a712195), Unsafe execute v1 readiness (#3266) (c4f1e8f), Use new data types in sql builder for functions and procedures (#3247) (69f677a), Misc: Add ShowByID filtering generation (#3227) (548ec42), Adress small task-related todos (#3243) (40de9ae), Apply masking (#3234) (c209a8a), fix missing references in toOpts and changes with newlines (#3240) (246547f), function tests (#3279) (5af6efb), Improve config builders (#3207) (425787c), Revert to proper env (#3238) (5d4ed3b), Use service user for ci (#3228) (2fb50d7))
Snowflake VS Code Extension 1.11.1 (Fixed false positive SSO prompting for remote host)
Snowflake VS Code Extension 1.11.0 (Features: Snowflake Native Application integration released to GA, Requires SnowCLI to be installed, Native Application panel provides commands and outline view support for multiple projects, Native Apps environment variable manager for templating and deployment, Bug Fixes: Fixed bug with object explorer search result caching, Fixed some SQL compilation errors showing up in output on login, Refine determination of whether user is on a remote host for SSO, Internals: Log the vscode environment remoteName and appHost when initializing to assist debugging remote SSO issues)
Streamlit 1.41.1 (bug fixes)
Streamlit 1.41.0 (New Features: Add pathlib support by @jrieke in #9711, Follow ups for pathlib support by @jrieke in #9783, Support primary color as text and background color in markdown by @lukasmasuch in #9676, Improve hover tooltips for st.line_chart by @lukasmasuch in #9674, Add support for ISO time/date format in st.time_input / st.date_input by @sfc-gh-tteixeira in #9753, Add async generator support to st.write_stream by @lukasmasuch in #8724, Add repr_html support to st.html, and avoid throwing for non-paths by @sfc-gh-tteixeira in #9877, Remove temporary padding fix for fullscreen button by @lukasmasuch in #9882, Allow pinning dataframe columns via column config by @lukasmasuch in #9535, Remove internal Streamlit details from exception traces by @sfc-gh-tteixeira in #9913, Borderless “tertiary” buttons by @mayagbarnes in #9923, Improvements to the showErrorDetails config by @sfc-gh-lwilby in #9909, Border for st.metric by @mayagbarnes in #9928, Border for st.columns by @mayagbarnes in #9927, Bug Fixes: Disallow double slash paths to redirect by @kmcgrady in #9754, Fix usage of hyphens in user key for CSS classes by @lukasmasuch in #9793, Fix #9763. Don’t show text keyboard for number input on IOS by @kajarenc in #9766, Fixes a bug where timezones with a non- x:00 utc offset would start at a non 00:00 timer for st.audio_input by @sfc-gh-nbellante in #9791, Fix dataframe fullscreen for large dataframes by @sfc-gh-nbellante in #9803, fix #9810 Update to latest Material symbols font by @kajarenc in #9813, Fix exception element styling by @jrieke in #9818, [Fix] Prevent rerun for text_input when value did not change by @raethlein in #9826, [Fix] Set text area value based on state by @raethlein in #9829, Correctly handle DeltaGenerator in st.write by @lukasmasuch in #9828, Add shared hooks for submitForm & inputChange by @raethlein in #9847, Set the default selection mode to ‘single’ in the overload methods of pills and segmented_control by @RubenVanEldik in #9801, [fix] st.graphviz_chart rendering bug with use_container_width=True by @sfc-gh-bnisco in #9867, Support URLPattern-compatible patterns in allowed domain matching by @sfc-gh-mgolebiowskiowczarek in #9849, Fix wrong dataframe return typing by @lukasmasuch in #9898, Fix URI decoding of link column display text by @lukasmasuch in #9895, Disallow invalid scopes for st.rerun by @kmcgrady in #9911, Fix slow rendering performance caused by complex CSS selector by @lukasmasuch in #9901, Update audio buffer to handle channels better by @kmcgrady in #9885, Only show st.cache decorator spinners for cache misses and outermost function by @raethlein in #9956, Have Images List not utilize the individual image width parameter by @kmcgrady in #9958, Other Changes: Clean up hello app by @jrieke in #9760, Adds a timestamp into the file name for st.audio_input by @sfc-gh-nbellante in #9768, Update watchdog to include 6.x.x by @RubenVanEldik in #9785 release/1.40.0 by @sfc-gh-bnisco in #9790Release/1.40.1 by @sfc-gh-bnisco in #9831, Release/1.40.2 by @kmcgrady in #9925, Set Python 3.13 as max version for Streamlit by @raethlein in #9635)

Client, Drivers, Libraries and Connectors Updates

New features:

Go Snowflake Driver 1.12.1 (golangci-lint and fixed some errors found by this tool, S3 logging configuration, changed RaisePutGetError to be true by default, implemented the DriverContext interface to reduce the time for creating new connections and to use context when connections are created using db.Conn(ctx), introduced the disableOCSPChecks configuration option, which replaces insecureMode. Deprecated insecureMode, provided the ability to set custom OCSP cache server URLs)
JDBC Driver 3.21.0 (support for regional Google Cloud Storage endpoints, Added the SKIP_TOKEN_FILE_PERMISSIONS_VERIFICATION option to skip token file permission verification, Added the JAVA_LOGGING_CONSOLE_STD_OUT option to override default java.util.logging.ConsoleHandler to write to stderr or stdout with a specific threshold, Added array bind supported log for prepared statements, Removed the experimental label from snowflake-jdbc-thin artifact, Changed the levels for some log messages, Updated the Google and Netty dependencies, Updated the javadoc documentation)
Snowflake Connector for Google Analytics Raw Data 1.8.0 ()
Node.js 2.0.0 (Beginning with version 2.0.0, the Node.js driver introduced the following breaking changes: Removed support for the Node.js library 14, 16, and 17 versions in the Node.js driver. Node.js no longer officially supports versions lower than 18 of its library. Snowflake encourages everyone using the Node.js versions lower than 18 environments to upgrade to Node.js version 22 (LTS), Changed the name of the insecureConnect configuration flag that allows skipping OCSP verification to disableOCSPChecks, The Node.js driver considers all types and methods described in the typings file to be part of the driver’s public API; other components are treated as internal, and new features: Extended logging at the transport layer, Improved URL data sanitation, support for GCS region-specific endpoints, implemented GCM encryption algorithms, bumped axios to version 1.7.7, replaced aws-sdk by smithy in version 3.2.5)
Snowflake Connector for Kafka 3.0.0 (ingest data into a Snowflake-managed Apache Iceberg™ table)
Snowflake Connector for Python 3.12.4 (Bumped the pyOpenSSL dependency from >=16.2.0,<25.0.0 to >=22.0.0,<25.0.0)
Snowpark Library for Python 1.26.0 (Added support for property version and class method get_active_session for Session class, Added new methods and variables to enhance data type handling and JSON serialization/deserialization: To DataType, its derived classes, and StructField:type_name: Returns the type name of the data, simple_string: Provides a simple string representation of the data, json_value: Returns the data as a JSON-compatible value, json: Converts the data to a JSON string, To ArrayType, MapType, StructField, PandasSeriesType, PandasDataFrameType and StructType:from_json: Enables these types to be created from JSON data, To MapType:keyType: keys of the map, valueType: values of the map, Added support for method appName in SessionBuilder, Added support for include_nulls argument in DataFrame.unpivot., Added support for following functions in functions.py:size to get size of array, object, or map column, collect_list an alias of array_agg, concat_ws_ignore_nulls to concatenate strings with a separator, ignoring null value, substring makes len argument optional, Added parameter ast_enabled to session for internal usage (default: False), Improvements: Added support for specifying the following to DataFrame.create_or_replace_dynamic_table:iceberg_config A dictionary that can hold the following iceberg configuration options:external_volume, catalog, base_location, catalog_sync, storage_serialization_policy, Added support for nested data types to DataFrame.print_schema, Added support for level parameter to DataFrame.print_schema, Improved flexibility of DataFrameReader and DataFrameWriter API by adding support for the following: Added format method to DataFrameReader and DataFrameWriter to specify file format when loading or unloading results, Added load method to DataFrameReader to work in conjunction with format, Added save method to DataFrameWriter to work in conjunction with format, Added support to read keyword arguments to options method for DataFrameReader and DataFrameWriter, Relaxed the cloudpickle dependency for Python 3.11 to simplify build requirements. However, for Python 3.11, cloudpickle==2.2.1 remains the only supported version, Snowpark pandas API : Added partial support for Series.map when arg is a pandas Series or a collections.abc.Mapping. No support for instances of dict that implement __missing__ but are not instances of collections.defaultdict, Added support for DataFrame.align and Series.align for axis=1 and axis=None, Added support for pd.json_normalize, Added support for GroupBy.pct_change with axis=0, freq=None, and limit=None, Added support for DataFrameGroupBy.__iter__ and SeriesGroupBy.__iter__, Added support for np.sqrt, np.trunc, np.floor, numpy trig functions, np.exp, np.abs, np.positive and np.negative, Added partial support for the dataframe interchange protocol method DataFrame.__dataframe__())
SQLAlchemy 1.7.2 (support for structured OBJECT and ARRAY datatypes)
SQLAlchemy 1.7.1 (support for PARTITION BY to the COPY INTO command)
Snowflake Connector for Google Analytics Raw Data 2.8.0 (Migrated to telemetry v2)
Snowflake Connector for ServiceNow® V2 5.15.0 (Behavior changes: The export process to store the connector's internal state changed, In addition to storing metadata in the __CONNECTOR_STATE_EXPORT table, the data is also split into multiple tables with a _SFSDKEXPORT_V1 suffix, New features: Snowflake Connector for ServiceNow® V2 now supports disaster recovery in another region, support for configuring deletion synchronization at the table level using the ENABLE_TABLE procedure)
Snowflake Connector for MySQL 6.9.1 (Behavior changes: The native application has been migrated to the new event-sharing model. Telemetry sharing is now mandatory for all event types, such as logs and traces, and it can’t be disabled for the new installations. For already existing installations, you will receive an email with our account locator “EOA66985” asking you to review and configure the required event sharing)
Snowflake Connector for PostgreSQL 6.9.1 (Behavior changes: The native application has been migrated to the new event-sharing model. Telemetry sharing is now mandatory for all event types, such as logs and traces, and it can’t be disabled for the new installations. For already existing installations, you will receive an email with our account locator “EOA66985” asking you to review and configure the required event sharing, The connector supports column types that are not listed in the pg_type table)

Bug fixes:

Go Snowflake Driver 1.12.1 (Propagated context to authentication processes, HTTP headers used to communicate with Azure are now case insensitive, Use correct length of IV for GCM encryption)
Ingest Java SDK 3.0.1 (Fixed an issue with schema evolution for structured data type, Fixed an issue with loading files greater than 16 MB using the MD5 hash, Upgraded io.netty to fix a security vulnerability.)
JDBC Driver 3.21.0 (Replaced raw asserts with exceptions, Changed the IV length to 12 bytes for GCM, Changed initialization of SecureRandom to use a default JVM random number generator, Fixed an issue with merging io.netty.versions.properties during shade, Fixed uncontrolled logging from Arrow library, Fixed native libraries relocation for Netty and Conscrypt during shade, and Fixed get object and get bytes support for the native arrow structured type.)
Node.js 2.0.1 (issue related to missing proxy ports during configuration processing)
Node.js 2.0.0 (Fixed nonempty logs when the log level is set to OFF)
Snowflake CLI 3.2.2 (Fixed the No module name 'pandas' warning)
Snowflake CLI 3.2.1 (Fixed an issue that caused failures when using older x86_64 Intel CPUs)
Snowflake Connector for Kafka 3.0.0 (Fixed dependency vulnerabilities)
Snowflake Connector for Python 3.12.4 (bug where multi-part uploads to Azure were missing their MD5 hashes, bug where OpenTelemetry header injection would sometimes cause Exceptions to be thrown, bug where OCSP checks would throw TypeError and make mainly GCP blob storage unreachable)
Snowpark Library for Python 1.26.0 (Removed warnings that dynamic pivot features were in private preview, because dynamic pivot is now generally available, Fixed a bug in session.read.options where False Boolean values were incorrectly parsed as True in the generated file format, Snowpark pandas API : Fixed a bug in df.loc where setting a single column from a series results in unexpected None values)
Snowpark Library for Scala and Java 1.15.0 (New functions:months_between, instr, format_number, from_unix_timestamp, to_unix_timestamp, Row.getAs, Support for SQL bind in Session.sql function)
SQLAlchemy 1.7.2 (Fixed an issue with quoting an underscore (_) as a column name, Fixed an issue with index columns not being reflected, Fixed an issue with the index reflection cache)
SQLAlchemy 1.7.1 (Fixed the BOOLEAN type not found error in snowdialect)
Snowflake Connector for Google Analytics Raw Data 2.8.0 (Data stream ingestions, which failed due to the Out of Memory errors, are now retried sooner)
Snowflake Connector for ServiceNow® V2 5.15.1 (Added migration to support old sync states in data export)
Snowflake Connector for ServiceNow® V2 5.15.0 (Unexpected responses from the ServiceNow API are now correctly handled in the procedures such as CHECK_ROW_COUNT)
Snowflake Connector for MySQL 6.9.1 (Stopped printing the warning in the agent: “JAXB is unavailable. Will fallback to SDK implementation which may be less performant”, Stopped printing the warning in the agent: “Unable to load native-hadoop library for your platform… using builtin-java classes where applicable”)
Snowflake Connector for PostgreSQL 6.9.1 (Stopped printing the warning in the agent: “JAXB is unavailable. Will fallback to SDK implementation which may be less performant”, Stopped printing the warning in the agent: “Unable to load native-hadoop library for your platform… using builtin-java classes where applicable”)
Snowflake Connector for SharePoint 1.0.5 (Fixed an issue that was causing empty values to be returned in the web_url column in the Cortex Search service responses)

Conclusion

December 2024 was a quieter month for the usual Snowflake crazy pace but still delivered powerful new capabilities also introduced behaviour changes that might require some extra attention; some of the good stuff were Azure Private Link integrations, advanced Snowpark Python features, and Document AI improvements, and a few good ICEBERG tables improvements. One of the things of note is that Snowflake Terraform provider reached 1.0, which is a great milestone.

Overall, this month maintains a pace of improvements to the ecosystem, including applications and containers. It continues to improve the overall system.

Enjoy the reading.

I am Augusto Rosa, VP of Engineering for Infostrux Solutions. I am also a Snowflake Data Super Hero and Snowflake SME. You can follow me on LinkedIn.

Subscribe to Infostrux Medium Blogs https://medium.com/infostrux-solutions for the most interesting Data Engineering and Snowflake news.

DEV Community