※This is an English translation of my original article in Japanese.
Hello, I'm Sagara.
As a consultant specializing in Modern Data Stack, I'm constantly keeping up with the flood of information in this space. In this article, I'll summarize some interesting Modern Data Stack updates from the past two weeks.
Note: This doesn't cover all the latest information about these products. It only includes information that **I personally found interesting.
Data Warehouse/Data Lakehouse
General
Is Apache Iceberg the New Hadoop? Navigating the Complexities of Modern Data Lakehouses
Data Engineering Weekly published an article comparing Apache Iceberg with Hadoop from implementation challenges and ecosystem perspectives.
The article points out that Iceberg itself is not a complete solution and requires a surrounding ecosystem to function effectively, including catalogs, compute engines, and maintenance processes. Hadoop similarly had a vast ecosystem, and managing it was a major source of complexity - suggesting Iceberg faces similar challenges.
I personally agree with these observations and am interested to see how various vendors will address these operational challenges.
https://www.dataengineeringweekly.com/p/is-apache-iceberg-the-new-hadoop
Snowflake
v9.7 and Bundle 2025_02 Details Released
Snowflake has published the details of their latest version v9.7 and Bundle 2025_02 on their release notes page.
Some notable updates include:
- (v9.7) XML format support
- (Bundle 2025_02) Grant privileges to users
- (Bundle 2025_02) Maximum size for each data type extended from 64MB to 128MB
https://docs.snowflake.com/release-notes/2025/9_07
Snowflake Can Now Define External Catalog Iceberg Tables by Referencing Amazon S3 Tables
A Snowflake Product Manager published a blog announcing that Snowflake can now define external catalog Iceberg tables by referencing Amazon S3 Tables. This feature is currently in preview.
(This became possible after Amazon S3 Tables added support for Apache Iceberg REST Catalog API on March 13.)
https://medium.com/snowflake/snowflake-integrates-with-amazon-s3-tables-d6cebf5fdcb2
BigQuery
Article on Cost Optimization with BigQuery Slot Management
Medium published a comprehensive article on cost optimization through BigQuery slot management.
The article systematically covers everything from explaining slots and cost structures to the various features available for slot management, which I found quite helpful.
https://medium.com/google-cloud/bigquery-cost-optimization-with-slot-management-e6eb50697265
MotherDuck/DuckDB
DuckDB Announces Support for Apache Iceberg REST Catalog
DuckDB has announced support for Apache Iceberg REST Catalog.
Previously, DuckDB could only reference Iceberg by directly specifying metadata files on storage, so being able to connect with a catalog allows access to the latest metadata files, which I personally find very exciting!
The article below discussing this feature mentions integration with S3 Tables.
https://duckdb.org/2025/03/14/preview-amazon-s3-tables
DuckDB Announces "Local UI" - A GUI Tool for Local PC Execution
DuckDB has announced "Local UI," a GUI tool that can be run on local PCs.
As shown in the image from the blog below, it allows you to execute and analyze queries in a notebook format while obtaining statistics for tables and columns.
https://duckdb.org/2025/03/12/duckdb-ui.html
Data Transform
dbt
Article on Git Branching Strategies for dbt
The dbt Developer blog published an article summarizing best practices for Git branching strategies when using dbt.
https://docs.getdbt.com/blog/git-branching-strategies-with-dbt
On a related note, I recently published an article about dbt project structure and development processes when using Dev/QA/Prod environments in dbt Cloud, which might also be helpful.
https://dev.classmethod.jp/articles/dbt-cloud-dev-qa-prod-environments/
Article on dbt's New Engine: minijinja
The dbt Developer blog published an article about dbt's new engine, which uses Rust-implemented minijinja to dramatically speed up project parsing. Compared to traditional Jinja-based parsing, it achieves over 30x speed improvements, especially for large projects!
https://docs.getdbt.com/blog/faster-project-parsing-with-rust
More details will be explained at the upcoming dbt Developer Day on March 19-20, 2025. If you're interested, please join.
https://www.getdbt.com/resources/webinars/dbt-developer-day
Business Intelligence
Looker
Release Notes for Version 25.4 Published
Looker has published the release notes for their latest version, 25.4.
I found these updates particularly interesting:
- Added
gemini_in_lookerrole as a permission to use Gemini in Looker - Studio in Looker activation is no longer required to use Conversational Analytics
- I've written a blog post about this feature that might be helpful
- Support for key pair authentication in Snowflake connection settings
https://cloud.google.com/looker/docs/release-notes
Omni
Raised $69M in Series B Funding on Their 3rd Anniversary
Omni announced that they've raised $69M in Series B funding on their third anniversary.
Translating a quote from the article: "We're proud to announce that Omni has completed a $69 million Series B funding round led by ICONIQ Growth, with participation from existing investors Theory Ventures, First Round Capital, Redpoint Ventures, GV, and Snowflake Ventures. At this milestone of our third year since founding, our valuation has reached $650 million, with 8x year-over-year growth in both revenue and customer usage."
https://omni.co/blog/omni-series-b-funding
Data Catalog
Atlan
Visualize ER Diagrams Based on Excel-Defined Definitions
Atlan announced a new feature that allows users to import ER diagrams into Atlan using Excel templates.
https://shipped.atlan.com/import-data-modeling-assets-into-atlan-203xo4
Select Star
Released New Features Including Fivetran Integration and Improved Tableau Integration
Select Star released new features including Fivetran integration and improved Tableau integration.
Regarding Tableau specifically, it's now possible to manage sheets and views separately, identifying which sheets are in which views and displaying lineage. (Personally, I was most surprised to learn that custom SQL can also be displayed in lineage!)
https://docs.selectstar.com/changelog/march-12-2025-fivetran-integration-tableau-updates-and-more
Data Activation (Reverse ETL)
Census
Census Announces "Census Store" to Manage Catalog and Data Updates in Iceberg Format
Census announced a new feature called "Census Store" that manages catalog and data updates in Iceberg format.
https://www.getcensus.com/blog/why-we-chose-iceberg-as-the-foundation-for-census-store
According to the documentation, it's possible to use both Census-managed S3 buckets and user-managed S3 buckets to manage data.
https://docs.getcensus.com/misc/data-storage/census-store
Documentation is also available for using Census Store with Snowflake as an example use case.
https://docs.getcensus.com/misc/data-storage/census-store/query-census-store-from-snowflake
Census also announced "SaaS Datasets" as a feature to load SaaS data into Census Store. It currently supports Salesforce and HubSpot.
Similar to Fivetran's Managed Data Lake Service, I'm seeing a gradual increase in features where products manage catalog and data updates themselves.
Data Orchestration
Dagster
Announced "dg CLI" and "Components"
Dagster's OSS version announced new features: "dg CLI" and "Components" (currently in preview).
"dg CLI" is a CLI tool for managing Dagster projects, and "Components" is a feature that allows defining Dagster-related resources using YAML.
Top comments (0)