All Data and AI Weekly #217-24Nov2025

#aisql #snowflake #apachenifi #apacheiceberg

Monthly NYC and Youtube Events
https://lu.ma/PINSAI

🎤 Where in the AI Landscape is Tim Spann

Tim Spann is having an incredibly busy season, driving innovation and sharing key insights across the industry.

Event / Project	Details	Link
New App Launch: RPIThermalStreaming	Tim has built and launched a new application for thermal streaming using RPi technology.	View Code
Speaking: LLM Day (NYC Q1 2026)	Topic: "From TrafficAI to GhostBreakers: Building Stateful AI Agents with Cortex & OpenFlow."	Event Details
Speaking: Conf42 IoT 2025	Scheduled presentation at the Conf42 IoT conference.	Event Details
Past Speaking	Tim presented at the Data Science Summit 2025.	Presentation Materials

Significant advancements have been made to Snowflake's core services, enhancing performance, governance, and developer productivity.

Next-Gen Snowpipe Streaming: The architecture has been updated for Snowpipe Streaming, delivering a new high-performance data ingestion experience.
- Deep Dive on New Architecture
- Engineering Blog Post
Snowpark Container Services (SPCS) Enhancements: New features include block storage volume encryption support and updates to image repository creation.
- Encryption Support for Block Storage
- Image Repository SQL Reference
Storage Lifecycle Policies (GA): These policies are now Generally Available, offering simplified storage management and optimized cost control.
- Official Blog Post
- User Guide Documentation
Iceberg Table Innovations: New guides cover declarative pipelines using Dynamic Iceberg Tables and querying Iceberg tables via external engines using Snowflake Horizon.
- Declarative Pipelines Guide
- External Query Engine via Horizon
Terraform Provider Update: Version v2.11.0 has been released for the Snowflake Terraform provider.
- Release Notes

The pace of development in the AI space continues to accelerate, with new guides focused on building sophisticated, production-ready applications.

Optimizing Cortex Analyst Performance: A deep-dive article on techniques for enhancing the efficiency of the Cortex Analyst function.
- Medium Article
Cortex Agents Best Practices: Essential guidance for designing and deploying robust Agentic applications.
- Best Practices Guide
Prompt Caching: Documentation on prompt caching within the Cortex REST API for improved latency and cost management.
- Cortex REST API Documentation
Knowledge Graph Development: Multiple guides on building Knowledge Graphs natively within the platform, powered by Snowflake Intelligence.
- Complete Knowledge Graph Guide
- Agentic Domain Knowledge Graph
ML Model Management: Importing and deploying models from external services like HuggingFace using the Snowsight UI, and resources for RAG evaluation.
- Model Registry Guide
- RAG Evaluation Cheatsheet

New open source tools and guides to streamline development workflows.

Iceberg https://docs.snowflake.com/en/user-guide/tables-iceberg-query-using-external-query-engine-snowflake-horizon
Data Connectivity: New guides for data connectivity using OpenFlow with PostgreSQL CDC and Snowpark Connect for Apache Spark.
- OpenFlow PostgreSQL CDC
- Snowpark Connect for Apache Spark
Open Source Tools:
- Firecrawl Simple: A simple project for web crawling and extraction. GitHub
- VisiData: A fast, interactive data analysis tool for the command line. Project Page
- CSVLens: Command-line CSV viewer. GitHub
Essential Reading: The Essential Guide to Data Engineering and a guide to FinOps courses.
- Essential Guide to Data Engineering
- FinOps Course
Webinar Replays:
- Cinebot Build: Virtual Hands-on Lab: Build a Cinebot with the Snowflake Native App Framework.
- Webinar
- Multimodal Document Pipeline: Building your first Multimodal Document Pipeline.
- Webinar