DEV Community

Tsubasa Kanno
Tsubasa Kanno

Posted on Edited on

Beyond Transcription - Multimodal Video and Audio Analysis with Snowflake AI_COMPLETE

#snowflake #llm #multimodal #sql

⚠️ Major revision notice (September 2026)

This article was originally published in May 2026. Cortex AI's multimodal support has expanded considerably since then, so I rewrote it in September 2026. The main changes are:

  • gemini-3.1-flash-lite joined the models that accept video and audio directly, adding a lighter-weight option
  • AI_TRANSCRIBE now accepts video files as-is, so you no longer need to extract the audio track first. The comparison section has been rewritten accordingly
  • AI_MULTI_EMBED enables semantic video search, which is covered in a new section
  • Per-request limits are now broken down by images, audio, and video, and there's a new section on passing multiple files in one call
  • AI_TRANSCRIBE's 10-second minimum billing duration is now documented, and is noted in the comparison section
  • The sample app is now model-selectable, and a new section covers how to create a stage for media files

Introduction

Snowflake's unstructured data analytics has just taken another big leap forward. After progressively adding support for images, documents, and audio, AI_COMPLETE can now accept video and audio files as direct input!

This update — the multimodal video and audio extension of AI_COMPLETE — is one of the Cortex AI Functions updates I've been watching most closely. You can now hand a video or audio file directly to AI from SQL, and get summaries, classification, sentiment analysis, and structured extraction in a single shot.

Until now, audio analysis usually meant a two-step pipeline with transcription in the middle, and video analysis required building your own preprocessing stack on something like Snowpark Container Services (SPCS). With this update, you can skip those steps and let the model directly understand vocal tone, pauses, and visual content — opening the door to analyses that were previously hard to attempt.

With images, documents, audio, and now video — nearly every major unstructured data format used in business is queryable from SQL. This means you can now build multimodal analytics in a much simpler way than before.

We'll work through the basics, how AI_COMPLETE compares with AI_TRANSCRIBE, semantic video search, and a sample app built with Streamlit in Snowflake — running each piece along the way.

Note: AI_COMPLETE's video and audio input is in Public Preview, so expect it to change in future updates. AI_TRANSCRIBE and AI_MULTI_EMBED, which this article also covers, are generally available (GA). Supported models and limits are expected to keep expanding.

Note: This article reflects my personal views and does not represent Snowflake's official position.

What's New: AI_COMPLETE Multimodal Extension

AI_COMPLETE is the central general-purpose LLM function in the Snowflake Cortex AI Functions family. It already supported text, images, and documents as multimodal input, but with this update you can now pass video and audio files directly.

Audio analytics used to follow a "transcribe with AI_TRANSCRIBE → analyze text with AI_COMPLETE" pattern. Video analytics required either extracting frames as images with Python libraries and feeding them into AI_COMPLETE's image input, or building an end-to-end frame-extraction-and-inference pipeline with SPCS. With this multimodal extension, you can skip those preprocessing steps and hand the audio or video file itself to AI_COMPLETE for summarization, sentiment analysis, and structured extraction.

Key Highlights

  • SQL-native: Same calling convention as before — just pass a video or audio file
  • Joint audio + visual understanding: For video, the model considers both visual content and audio track
  • Multimodal composition: Combine with text prompts for conditional analysis or JSON-structured output
  • Wide format support: Most common video and audio formats are accepted
  • Secure processing: Processing stays inside Snowflake

Part of the Cortex AI Functions Family

The multimodal extension shines when you chain its output into downstream AI Functions. After extracting a summary from a video, common follow-up patterns include:

  • AI_CLASSIFY: Categorize the extracted summary into industries or risk levels
  • AI_SENTIMENT: Score sentiment along multiple dimensions
  • AI_AGG: Aggregate insights across many media files
  • AI_EMBED: Embed extracted text or images for cross-corpus search
  • AI_TRANSCRIBE: A purpose-built function for transcription, ideal when you need timestamps and speaker labels from audio or video
  • AI_MULTI_EMBED: Embeds the video itself, enabling semantic search like "find scenes similar to this one" (covered later)

Creating a Stage for Media Files

Cortex AI's media functions read files from a stage via TO_FILE. The stage has to meet a couple of requirements, so let's create one first.

CREATE OR REPLACE STAGE media_stage
  DIRECTORY = ( ENABLE = TRUE )
  ENCRYPTION = ( TYPE = 'SNOWFLAKE_SSE' );
Enter fullscreen mode Exit fullscreen mode

Two things matter here:

  • Specify server-side encryption (SNOWFLAKE_SSE): internal stages default to client-side encryption, and media functions can't read from client-side encrypted stages
  • Add DIRECTORY = ( ENABLE = TRUE ): required when you want to process every file in the stage via DIRECTORY()

Note that user stages (@~) and table stages aren't supported. If you use an external stage on Amazon S3 or similar, enable encryption and a directory table the same way.

The examples below assume your media files live in this @media_stage.

Basic Usage

Passing a video or audio file directly to AI_COMPLETE follows the same simple syntax as image input.

AI_COMPLETE(
    <model>,
    <prompt>,
    <file>
)
Enter fullscreen mode Exit fullscreen mode

Parameters

  • model: Name of an LLM that supports video/audio input. gemini-3.1-pro, gemini-3.1-flash-lite, and gemini-3.5-flash are listed as supported, and the lineup keeps growing.
  • prompt: Natural-language instruction
  • file: A reference to a staged file via TO_FILE

Example 1: Summarizing a Video File

The simplest use case is to hand a video to the model and ask for a summary that takes both visuals and audio into account.

This article uses Pexels: Pouring Fresh Coffee into Ceramic Mug (Pexels License) as the sample video.

SELECT AI_COMPLETE(
    'gemini-3.1-pro',
    'Summarize this video in about 70 words. Include what is shown, the mood, and the likely scene.',
    TO_FILE('@media_stage', 'pouring_coffee.mp4')
) AS summary;
Enter fullscreen mode Exit fullscreen mode

The actual response looks like this:

A glass coffee server slowly pours freshly brewed coffee into a uniquely designed
small mug shaped like a blue character, sitting on a white table. The cute design
of the cup combined with the quiet motion of the cup gradually filling up evokes
a warm, relaxed atmosphere — exactly the kind of feeling that lets you take a
breather. It feels like a calm morning coffee at home or an afternoon break.
Enter fullscreen mode Exit fullscreen mode

The same pattern works for audio. Hand an audio file of English narration to the model and ask for a summary, and you get an output that captures both the content and the speaker's tone:

SELECT AI_COMPLETE(
    'gemini-3.1-pro',
    'Summarize this audio in about 50 words. Take the speaker''s emotion and atmosphere into account.',
    TO_FILE('@media_stage', 'english_narration.m4a')
) AS summary;
Enter fullscreen mode Exit fullscreen mode 
For Sherlock Holmes, she (Irene Adler) is always "the woman" — a uniquely special
presence. The narrator, Watson, says he has rarely heard Holmes refer to her by
any other name. The calm, reminiscent delivery quietly conveys the deep respect
Holmes holds for her, and her overwhelming presence.
Enter fullscreen mode Exit fullscreen mode

Japanese audio works the same way. The audio here was synthesized with the macOS say command for reproducibility — you can also generate equivalent audio with Mozilla Common Voice's Japanese dataset (CC0) or VOICEVOX.

SELECT AI_COMPLETE(
    'gemini-3.1-pro',
    'Summarize this audio, and state the speaker''s request and emotional tone in one or two sentences.',
    TO_FILE('@media_stage', 'business_call_jp.m4a')
) AS summary;
Enter fullscreen mode Exit fullscreen mode 
The speaker is reaching out to confirm the current status of a delayed delivery
of a recently purchased item. The emotional tone is highly polite and composed.
Enter fullscreen mode Exit fullscreen mode

Example 2: Going Beyond Words — Tone, Pitch, and Pace

This is a significant difference from AI_TRANSCRIBE. With AI_COMPLETE, the model can evaluate the vocal delivery itself — pitch, pace, tone, volume, and pauses (the official docs explicitly demonstrate this in the call-center example). The phrase "Hello" can carry completely different meaning depending on whether it's said with a bright, upbeat tone or a heavy, somber one. That nuance is lost the moment you transcribe to text — and AI_COMPLETE can capture it.

To verify, I synthesized the exact same Japanese phrase ("Hello, thank you for joining us today.") in two different tones — bright/fast vs. dark/slow — and passed each to AI_COMPLETE.

SELECT AI_COMPLETE(
    'gemini-3.1-pro',
    'You are an acoustic analyzer. Listen to the attached audio and evaluate the vocal delivery (pitch, pace, tone, volume). Return a short description of the vocal delivery only (do not transcribe content).',
    TO_FILE('@media_stage', 'tone_bright.m4a')
) AS bright_analysis;
Enter fullscreen mode Exit fullscreen mode

Output for the bright/fast version:

Spoken with a very high pitch and fast pace. The tone is bright and slightly
artificial (sounds sped-up or processed). Volume is moderate.
Enter fullscreen mode Exit fullscreen mode

Output for the dark/slow version:

Pitch is moderate and steady, and the pace is calm and easy to follow. The tone
is polite and composed, with consistent moderate volume.
Enter fullscreen mode Exit fullscreen mode

The actual words are identical, but the model returned clearly different evaluations based on how the voice was delivered. This is information that transcription alone doesn't surface. For use cases where the way something is said matters as much as what is said — call center quality scoring, interview evaluation, escalation detection — it's genuinely useful.

Example 3: Structured Analysis of a Video File

Use a JSON Schema to get structured output that downstream pipelines can consume directly.

Two extra arguments come into play here. The fourth argument holds model parameters such as temperature; pass {} when you don't need them. The fifth argument is the output schema, where you set 'type': 'json' along with your JSON Schema.

SELECT TO_JSON(AI_COMPLETE(
    'gemini-3.1-pro',
    'Analyze this video and return scene summary, visible objects, and the overall mood as JSON.',
    TO_FILE('@media_stage', 'pouring_coffee.mp4'),
    {},
    {
        'type': 'json',
        'schema': {
            'type': 'object',
            'properties': {
                'summary': {'type': 'string'},
                'objects': {'type': 'array', 'items': {'type': 'string'}},
                'mood': {'type': 'string'}
            },
            'required': ['summary', 'objects', 'mood']
        }
    }
)) AS analysis;
Enter fullscreen mode Exit fullscreen mode

Actual output:

{
  "mood": "Calm, relaxed",
  "objects": ["Coffee pot", "Mug", "Coffee"],
  "summary": "Coffee being poured from a black coffee pot into a unique mug featuring a blue bird's face design."
}
Enter fullscreen mode Exit fullscreen mode

The keys and types defined in the schema are honored, which makes it trivial to forward the output into the next stage of a data pipeline.

The same pattern applies to audio. Here it is against a recording of a meeting:

SELECT TO_JSON(AI_COMPLETE(
    'gemini-3.1-pro',
    'Analyze this audio and return a JSON object with a summary, the likely topic genre, and the overall tone.',
    TO_FILE('@media_stage', 'meeting_clip.mp3'),
    {},
    {
        'type': 'json',
        'schema': {
            'type': 'object',
            'properties': {
                'summary': {'type': 'string'},
                'genre': {'type': 'string'},
                'tone': {'type': 'string'}
            },
            'required': ['summary', 'genre', 'tone']
        }
    }
)) AS analysis;
Enter fullscreen mode Exit fullscreen mode 
{
  "genre": "Business/Marketing",
  "summary": "Monthly meeting for Nimbus Cloud highlighting a 32% improvement in paid search CPA, exceeding demo targets with 420 applications, and outlining future plans for LinkedIn targeting and CRM integration.",
  "tone": "Professional"
}
Enter fullscreen mode Exit fullscreen mode

Example 4: Conditional Analysis with Text Prompts

When you want only a specific aspect extracted, that's a matter of prompt design. Including instructions like "if there are none, just return 'none'" helps keep the model from guessing when there's nothing to report.

This article uses Pexels: Person Having an Interview (Pexels License). The video doesn't contain any product names, so the expected output is "none".

SELECT AI_COMPLETE(
    'gemini-3.1-pro',
    'Extract only the moments where product names (brand names) appear in this video, and list each name with its timing as bullet points. If there are none, just return "none".',
    TO_FILE('@media_stage', 'interview.mp4')
) AS extracted;
Enter fullscreen mode Exit fullscreen mode

The model returned exactly none, faithfully following the prompt.

Supported Models and Formats

Supported Models

According to the official documentation, the models listed as supporting direct video/audio input are gemini-3.1-pro, gemini-3.1-flash-lite, and gemini-3.5-flash. It started with gemini-3.1-pro only, and lighter options were added over time, so you can now pick your balance between analysis quality and speed. All three have a context window of roughly one million tokens, with the same supported formats and file-size limits. Snowflake partners with multiple model providers, so the lineup of multimodal-capable models keeps growing.

Model Context window When to use
gemini-3.1-pro 1,000,000 tokens When you prioritize higher analysis quality
gemini-3.1-flash-lite 1,048,576 tokens When you want a lighter model for higher volume
gemini-3.5-flash 1,000,000 tokens When you want faster, lighter-weight processing

Another thing I appreciate about AI_COMPLETE is that the model you used stays visible in the query. The function takes the model name as its first argument by design, so you always know which model produced a result, and you can tune the cost-versus-quality balance deliberately.

Supported Formats

A broad set of mainstream formats is supported, so you can work with most video and audio files you'd find in the wild.

Type Supported formats
Video mp4, mpeg, mov, avi, flv, mpg, webm, wmv, 3gpp
Audio wav, mp3, aiff, aac, ogg, flac, m4a, mp4, pcm, webm

Per-Request Limits

The per-request limits are as follows. They're comfortable for testing and proof-of-concept work, and are expected to be relaxed as the feature matures.

Item Limit
Total size per request 100 MB
Video files per request Up to 10
Audio files per request Up to 10
Image files per request Up to 20

File counts are tallied independently per type. For longer videos, splitting by scene or duration before sending lets you analyze plenty of content within current limits.

Passing Multiple Files in One Call

Every example so far passed a single file as the third argument. When you want several files in the same request so the model can compare them, use the PROMPT function and reference each file with {0}, {1}, and so on.

SELECT AI_COMPLETE(
    'gemini-3.1-pro',
    PROMPT('Do video {0} and image {1} look like they belong to the same campaign? Answer in 30 words or fewer.',
        TO_FILE('@media_stage', 'promo_spot.mp4'),
        TO_FILE('@media_stage', 'creative.png'))
) AS consistency_check;
Enter fullscreen mode Exit fullscreen mode 
Yes, they share the same visual identity, so they appear to belong to the same
campaign.
Enter fullscreen mode Exit fullscreen mode

That makes it a single query to check whether a campaign video and its creative artwork look consistent.

The PROMPT function reference currently focuses on text plus images and documents. Before putting a combination that includes video into a production pipeline, it's worth confirming the behavior in your own environment.

Regional Availability

Check the Cortex LLM Regional availability page for the native regions of the video/audio-capable models. If your region isn't natively supported, you can enable cross-region inference with a single SQL statement and use the same functionality.

ALTER ACCOUNT SET CORTEX_ENABLED_CROSS_REGION = 'ANY_REGION';
Enter fullscreen mode Exit fullscreen mode

There may be slightly higher latency when going via cross-region inference, but the functionality remains the same.

AI_TRANSCRIBE vs. AI_COMPLETE — Cost and Quality

"Should I use AI_TRANSCRIBE or AI_COMPLETE for audio and video?" is a question many people run into.

The fundamental thing to understand is that AI_TRANSCRIBE is a purpose-built, dedicated AI Function for converting media to text. The output is text by default, with optional word-level or speaker-level timestamps. That gives you less flexibility — but in return, it's simple to use when your goal is clear and costs are predictable per second of audio. AI_COMPLETE is a general-purpose LLM, so prompts can express anything you want, and the model can incorporate vocal delivery and visual information into its interpretation. The two aren't competitors — they're complementary: AI_TRANSCRIBE for clear, fixed goals; AI_COMPLETE for free-form interpretation.

AI_TRANSCRIBE Accepts Video Files Directly Too

When this article was first published, AI_TRANSCRIBE was audio-only. Today you can hand it a video file and have it transcribe the audio track, which removes the preprocessing step when you're building minutes from a meeting recording or a webinar archive.

SELECT AI_TRANSCRIBE(TO_FILE('@media_stage', 'meeting.mp4'));
Enter fullscreen mode Exit fullscreen mode

The response has the same shape as it does for audio:

{
  "audio_duration": 15.04,
  "text": "Let's begin this month's regular review for Nimbus Cloud. This month, paid search CPA improved by 32%. Demo signups came in at 420, ahead of target."
}
Enter fullscreen mode Exit fullscreen mode

Supported formats are as follows. A video file must contain at least one audio track, encoded with one of AAC, ALAC, FLAC, MP3, Opus, Vorbis, or PCM.

Type Supported formats
Audio aac, flac, m4a, mp3, mp4, ogg, wav, webm
Video mkv, mov, mp4, ogv, webm

Note that these formats differ from AI_COMPLETE's. An .avi recording works with AI_COMPLETE, for example, but needs converting before AI_TRANSCRIBE will take it.

With timestamps enabled, the maximum duration is 60 minutes; without them it's 120 minutes, and the file size limit is 700 MB. That's larger than AI_COMPLETE's limit of 100 MB combined per request, so AI_TRANSCRIBE is often the easier route for long recordings you want to process whole. AI_TRANSCRIBE is also available in a limited set of regions, so check the official reference alongside the model availability page.

Timestamps and Speaker Labels Are Opt-In

The example above returned only audio_duration and text. To get timestamps or speaker labels, pass timestamp_granularity as the second argument — word splits per word, speaker splits at each change of speaker.

SELECT AI_TRANSCRIBE(
    TO_FILE('@media_stage', 'meeting.mp4'),
    {'timestamp_granularity': 'speaker'}
);
Enter fullscreen mode Exit fullscreen mode 
{
  "audio_duration": 15.04,
  "segments": [
    {
      "speaker_label": "SPEAKER_00",
      "start": 0.051,
      "end": 14.78,
      "text": "Let's begin this month's regular review for Nimbus Cloud. ..."
    }
  ],
  "text": "..."
}
Enter fullscreen mode Exit fullscreen mode

Functional Differences

Aspect AI_TRANSCRIBE AI_COMPLETE (audio / video direct)
Primary purpose Structured media-to-text conversion Free-form analysis (summary, classification, extraction)
Output Text, with optional timestamps and speaker labels Whatever you ask for in the prompt (incl. JSON)
Speaker separation Set timestamp_granularity to speaker for labels (SPEAKER_00, etc.) Achieved by prompt instruction
Timestamps Word- or speaker-level Achievable via prompt + structured output
Visual content Audio track only Can interpret what's on screen
Vocal tone / emotion Text only Includes pitch, pace, tone, volume, pauses
Input size limit Up to 700 MB 100 MB combined per request
Multilingual Broad coverage LLM-native flexibility

Cost Comparison

Function Pricing characteristic
AI_TRANSCRIBE 1 sec of audio = 50 tokens (flat). 180,000 tokens per hour. 10-second minimum billing duration
AI_COMPLETE (audio / video direct) Standard LLM token pricing — varies by input size and output length

AI_TRANSCRIBE bills linearly with audio duration, so it shines when you want "just the transcript" or stable, predictable costs in big batch jobs. Keep in mind that files shorter than 10 seconds are billed as 10 seconds, so if you have a large number of short clips, batching them into one file and slicing by timestamp avoids waste. AI_COMPLETE varies with prompt and output size, which can swing costs depending on the task. In return, you explicitly choose the model in the first argument, giving you direct control over the cost-vs-quality trade-off in your application.

Quality Comparison

Scenario Recommendation
Meeting minutes drafting (need accurate transcript) AI_TRANSCRIBE → AI_COMPLETE chain
Transcript only, straight from a meeting recording AI_TRANSCRIBE (pass the video as-is)
Call center sentiment analysis (with vocal tone) AI_COMPLETE direct
Large-scale, fast batch transcription AI_TRANSCRIBE
Video (visual + audio) summarization & moderation AI_COMPLETE direct
JSON-structured extraction from audio AI_COMPLETE direct
Finding a specific scene across a large video library AI_MULTI_EMBED (covered next)

Rule of thumb: "Need text to keep on file" → AI_TRANSCRIBE. "Need the model to interpret content" → AI_COMPLETE. The "AI_TRANSCRIBE → AI_COMPLETE" pipeline is also a well-balanced option for cost and quality in many use cases.

Semantic Video Search with AI_MULTI_EMBED

Everything so far has been about reading one piece of media deeply. In practice you also run into a different need: finding the right moment across a large library of videos. AI_MULTI_EMBED is built for that.

Where AI_EMBED returns a single vector, AI_MULTI_EMBED segments the video by scene and returns one vector per segment. It also embeds three separate modalities — visual, audio, and the transcription of speech — so you can search for "scenes that look similar" separately from "scenes where similar things were said."

A quick glossary: a vector turns meaning into a list of numbers, so measuring the distance between two vectors tells you how similar their meanings are. A segment is a slice of the video cut at a scene boundary, and a modality is the kind of signal — visual, audio, or speech.

The available model is Twelve Labs' twelvelabs-marengo-embed-3-0.

SELECT AI_MULTI_EMBED(
    'twelvelabs-marengo-embed-3-0',
    TO_FILE('@media_stage', 'promo_spot.mp4')
) AS embeddings;
Enter fullscreen mode Exit fullscreen mode

What comes back is an array with one element per segment-and-modality combination. Because each element carries start_sec and end_sec, you can jump straight to the playback position of whatever matched.

{
  "error": null,
  "value": [
    {
      "embedding": [-0.0220, 0.0053, 0.0242, "..."],
      "embedding_option": "visual",
      "embedding_scope": "clip",
      "start_sec": 0,
      "end_sec": 4.5
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Flatten that into a table and you can search scenes with a natural-language query. The vectors are 512-dimensional, so cast them to VECTOR(FLOAT, 512) before handing them to a similarity function. If your stage also holds audio or images, filter by extension first.

-- Build a catalog of scenes
CREATE OR REPLACE TABLE video_embeddings AS
WITH embedding_index AS (
    SELECT
        TO_FILE('@media_stage', RELATIVE_PATH) AS video_file,
        AI_MULTI_EMBED('twelvelabs-marengo-embed-3-0',
            TO_FILE('@media_stage', RELATIVE_PATH)) AS embeddings
    FROM DIRECTORY(@media_stage)
    -- Common video extensions; check the docs for the full supported list.
    -- Snowflake's RLIKE rejects the (?i) inline flag, so lower-case the path instead.
    WHERE LOWER(RELATIVE_PATH) RLIKE '.*\\.(mp4|mov|avi|mkv|webm)'
)
SELECT
    video_file,
    f.value['embedding']::VECTOR(FLOAT, 512) AS embedding_vec,
    f.value['embedding_option']::STRING      AS embedding_option,
    f.value['start_sec']::FLOAT              AS start_sec,
    f.value['end_sec']::FLOAT                AS end_sec
FROM embedding_index,
LATERAL FLATTEN(input => embedding_index.embeddings['value']) f;
Enter fullscreen mode Exit fullscreen mode 
-- Search scenes with a natural-language query
WITH query AS (
    SELECT (AI_MULTI_EMBED(
        'twelvelabs-marengo-embed-3-0',
        'a scene where coffee is being poured'
    )):value[0]['embedding']::VECTOR(FLOAT, 512) AS query_embedding
)
SELECT
    v.video_file,
    v.start_sec,
    v.end_sec,
    VECTOR_COSINE_SIMILARITY(v.embedding_vec, q.query_embedding) AS similarity
FROM video_embeddings v
CROSS JOIN query q
WHERE v.embedding_option = 'visual'   -- swap for 'transcription' to search what was said
ORDER BY similarity DESC
LIMIT 10;
Enter fullscreen mode Exit fullscreen mode

Filtering on embedding_option is what lets you choose between searching how a scene looks and searching what was said in it. Without the filter, all three modalities compete in the same ranking.

The size limits are generous too: up to 4 hours and 6 GB per video. Putting AI_MULTI_EMBED in front of AI_COMPLETE — narrowing down which scenes are worth looking at, then reading those in detail — is an efficient way to work through a large video library.

twelvelabs-marengo-embed-3-0 is natively available in a limited set of regions, but enabling cross-region inference (above) lets you use it the same way. AI_MULTI_EMBED also differs from the other multimodal functions in that it supports client-side encrypted (CSE) stages and stages protected by network policies.

Business Use Cases

Here are some of the business scenarios that Cortex AI's video and audio support opens up.

1. Call Center Quality Scoring

Hand call recordings directly to AI_COMPLETE and get a multi-dimensional quality score — agent professionalism, customer anger level, escalation signals — that takes vocal delivery into account.

  • Tone-aware sentiment analysis: Catches subtle nuances that transcription would miss
  • Structured output integration: Returns each metric as JSON for dashboarding
  • Agent training: Identify improvement areas by comparing against high-performer recordings

2. Structured Meeting Minutes from Video

Pass a long meeting recording and get summaries, decisions, and action items broken down by topic and participant.

  • Topic-aware structuring: Prompt-driven chapter segmentation
  • Action item extraction: Get owner and due date as JSON
  • Cross-modal context: Whiteboards and on-screen materials are also picked up
  • Verbatim record: When you also need a word-for-word transcript, pass the same recording to AI_TRANSCRIBE

3. Compliance Checks for Promotional Videos

Run pre-publication checks for marketing videos to catch potential regulatory issues or brand-guideline violations.

  • Custom condition detection: Define check criteria via prompt
  • Cross-modal consistency: Spot inconsistencies between on-screen captions and spoken content
  • Multi-language QA: Verify localized versions in batch via SQL

4. Auto-Chaptering and Scene Search for Educational Content

Analyze online learning content with AI_COMPLETE to generate chapter titles, summaries, and timestamped indexes for viewers.

  • Topic-shift detection: AI identifies content boundaries
  • Audience-specific summaries: Generate multiple versions per skill level
  • Scene search: Embed the video itself with AI_MULTI_EMBED so learners can jump straight to "the part that explains this operation"

5. Social / UGC Moderation

UGC (User Generated Content) refers to videos, audio, and images posted by general users on social media or video platforms. You can implement harmful or policy-violating content detection as part of your SQL workflow.

  • Batch moderation: Auto-process newly arriving videos via Streams + Tasks
  • Reviewer notification: Send results to moderators via Snowflake Notifications
  • Multi-axis scoring: Score for violence, discriminatory speech, copyright concerns in parallel
  • Near-duplicate detection: Use AI_MULTI_EMBED to measure similarity against videos previously flagged as violations

Sample App: Video Analysis with Streamlit in Snowflake

To give you an instant feel for AI_COMPLETE's video support, here's a simple Streamlit in Snowflake app that lets you upload a video and analyze it on the spot.

What the App Does

  1. Video upload: Upload directly via browser; auto-saved to a stage
  2. Video preview: Play the uploaded video in place
  3. AI analysis: Run AI_COMPLETE with video input for summarization, extraction, classification
  4. Model switching: Pick the model used for analysis from the sidebar
  5. Editable prompt: Tweak the analysis prompt freely from the sidebar

Prerequisites

  • A Snowflake account where Streamlit in Snowflake is available
  • A video-capable model available natively, or cross-region inference enabled
  • Video files smaller than 100 MB
  • Streamlit version 1.52.2. Streamlit in Snowflake can default to an older Streamlit release, so pin the version explicitly to get current widget behavior
  • CREATE STAGE on the schema, plus the SNOWFLAKE.CORTEX_USER database role to call Cortex AI Functions

Specifying the Streamlit Version (Important)

Streamlit in Snowflake's default version may be older than expected, so I recommend pinning the version explicitly. Just place an environment.yml next to your app files in the stage:

# environment.yml
name: app_environment
channels:
  - snowflake
dependencies:
  - streamlit=1.52.2
Enter fullscreen mode Exit fullscreen mode

If you create the SiS app from Snowsight, you can change the streamlit version from the "Packages" dropdown in the upper right of the editor — pick the newest version offered there.

How to Set It Up

  1. Create a new SiS app from the Snowsight left pane → "Streamlit" → "+ Streamlit App"
  2. Paste the sample code below; the stage is created automatically on first run
  3. Click "Run" in the upper right, upload a video, and click "Analyze with AI"

Sample Code 

import streamlit as st
import io
import os
import re
import uuid
from datetime import datetime
from snowflake.snowpark.context import get_active_session

session = get_active_session()
STAGE_NAME = "VIDEO_ANALYSIS_STAGE"

db = session.get_current_database().strip('"')
schema = session.get_current_schema().strip('"')
STAGE_FQN = f"{db}.{schema}.{STAGE_NAME}"

st.set_page_config(layout="wide")
st.title("🎬 Video Analyzer")
st.caption("Analyze videos on the spot with AI_COMPLETE's video input")

# The model name has to be a string literal in SQL, so offer an allow-list.
# Add to this list as more models gain support.
AVAILABLE_MODELS = [
    "gemini-3.1-pro",
    "gemini-3.1-flash-lite",
    "gemini-3.5-flash",
]

st.sidebar.title("⚙️ Settings")
model = st.sidebar.selectbox("🧠 Model for analysis", AVAILABLE_MODELS)

prompt_template = st.sidebar.text_area(
    "📝 Analysis prompt",
    value=(
        "Summarize this video in about 100 words. "
        "Include the people involved, key scenes, and the emotion conveyed by the audio."
    ),
    height=180
)

@st.cache_resource
def setup_stage():
    try:
        session.sql(f"DESC STAGE {STAGE_FQN}").collect()
    except Exception:
        try:
            session.sql(f"""
                CREATE STAGE IF NOT EXISTS {STAGE_FQN}
                ENCRYPTION = (TYPE = 'SNOWFLAKE_SSE')
                DIRECTORY = (ENABLE = TRUE)
            """).collect()
        except Exception as e:
            st.error(f"Couldn't create the stage. Check your privileges: {e}")
            st.stop()

setup_stage()


def clean_ai_response(text):
    """Strip wrapping double quotes and unescape newlines from AI response."""
    if not isinstance(text, str):
        return text
    text = text.strip()
    if len(text) >= 2 and text.startswith('"') and text.endswith('"'):
        text = text[1:-1]
    text = text.replace("\\n", "\n").replace('\\"', '"')
    return text


uploaded_file = st.file_uploader(
    "Upload a video file (under 100 MB)",
    type=["mp4", "mov", "webm", "avi", "mpeg", "mpg", "flv", "wmv", "3gpp"]
)

if uploaded_file is not None:
    # AI_COMPLETE accepts up to 100 MB combined per request, so reject early
    if uploaded_file.size > 100 * 1024 * 1024:
        st.error("Please use a video under 100 MB.")
        st.stop()

    st.subheader("▶️ Video Preview")
    st.video(uploaded_file)

    if st.button("🤖 Analyze with AI", use_container_width=True, type="primary"):
        try:
            ext = os.path.splitext(uploaded_file.name)[1].lower() or ".mp4"
            safe_ext = ext if re.fullmatch(r"\.[a-z0-9]{2,5}", ext) else ".mp4"
            video_filename = (
                f"video_{datetime.now().strftime('%Y%m%d_%H%M%S')}_"
                f"{uuid.uuid4().hex[:8]}{safe_ext}"
            )

            with st.spinner("Uploading video to stage..."):
                video_stream = io.BytesIO(uploaded_file.getvalue())
                session.file.put_stream(
                    video_stream,
                    f"@{STAGE_FQN}/{video_filename}",
                    auto_compress=False,
                    overwrite=True
                )

            with st.spinner("AI is analyzing the video..."):
                # The model name must be a literal, so inject only allow-listed
                # values; the prompt and filename go through bind variables
                if model not in AVAILABLE_MODELS:
                    st.error("Unsupported model selected.")
                    st.stop()
                query = f"""
                    SELECT AI_COMPLETE(
                        '{model}',
                        ?,
                        TO_FILE('@{STAGE_FQN}', ?)
                    ) AS analysis
                """
                result = session.sql(query, params=[prompt_template, video_filename]).collect()

            st.subheader("📊 Analysis Result")
            if result and len(result) > 0 and result[0]["ANALYSIS"]:
                st.markdown(clean_ai_response(result[0]["ANALYSIS"]))
            else:
                st.warning("Couldn't get an analysis result. Try a different prompt or video.")

        except Exception as e:
            st.error(f"Something went wrong: {str(e)}")
else:
    st.info("👆 Upload a video file to start analyzing.")
Enter fullscreen mode Exit fullscreen mode

Implementation Notes

  • No extra packages: Works with default Streamlit in Snowflake packages
  • Auto-created stage: Created with server-side encryption and a directory table on first run
  • Model from an allow-list: AI_COMPLETE needs the model name as a string literal, so the app uses a selectbox with fixed choices rather than free text
  • Editable prompt: Update analysis prompt instantly from the sidebar
  • Video preview: Use st.video to play the uploaded file in-browser

The model name can't be a bind variable. AI_COMPLETE's first argument must be a string literal; passing ? produces a needs to be a string literal compilation error. If you let users choose a model, inject only values that came from an allow-list, as above, and never pass free-form input straight through.

Verified Behavior

The core of the app — sending the file to a stage with session.file.put_stream, then analyzing it with AI_COMPLETE — was verified on a live Snowpark Python session. Every model in the allow-list was also confirmed to accept video input. Paste the code into Streamlit in Snowflake and the same flow runs behind the UI.

App Screenshots

The demo video shown is Pixabay: Meadow, Grass, Spring Meadow, Wind (Pixabay Content License).

Right after uploading a video
Right after uploading a video

Right after running the AI analysis
Right after running the AI analysis

The sidebar settings panel
The sidebar settings panel

Cost

AI_COMPLETE billing follows token-based pricing, like other AI Functions. For direct video/audio input, factor in:

  • Input tokens: Vary with file length and content
  • Output tokens: Vary with the requested output length
  • Model rate: The credit rate of the model you selected applies

See the Snowflake Service Consumption Table for current pricing.

Cost optimization tip: When you only need transcription, use AI_TRANSCRIBE; when you need interpretation, use AI_COMPLETE. A two-stage pipeline that filters out clearly irrelevant videos with a simple prompt before deeper analysis works well too — as does narrowing candidates with AI_MULTI_EMBED before handing them to AI_COMPLETE.

Sample Materials Used in This Article

All free assets used here are commercially licensed so you can reproduce the examples yourself.

The Japanese audio was synthesized with the macOS say command. Note also that meeting.mp4, promo_spot.mp4, creative.png, and meeting_clip.mp3 are placeholder filenames used to illustrate the syntax — substitute your own.

Conclusion

AI_COMPLETE's video and audio support is a meaningful step toward simpler multimodal analytics on Snowflake. With video joining images, documents, and audio, nearly every major unstructured data format can now be queried straight from SQL. Add AI_TRANSCRIBE's video support and AI_MULTI_EMBED's video search, and all three modes — reading media, turning it into text, and searching it — are available from SQL.

Key Benefits

  1. Pass media as-is: Skip transcription — the model picks up vocal tone and visual context
  2. Pick the function that fits the goal: AI_COMPLETE to interpret, AI_TRANSCRIBE to transcribe, AI_MULTI_EMBED to search scenes
  3. Composes with other AI Functions: Chain with AI_CLASSIFY / AI_SENTIMENT / AI_AGG for flexible pipelines
  4. Governance preserved: Your media data never leaves Snowflake
  5. SQL-native: Drops cleanly into your existing data pipelines

Supported models and limits will keep expanding, opening up even more use cases over time. Whether it's call centers, meeting analysis, content moderation, education, or marketing, I hope you'll use Cortex AI's multimodal support to unlock new value from media that previously sat untouched!

Promotion

Snowflake Japan's Official YouTube Channel

The official Snowflake Japan YouTube channel has launched, and it already includes a walkthrough of Snowflake App Runtime. The video covers what App Runtime is and how to get an app deployed, with a live demo. It's in Japanese, but the demo sections should still be easy to follow.

Snowflake What's New Updates on X

I share Snowflake What's New updates on X. Follow for the latest insights:

English Version

Snowflake What's New Bot (English Version)

Japanese Version

Snowflake's What's New Bot (Japanese Version)

Change Log

(20260517) Initial post
(20260606) Updated the supported-models section: gemini-3.5-flash was added for video/audio input
(20260923) Major revision. Added gemini-3.1-flash-lite to the models that accept video and audio directly. Rewrote the comparison section to reflect that AI_TRANSCRIBE now accepts video files as-is, and documented how to request speaker-level timestamps. Added a new section on semantic video search with AI_MULTI_EMBED. Broke per-request limits down by media type, and added sections on passing multiple files via the PROMPT function and on creating a stage for media files. Noted AI_TRANSCRIBE's 10-second minimum billing duration and how its supported formats differ from AI_COMPLETE's. Made the sample app model-selectable and documented that the model name cannot be a bind variable

Original Japanese Article

Snowflake AI_COMPLETE 関数で動画と音声を直接分析

favicon zenn.dev

Top comments (0)