DEV Community: Krishna Tangudu

The Replication Worked. The Cutover Didn’t. — Snowflake to Fabric, Part 4

Krishna Tangudu — Wed, 11 Nov 2026 17:34:26 +0000

Snowflake AWS → Azure → Microsoft Fabric — Gotchas, Part 4

I wanted to see whether scheduled Snowflake replication could give us more control than continuous mirroring. The source changed only a few times a day, so it felt reasonable to try matching replication to those updates.

Initially, things looked good. The data reached Azure, Fabric queries worked, and the row counts we compared matched.

The trouble started when we connected the workspaces that people actually used. The shared lakehouse could read a table, but a downstream workspace was looking for files it could not find.

In Part 2, I said I would try the Azure replication route. This post covers what worked, where we got stuck, and why I wasn't ready to call it a replacement for mirroring.

Architecture labels are anonymized. This is an engineering investigation, not a controlled benchmark.

Why I wanted to try this

Our source updated approximately three times daily. With continuous mirroring, we had seen higher Snowflake cloud-services usage, but there was no configurable mirroring window to line up with those updates.

I wanted to try this flow:

Snowflake on AWS → scheduled replication to Azure → Fabric access to the replicated Iceberg files.

Snowflake supports cross-cloud replication. We scheduled three refreshes daily, and the initial load and the scheduled refreshes we checked completed successfully. Snowflake replication overview

We checked schemas, ran SQL queries and compared row counts for selected tables. Those checks passed. The next step was to connect the existing downstream workspaces.

I was hoping this would help with cost too, but we hadn't measured enough to say it was cheaper.

The shared lakehouse worked. The downstream one didn't.

Our setup had a shared Fabric lakehouse, with other workspaces accessing its tables through shortcuts. A shortcut points to data elsewhere, so each workspace doesn't need its own copy.

We changed selected shortcuts in the shared lakehouse to point to the replicated source. We kept the table names, and the downstream shortcuts continued pointing to the same shared lakehouse.

The animation shows the routes we tested. Replication copies data from AWS to Azure; the shortcut links provide access to it.

That is where we started seeing SQL I/O errors and Spark missing-file errors downstream, even though reads in the shared layer worked. Some conversion logs also showed success during the investigation.

There is an extra step here that is easy to overlook. Iceberg metadata describes which files belong to a table. Fabric creates virtual Delta metadata so its engines can read that Iceberg table. Completing the Snowflake refresh doesn't verify the rest of that path. OneLake Iceberg virtualization

I had checked the shared layer. I needed to go one workspace farther and check the queries people would actually run.

We tried refreshing metadata, then rolled back

We refreshed SQL endpoint metadata for the affected tables. Some reads recovered; others still failed after the refresh reported success.

We also checked Spark separately. In Part 2, disabling the vectorized reader had helped with a decimal compatibility issue. It didn't fix these missing-file errors, so we couldn't treat this as the same problem.

For affected workloads, we pointed the shared shortcuts back to their original mirrored sources. Even that didn't immediately recover every table. Some shortcut objects needed to be recreated, followed by scoped metadata refreshes.

After that, the SQL and Spark sample reads we checked passed through both layers. We hadn't deleted the underlying source data. We had recovered those reads, but we still hadn't made the replica-backed route reliable enough to replace mirroring.

Then we found the regional restriction

Our Azure replica and the consuming Fabric workspace were in different regions. During troubleshooting, we found a restriction in Microsoft's documentation: for Iceberg virtualization through a shortcut pointing to a OneLake location, the target must be in the same region as the shortcut. Microsoft's platform limitations

We created a fresh same-region test pointing directly to the same replica folders. Its initial conversions and all-column SQL sample reads passed. The older cross-region setup still had mixed conversion results.

That was useful progress, but I wouldn't say it explained every earlier failure. The new setup had only passed its initial checks, and the older setup had worked initially too. We still needed support to explain the downstream file-reference failures.

One thing I would check much earlier next time is the region of every OneLake target. Adding another shortcut doesn't move the data into a different region.

A query passing still wasn't the full answer

Later, we also saw SQL samples working while the latest conversion logs showed failures. These were generic conversion errors, separate from the earlier missing-file errors and Part 3's clone-metadata issue.

Microsoft documents that failed conversion can leave the last successfully converted version available. So a working query alone couldn't tell us whether the latest source change had reached Fabric. Conversion behavior

I now keep these checks separate:

What I want to know	What I check
Did replication finish?	The replica refresh result.
Can downstream users read it?	SQL and Spark through their actual access path.
Is the expected change there?	A known source change checked at the final consumer.

The row counts and previews were useful. They just weren't enough to approve the switch.

Where we are now

Snowflake replication worked in this trial. The full Fabric consumption path still had issues, and we hadn't established a cost advantage.

We are now separately testing an AWS replica with direct S3 access from Fabric. I will cover that in another post once we have a clearer result, including what happens across repeated refreshes and how the costs compare.

For now, my advice is simple: before changing the source behind a shared shortcut, check the downstream workspace too. Keeping the same table name doesn't mean the queries behind it will keep working.

Previous gotchas: Part 1 · Part 2 · Part 3.

Salesforce to Snowflake, With Less Friction: Zero-Copy V2 in Practice

Krishna Tangudu — Wed, 28 Oct 2026 11:09:14 +0000

In Part 1, I described connecting Salesforce data to Snowflake. It worked, but the credential setup and Salesforce-style names added housekeeping before analysts could use it.

I have since tested the V2 connector. The enrollment-code handoff is simpler, authentication uses OIDC, and friendly names remove some SQL cleanup. We also kept legacy schema aliases and compared grants before and after migration to preserve existing access.

Here is the setup, what improved, and where my incremental test stopped.

The handoff is much simpler

Snowflake admin                  Salesforce admin
Create connector
Get Enrollment ID -------------> Create Snowflake V2 target
                                 Link selected Data Shares
Mount shared data <------------- Shares become available
Query with SQL

V2 uses OpenID Connect (OIDC). For administrators, the enrollment-code flow replaces the manual credential configuration in my original walkthrough. Salesforce V2 release notes

The source is Salesforce Data Cloud, now called Data 360. CRM data must already be available there. Snowflake exposes it through a catalog-linked database without replicating the shared data into native Snowflake tables. How it works

Step 1: Create the connector and get the code

I used SQL. The example below follows that flow with generic names, using an existing administration schema:

CREATE ZEROCOPY CONNECTOR
  PLATFORM_ADMIN.ADMINISTRATION.SALESFORCE_ZC
  PARTNER = 'SALESFORCE';

SELECT SYSTEM$GET_ZEROCOPY_CONNECTOR_CONFIG(
  'PLATFORM_ADMIN.ADMINISTRATION.SALESFORCE_ZC'
);

The returned configuration contains enrollment_code—the value the UI calls the Enrollment ID. Pass that value securely to your Salesforce administrator. Snowflake connector setup

You can also create the connector through Ingestion → Zero-Copy → Overview in Snowsight. Choose Connect on the Salesforce card; after creation, the setup panel displays the same Enrollment ID.

Your role needs the relevant database/schema access and connector privileges. In particular, creating the connector requires CREATE ZEROCOPY CONNECTOR on its schema; retrieving its configuration requires OPERATE. Privileges

Step 2: Choose what Salesforce shares

The Salesforce administrator needs the Data Cloud Architect permission set and V2 enabled for the org.

In Salesforce:

Open Data Share Targets, create a target, and choose Snowflake V2.
Enter its label and paste the Enrollment ID. Save it.
Create or open a Data Share, selecting the Data 360 objects to expose and reviewing its consent/access settings.
On that share, choose Link/Unlink Data Share Target, select the V2 target, and save.

The target identifies the receiving account. The share identifies the data. Creating a target alone does not publish any objects. Salesforce-side setup in Snowflake’s guide

Back in Snowflake, check the connection:

DESC ZEROCOPY CONNECTOR
  PLATFORM_ADMIN.ADMINISTRATION.SALESFORCE_ZC;

Wait for CONNECTED before mounting.

Step 3: Mount the shares with legacy names available

I mounted all shares and enabled the legacy schema alias:

SELECT SYSTEM$ZEROCOPY_CONNECTOR_LIST_SHARES(
  'PLATFORM_ADMIN.ADMINISTRATION.SALESFORCE_ZC'
);

CREATE OR REPLACE DATABASE SALESFORCE_SHARED
  LINKED_ZEROCOPY_CONNECTOR = (
    CONNECTOR_NAME = 'PLATFORM_ADMIN.ADMINISTRATION.SALESFORCE_ZC',
    ALL_SHARES = TRUE,
    LEGACY_SALESFORCE_SCHEMA_ALIAS = TRUE
  );

SHOW SCHEMAS IN DATABASE SALESFORCE_SHARED;

LEGACY_SALESFORCE_SCHEMA_ALIAS = TRUE keeps the schema_<share_name> alias available, helping existing queries survive the migration. For a generic share called CRM_DATA, this example uses "schema_CRM_DATA". Use the exact case returned in your environment:

SHOW VIEWS IN SCHEMA SALESFORCE_SHARED."schema_CRM_DATA";

Use those views for queries. They can take about a minute to appear after mounting. Mounting and querying

The example reflects the CREATE OR REPLACE form I used. For a new database, plain CREATE DATABASE is sufficient; replacing an existing database is a separate operation, not a required connection step.

Step 4: Give the consuming role access

Mounting the database does not finish the consumer setup. I granted database and schema access, followed by access to both existing and future views:

GRANT USAGE ON DATABASE SALESFORCE_SHARED
  TO ROLE SALESFORCE_ANALYST;

GRANT USAGE ON SCHEMA SALESFORCE_SHARED."schema_CRM_DATA"
  TO ROLE SALESFORCE_ANALYST;

GRANT SELECT ON ALL VIEWS
  IN SCHEMA SALESFORCE_SHARED."schema_CRM_DATA"
  TO ROLE SALESFORCE_ANALYST;

GRANT SELECT ON FUTURE VIEWS
  IN SCHEMA SALESFORCE_SHARED."schema_CRM_DATA"
  TO ROLE SALESFORCE_ANALYST;

ALL VIEWS covers existing views; FUTURE VIEWS covers subsequently created views in that schema. Repeat for other permitted schemas. These grants do not cover future schemas or warehouse access. Snowflake grant syntax

Cleaner names without breaking old references

Legacy experience	V2 naming option
`schema_brz_data` schema alias	Optional; I enabled it in this setup to retain the legacy name
`Case_Home__dll` object	An additional friendly view named `Case_Home`
`AccountId__c` column	`AccountId` in the friendly view

These are aliases, not a rename of Salesforce’s source objects. Friendly names default to enabled, so my SQL leaves SALESFORCE_USER_FRIENDLY_NAME unset. Both naming options are creation-time choices. Naming options

Where my incremental test stopped

Being able to query current data is different from knowing which records changed since the last run.

My stream test on an exposed view failed with this error (object identifiers omitted):

Change tracking is not supported on queries with
'Unmanaged Iceberg Table' inside views.

My understanding is that this exposed-view path did not support the change tracking needed for the stream. Current-state queries worked; an insert/update/delete feed through that path did not.

That is narrower than saying “V2 does not support incremental processing.” Salesforce’s release notes explicitly describe using streams and tasks with V2. Snowflake’s general documentation also distinguishes stream support by table type and Iceberg version; streams on views require change tracking on the underlying tables. An insert-only stream is not a substitute for a complete update/delete feed. Salesforce’s stated capability, Snowflake stream support and limitations

The connector guides do not resolve this specific error. Before relying on downstream history, I would confirm the supported source object and stream configuration, then test one insert, one update, and one delete end to end.

Moving an existing connection

We captured grants before the change, applied the required grants to the new setup, and captured them again to compare. We did not assume that mounting the database would preserve access automatically.

That gives the migration two separate checks:

References: do the database, schema, and view names used by downstream queries still resolve?
Access: do the consuming roles retain the required privileges on the new objects?

Keep the legacy target linked while validating V2, and test representative queries under the consuming roles before unlinking it. Legacy schema aliases help preserve schema references, but do not replace the grants check. Snowflake documents the connection and database-switch procedure; existing connections are not upgraded automatically. Migration guide

For existing users: Salesforce’s current release notes specify October 31, 2026 for migrating legacy targets. Check that notice against your org’s rollout and migration plan. Migration notice

Have you used streams successfully with these Salesforce V2 objects? Which source object and stream configuration worked for you?

The GitHub examples include the V2 setup SQL, consumer grants, and a before-and-after migration checklist.

When Zero-Copy and Mirroring Clash: An Iceberg Metadata Gotcha — Part 3

Krishna Tangudu — Wed, 21 Oct 2026 12:22:51 +0000

Zero-copy and mirroring were supposed to be teammates. Iceberg metadata became the sticking point.

Snowflake's zero-copy clones let us create table copies without duplicating the existing data files. In our Fabric mirroring workflow, the Iceberg source also needed to convert into virtual Delta metadata for consumption.

The source table worked in Snowflake. Fabric's conversion log said BAD_METADATA. The data was present; the clash was in how the table's metadata was handled.

I got all eight tables in this test converting again with a controlled insert, delete, and retry. The temporary row was removed. The business data ended where it started—but the conversion recovered.

Here's what made this a new gotcha.

Zero-copy still needs the right metadata

Iceberg stores data in files and uses metadata to describe which files make up a table. Think of the metadata as its map.

In our setup, Fabric accessed the Iceberg source through a shortcut and generated virtual Delta metadata for consumption. The source being readable in Snowflake didn't prove that this conversion had succeeded.

The diagnostic reported competing metadata sets: a low-version metadata file had been modified more recently than a higher-version file.

The newest file wasn't necessarily the latest version of the production table.

A clone brings another table identity

We had cloned the source tables into other databases. In the cases inspected, source and clone metadata paths led into the same metadata directory.

Cloning doesn't need to change the production rows to create new metadata. Snowflake gives an Iceberg clone distinct metadata, including its own table-uuid. It shares underlying storage, but represents a separate table. Snowflake documents this behavior.

That combination matters for quiet reference tables: production may not have had a recent write, while a newer clone has introduced metadata for another table identity.

Our working explanation is that clone metadata interfered with Fabric's discovery of the intended source metadata. Working tables also had clones, so clone presence alone wasn't a diagnosis; the exact file-selection mechanism remains under investigation.

The recovery I tested

For the affected source tables, I:

Inserted a temporary row.
Deleted that exact row.
Retried the Fabric conversion.

Fabric conversion succeeded for all eight tested tables.

I checked the metadata after the insert and delete. Each of the eight tables advanced by two metadata versions, and the production metadata files had the newest S3 modification timestamps. The writes changed the metadata even though the temporary rows were removed.

Microsoft's guidance also describes committing a source change to trigger another conversion attempt. OneLake troubleshooting.

This is a targeted recovery, not a permanent fix. A temporary row needs a verified unused identifier, and downstream change consumers may still see the insert and delete. I wouldn't automate dummy writes across production.

What to check on the next failure

Read latest_conversion_log.txt under the shortcut's virtual _delta_log folder.
Compare the source and known clone metadata paths, table identities, and actual storage modification times.
After a retry, check conversion status and validate the intended table state through Fabric.

LAST_ALTERED can help prioritize candidates, but it isn't an S3 file timestamp or proof that two tables are clones. It includes DDL, DML, and background metadata operations. Snowflake's TABLES view notes.

Microsoft documents multiple metadata sets in one Iceberg folder as a conversion limitation. Its cleanup example concerns dropped and recreated tables; don't apply that cleanup blindly to files used by live clones.

The lasting fix needs to make the intended table metadata unambiguous to the consumer. For now, the tested recovery gives us a way forward while that is investigated.

Zero-copy can share the files. Iceberg metadata still has to tell the reader which table those files belong to.

Previous gotchas: Part 1 · Part 2.

Have you seen a source table stay readable while its Fabric conversion failed?

Same Shared Data. Different Answers. The Snowflake Timezone Trap

Krishna Tangudu — Tue, 13 Oct 2026 19:04:20 +0000

The provider could see the amount. The consuming application received NULL.

Both were using the same shared data. The query ran successfully. The orders were there. Yet a calculated amount was missing, and a downstream application workflow was blocked.

We started with the usual suspects: permissions, masking policies and changes to the share. The issue turned out to be inside a currency-rate join—where a calendar date was being compared with a timestamp.

The data was present. The join was missing it.

An anonymized case with simplified SQL and illustrative visuals.

Why the order appeared but the amount disappeared

The application consumed a view built on shared data. Its amount field depended on finding an exchange rate for the order's currency and booking date.

The relevant logic looked like this:

SELECT
    o.order_id,
    o.local_amount * r.conversion_rate AS converted_amount
FROM orders o
LEFT JOIN exchange_rates r
    ON o.currency_code = r.currency_code
   AND o.book_date = r.conversion_timestamp;

The currency condition looked ordinary. The date condition hid the problem:

book_date was a DATE: a calendar day.
conversion_timestamp was a TIMESTAMP_LTZ: a moment interpreted using the session timezone.

The query was asking whether a day and an instant were equal. That required an implicit conversion, and the session's timezone became part of the comparison.

Because it was a LEFT JOIN, a missed match preserved the order and supplied NULL for the rate. Multiplying the amount by that missing rate produced NULL too.

That is why a successful query and valid source data were not enough to rule out a calculation problem.

The timezone difference hiding behind the SQL

The provider environment used UTC. The consumer was believed to be using Pacific time. That difference gave us a reason to examine the date comparison instead of treating the NULL as a sharing problem.

Snowflake stores TIMESTAMP_LTZ internally in UTC, while its operations use the session timezone. Snowflake timestamp types

Consider a rate timestamp at midnight UTC. A date interpreted as midnight in Los Angeles represents a different instant. During daylight saving, local midnight is seven hours later in UTC; in winter, it is eight hours later.

The same source timestamp can therefore be involved in a matching comparison in one session and a failed comparison in another.

No rows need to change for this to happen. A session setting is enough to change the interpretation.

Possible fixes and what each changes

Option 1: align the consumer's session timezone with the provider.

For a UTC business-date rule, the session workaround is:

ALTER SESSION SET TIMEZONE = 'UTC';

With the session aligned to UTC, the example's midnight comparison can match. This is a useful diagnostic and possible workaround, but every consuming connection must use the intended setting. Changing it can also affect other date and timestamp logic in the application.

Option 2: extract the date before comparing.

If the intended comparison is at calendar-date grain, make the conversion explicit on the rate side:

-- Before
AND o.book_date = r.conversion_timestamp

-- Explicit date comparison
AND o.book_date = TO_DATE(r.conversion_timestamp)

The result is a comparison between two DATE expressions. It removes the time-of-day equality requirement, but the extracted date must still be the intended business date. With an LTZ input, TO_DATE() can return different calendar dates under different session timezones.

The ideal fix for a UTC business-date rule

Normalize to the business timezone first, then extract the date. This makes the date rule explicit in the shared view instead of depending on each consumer's session setting.

For LTZ rate timestamps whose business date is defined in UTC:

AND o.book_date = TO_DATE(
    CONVERT_TIMEZONE('UTC', r.conversion_timestamp)
)

The conversion produces a timestamp with an explicit UTC offset; TO_DATE() then extracts that calendar date. For the same LTZ instant, the resulting UTC date stays consistent whether the consuming session uses UTC or Los Angeles. For a regional business calendar, substitute the intended region. NTZ inputs also need their source timezone defined. Snowflake timezone conversion

Why does that matter? Midnight UTC can be the previous calendar day in Pacific time. If a rate exists for the adjacent day, a shifted date might return an amount rather than NULL while selecting a different day's rate.

So the validation question is more precise than “did the amount come back?” It is: “did the order match the intended business date's rate?”

Approach	Expected behavior for the midnight-UTC example	Remaining dependency
Set the session to UTC	The date-to-timestamp comparison can match	Every consuming connection needs the intended setting
Use `TO_DATE()` alone	Compare calendar dates instead of exact instants	The extracted LTZ date can shift with the session timezone
Convert to UTC, then use `TO_DATE()`	Compare the explicitly defined UTC business date	UTC must be the correct business calendar; the rate key must identify the intended row

These are expected behaviors of the simplified example, not measured production outcomes. Explicit timezone conversion cannot supply a genuinely missing rate or resolve multiple rates for the same key.

What I would test for the next shared view

A small set of checks can expose this class of issue early:

Inspect the effective timezone in the actual application connection with SHOW PARAMETERS LIKE 'TIMEZONE' IN SESSION.
Test the same date logic under the provider and consumer session settings, including winter and summer boundaries.
Check the selected rate's business date, not just whether the calculated amount is populated.
Confirm that currency plus the chosen date identifies the intended rate. Multiple matches can duplicate orders.

Account defaults are useful context, but users and sessions can override them. The application's effective setting is what matters. Snowflake parameter hierarchy

The lesson I took from this was simple: when the source has a value and the consumer sees NULL, follow the calculation. An unmatched join can look like missing data, and an implicit conversion can quietly make the result depend on who runs the query.

A shared view should make the business meaning of its dates explicit.

Have you encountered a query that succeeded while an important join quietly failed? Share the pattern in the comments, or connect with me on LinkedIn.

Leaving SAP HANA? Understand What You’re Taking With You.

Krishna Tangudu — Sun, 04 Oct 2026 07:33:53 +0000

How we are using an enterprise data agent to understand HANA logic before deciding what to rebuild in Snowflake or Microsoft Fabric.

A HANA report can be rebuilt in Snowflake or Microsoft Fabric, run successfully, and still change what its columns mean.

Take a reference number. Some records get it from a table, others through an older reporting model, and manually entered records get fixed text. Rebuilding all three the same way could change reporting behavior without producing a SQL error.

Our enterprise data agent runs on Snowflake Cortex and helps investigate that logic. The useful outcome is a handoff an engineer can inspect before deciding what to preserve, simplify or retire: what is understood, what is missing, and what still needs testing. The format below is our proposed handoff, not an automatically validated migration deliverable.

Engineering cases are anonymized. The service-report example is fictional, not SAP-delivered content or production output. Results come from selected tests, not a controlled benchmark.

One report column, three different routes

Think of a HANA calculation view as a recipe for a dataset: where records come from and how to combine, filter or calculate them. A view can read other views, creating layers. It can also combine separate inputs, or branches, each of which may pass through several layers.

Think of a service report that combines current records, archived records and manually entered records. All three contribute to the same ReferenceNumber column:

The moving dots follow each input path, stopping where archive evidence is missing. An engineer needs to know what happens along those paths: which field supplies the value, how it changes, and whether records are filtered out. Our early answers could name the view without finishing that explanation.

Find the logic, then follow its sources

First, the agent needed the actual logic. In one case, its searchable material contained a model's outer definition but not the calculations underneath—like a recipe title without the instructions. We added deployed view definitions and executable code, preserving full names and locations to distinguish similar models.

Then we changed how the material was split up.

Large definitions had been divided into smaller pieces for search. This is often called chunking. A search for ReferenceNumber might return the current-record mapping while missing the archived-record mapping elsewhere in the definition.

For our example, the current-record mapping says which field supplies the reference number. The archive mapping describes another source, and the manual mapping supplies fixed text. Returning just the first one could make an engineer miss two different behaviors.

We changed the grouping so those related mappings could be retrieved together. Search helped find the model; an exact lookup then retrieved its evidence and checked whether every expected input was present.

The lesson was practical: the way you divide source material affects what the agent can explain.

Even the right source name may be an intermediate step. HANA models can use a synonym, an alias that points to another object. We added those deployed alias connections and a field catalog describing mappings and relationships. If an alias led to another view, the investigation had to continue inside it.

For this report, that means the current-record path can reach an activity table. The archived-record path must remain unresolved until the deeper definition is available. The manual-record path ends at fixed text; there is no source field behind that value.

After these changes, a recorded test of a selected model and column returned physical source paths that the earlier answer had left unresolved. That was a useful improvement, with a clear limit: resolving those paths did not prove complete coverage of every HANA model.

Correct evidence still needed a faithful explanation

Our previous article asked whether an agent's evaluation was judging the right thing. Here, the practical test was whether its explanation preserved the source behavior.

We found answers that retrieved the right mappings but counted them incorrectly. A parser—a small program that reads the model definition and checks its structure—could count the inputs, but the agent did not consistently run it. We put the count checks into the existing lookup so the answer received checked facts.

Another answer described a fixed placeholder as SQL NULL. In the fictional report, that would mean treating NOT_APPLICABLE as a missing value. NULL means no value; NOT_APPLICABLE is actual text. A filter or report can treat them differently. An engineer following the explanation could therefore change the behavior accidentally.

We corrected the instructions and repeated an ordinary question. The selected test returned the expected paths and preserved the literal values. One generated query still failed before the agent recovered, so this was progress rather than error-free execution.

We also made incomplete answers harder to overlook. The deployed response format uses an orange PARTIAL notice for missing evidence and a red UNVERIFIED notice for unsupported conclusions. In our fictional example, the useful message is specific: “The archive path stops at a view whose definition is unavailable.” A warning should name the gap, not just say confidence is low.

Our own guidance also needed attention: one rule allowed guesses when local evidence was missing, while another required evidence. Parsing was optional in one place and required in another. The design now gives shared rules to the agent instructions and detailed HANA steps to a skill, a task-specific playbook. We also deployed a cleanup that removed duplicated instructions while retaining each rule once. This is easier to maintain; better accuracy or speed still requires evaluation.

For an engineer, the acceptance test remains concrete: account for every input path, preserve calculations and fixed values, and state where the evidence stops.

To make recurring questions easier, we added reviewed question-and-SQL examples through Cortex Analyst's Verified Query Repository. A query that worked directly against the database initially failed through the agent's more limited tool. Splitting retrieval across the appropriate tools corrected that tested request. The lesson: test through the interface the agent actually uses; a verified example guides retrieval but does not certify the answer.

Understanding the old design gives you choices

Now return to the migration decision. Knowing how the report works does not mean rebuilding every view exactly as it was.

An engineer might preserve a business rule, simplify repeated calculations, or retire an input that the business no longer needs. For this report:

Evidence in the fictional example	Engineering decision	Next validation
Current records resolve to a known field	Preserve its meaning in the target mapping.	Compare matched records from the same source snapshot, including any transformations.
The archived path stops at another view	Investigate before implementing, retaining or retiring that path.	Obtain the missing definition and check its mappings and filters.
Manual records contain `NOT_APPLICABLE`	Keep the literal, or explicitly agree a different representation.	Test the literal separately from SQL `NULL`, including how reports filter each.

These are proposed engineering checks, not tests completed for this fictional model.

A source explanation is not a data test.

Reading a model definition can establish mappings and filters. It cannot establish the row counts produced by a rebuilt query. If the agent cannot execute that query through an available tool, label the handoff “Query prepared — not executed” and leave the results unknown. Asking it to continue does not create access.

In Snowflake or Fabric, one possible design is an activity fact table, where each row represents a defined event, with shared customer and service dimensions holding descriptive information. In Fabric, a Power BI semantic model could then define the reporting measures and relationships. Some workloads may still benefit from views.

This is an architectural choice, not a rule that HANA uses views and other platforms use facts and dimensions. HANA supports dimensional modeling too. SAP HANA modeling documentation

Conversion tools have a place in that journey. SnowConvert AI and Microsoft's Fabric Migration Assistant support migration work for their documented source platforms. SAP HANA is not listed on those two capability pages checked on October 3, 2026. Our work explores how source understanding can inform the decisions around a migration; we have not demonstrated automatic migration or equivalent results on a target platform.

For teams building migration tools, that raises an interesting question: could the workflow show engineers what they are about to rebuild—and the evidence behind it?

This continues our series from recovering legacy knowledge, through governed semantics and evaluating agent answers, into a specific engineering task.

Before asking how to move a HANA model, we can ask a better question: what does it do, and which parts still belong in the new design?

Want to explore the approach or ask about the code? Connect with me on LinkedIn. Tell me which part interests you: collecting HANA definitions, finding the relevant logic, or checking where fields come from.

My Snowflake Agent Was Wrong. So Was My Evaluation.

Krishna Tangudu — Mon, 28 Sep 2026 03:00:59 +0000

What production conversations taught me about fixing the right layer—and checking whether the fix actually worked.

When an agent gets a disappointing evaluation score, I now ask three questions:

Did it answer the question correctly?
Did it take an appropriate path to the answer?
Was the evaluation measuring the behavior I actually wanted?

In my Snowflake agent, those answers did not always agree. A revised lookup recovered an object but still received partial tool-selection credit. Other test expectations omitted steps my instructions required. And one application counter made missing telemetry look like zero tool use.

I was maintaining three things at once: the agent, the evidence about its behavior, and the tests judging it. Changing the prompt was only one possible fix.

These examples are drawn from my work with a Snowflake Cortex Agent and have been anonymized. The outcomes described are specific observations and retests, not a controlled benchmark.

The fix lived between the agent and its tool

My agent said an object did not exist. The metadata contained it—but as a source consumed by other views, not as the view the agent was searching for.

I added a fallback instruction. The lookup still failed.

The investigation then suggested the semantic tool could not search by source. Inspecting its definition corrected that explanation: the source dimension already existed. Its SQL-generation guidance emphasized view-name searches.

The successful revision changed both layers. The agent received a fallback telling it when to ask which views consume this source? The semantic view received guidance explaining that lookup to Cortex Analyst. The recorded retest recovered the object and its consumers.

An anonymized reconstruction. The source dimension already existed; the change clarified its use.

This matters because “the data is missing,” “the tool cannot do it,” and “the agent did not ask correctly” lead to very different fixes. The investigation briefly entertained each explanation. Inspecting the definition and retesting was more useful than accepting the first diagnosis.

Even the successful retest had a boundary. Finding downstream consumers did not establish how every upstream object was loaded. A later correction in the conversation made that distinction explicit. A lineage lookup should not turn into an unsupported explanation of the ingestion architecture.

The evidence I actually used

I read real questions, responses, and follow-ups alongside native execution traces. A scheduled Cortex Code review triaged instruction gaps, data gaps, and tool limitations; its findings were leads to investigate. I materialized trace summaries to retain a longer investigation trail. That addressed the history available in my environment, not a universal retention limit. Snowflake monitoring documentation

Comparing sources mattered: my application's tool-call counter defaulted missing metadata to zero, while native traces showed activity it had missed. A missing measurement had been made to look like a measured zero.

How I judge an evaluation

“What was the score?” needs a second question: the score for what?

Snowflake separates several checks:

Check	What it judges
Answer correctness	An LLM judges the answer against expected content.
Tool selection accuracy	Deterministic matching of expected and actual tool names and call counts; order is ignored.
Tool execution accuracy	Expected inputs and outputs are compared with matching tool invocations.
Logical consistency	An LLM checks consistency across instructions, planning, and actions without reference answers.

Tool selection can penalize extra calls even when the answer improves. A low tool-selection score is therefore not an answer-accuracy percentage. The mechanics are in the appendix and Snowflake’s evaluation documentation.

That distinction explained some of my confusing results. Expected tool lists sometimes omitted prerequisites required by the instructions. Other cases allowed only one route where more than one route could be appropriate.

But I should not make a test easier just because the agent failed it. Each change to the expected behavior needs an independent reason: a verified alternative route, a documented prerequisite, or a correction to the case itself.

The missing-object case illustrates why I inspect individual records. The combined fix recovered the lookup, while extra calls still limited its tool-selection score. That supported a narrow conclusion: the retrieval behavior improved in that retest. It did not prove that every returned statement was correct or that the whole agent improved.

Real questions are good test inputs. Old answers are not automatically ground truth.

Real conversations supply questions I would not invent in a demo. Turning them into tests also creates traps. A follow-up such as “generate the query for this model” loses its meaning when detached from the preceding conversation. A previously successful API response may contain a wrong answer. A time-sensitive answer can become stale.

For a useful regression case, I need the question's context, independently checked expectations, acceptable uncertainty, and the behavior that must not recur. I also keep successful examples so that a targeted fix does not quietly damage an existing workflow.

Changing the questions or expected answers creates a new test baseline. Comparing its average with an older baseline as if only the agent changed would overstate the improvement.

Did the test exercise the capability?

The traces also challenged what I thought my tests covered. After enabling a Python sandbox, even XML-focused cases did not show its use in the recorded inspection. A targeted test described programmatic parsing but followed the existing retrieval-and-skill path. Configured, mentioned in planning, and invoked are different states. A correct answer through another route could pass an answer test while leaving the sandbox untested. I needed evidence of invocation and correct extraction before attributing an improvement to Python.

The right answer can still be the wrong interaction

Another incident involved an object name with words in the wrong order. The agent found a plausible alternative and began analysing it. The user had to correct the selection.

I introduced similar-name search and confirmation. Tool checks showed that candidate retrieval worked. Then I tested through the application.

The revised agent found the intended candidate—and continued into analysis without asking me to confirm it.

Retrieval had improved. The interaction still violated the requirement.

Illustrative animation. The conversation records my confirmation that the strengthened rule worked in a manual retest; it does not establish a broad success rate.

The stronger instruction specified the stopping boundary: present candidates, ask which one to use, and end the response before doing lineage or column analysis. It also included examples of the unwanted and intended behavior.

For this case, my proposed regression checks are concrete:

Situation	Expected behavior
Exact object exists	Analyse that object.
Exact object is absent; alternatives exist	Present candidates and ask; do not analyse a substitute yet.
No candidates exist	Explain the search boundary without inventing an object.
User confirms a candidate	Continue with the confirmed object.

Here is a synthetic test pattern a reader can adapt. It is a proposed regression fixture, not a reproduced production test or response:

Test: approximate object name requires confirmation
Given:
  Exact-name lookup returns no match.
  Candidate lookup returns synthetic objects A and B.
Before: the agent chooses a candidate and begins analysis.
Required after:
  Present A and B with distinguishing context.
  Ask the user to choose, then end the turn.
Pass only if:
  The final response asks for a choice AND contains no
  lineage results, column analysis, or assumed selection.
Fail if:
  The agent analyses either candidate before confirmation,
  even if it also includes a question.
Next turn:
  User selects B; analysis must refer to B, not A.

Use fixed lookup fixtures to isolate the interaction rule, then repeat through the real application with its actual tools. A question mark alone is not a pass. Inspect the meaning of the response and the trace for premature analysis.

A final-answer similarity score alone would miss part of that contract. I need to check the turn where the agent was supposed to stop.

There was a deeper reason to care about that pause. During these reviews, I worried that someone less familiar with our domain might accept a confident response without knowing when to challenge it. A domain expert might catch the wrong object; another user might build on the explanation. That concern was not a measured comparison between user groups. It changed how I reviewed answers: lack of pushback could not count as evidence of correctness. Asking for confirmation exposes a choice the agent would otherwise make silently, though confirmation alone does not verify the analysis that follows.

Each revision should explain its reason

My changes were not all additions to the main prompt. They addressed different layers:

Symptom	Layer to investigate
Wrong object selected without confirmation	Agent interaction instructions
Metadata exists but the lookup asks the wrong question	Agent routing and semantic-view guidance
Specialist parsing or traversal is incomplete	Domain skill and source coverage
Needed computation is unavailable	Tool capability, followed by invocation tests
Generated output cannot be accessed in the application	Delivery channel and response format
Plausible answer receives unexpected evaluation penalties	Expected behavior and per-case scoring
Tool use appears absent in one dashboard	Instrumentation and native trace evidence

A deployment mistake changed my rule. The agent was recreated during this work, resetting its native version history while our conversation still used the old labels. I explicitly said not to recreate it again without asking me. For routine revisions, my preferred workflow became modifying and committing a version of the existing agent; recreation needed a separate, explicit decision. A configuration edit should not casually become an object replacement.

That experience made the release record concrete: connect each change to the native agent identity and version, skill revision, semantic-view definition, dataset, and scoring configuration. A conversation label cannot substitute for that record. The appendix shows selected version reasons.

None of this happened in a quiet, finished post-mortem. I was correcting evaluation expectations while also checking whether recent questions from stakeholders had received reasonable answers, responding to feedback, and inspecting the next interaction. The service remained in use while I was learning how to evaluate it. The tidy sequence in this article emerged from that overlap; it was not a process I had perfected before users arrived.

What I would do first on the next agent

I did not begin with a formal AI lifecycle. I began with people correcting the agent.

That grew into a repeatable engineering practice: retain the relevant evidence, investigate the failure, change the appropriate layer, and test the behavior again. Keep observations separate from hypotheses. Keep a successful component test separate from an application retest. Keep a better evaluation score separate from a better answer.

If I were starting again, I would write the regression case before the fix:

What should the agent do differently when someone asks this again—and what evidence would convince me it did?

That question has been more useful than asking whether the agent is finally “good.”

Appendix: the details behind the story

Evaluation mechanics

For tool selection, the documented formula is matched calls divided by the larger of expected entries or actual calls. An illustrative, invented example: one expected call and four actual calls, with one match, scores 0.25. That does not mean the answer is 25% correct. Tool execution handles extra calls differently. Its inputs and outputs are optional; omitting both checks invocation presence rather than execution quality. Tool-level coverage is limited, so unsupported tool behavior needs another check. Cortex Agent evaluations

Retaining an investigation trail

I materialized trace summaries on a schedule to extend the investigation window, with known limitations around join precision and late-arriving spans.

Selected version reasons

Selected native versions and their reasons. These labels belong to the retained history after agent recreation; they do not measure performance gains.

Natural Language Is the Interface, Not the Semantic Layer

Krishna Tangudu — Tue, 15 Sep 2026 18:48:24 +0000

This is Part 2 of a two-part technical series. Part 1 showed how an Enterprise Data Discovery Assistant recovers legacy logic and produces grounded SQL. Rupak's article on AI-ready data products provides the larger context: agents need machine-readable semantics, active contracts, and engineering guardrails. This post follows the next step—from a business question to governed execution and an explained answer.

All names, identifiers, queries, and results below are synthetic. Quantities are rounded to communicate scale without publishing an internal inventory.

Natural language does not remove the need for data modeling. It makes good data modeling visible to more people.

Our preview tests asked whether business users could explore an enterprise-scale Customer 360 domain—more than 100 million records across multiple curated tables—without first learning schemas, joins, or SQL.

The answer was encouraging, with an important qualification: natural language is the interface, not the semantic layer.

One question hides several decisions

Consider a straightforward request:

Which customer segments had the highest order value last quarter, and how did that change from the previous quarter?

Before SQL can run, the system must resolve what order value means, which field represents customer segment, how orders relate to customers, which calendar defines last quarter, whether the user is allowed to see the requested data—and whether the data product is trustworthy right now.

A language model looking only at table and column names may guess. A semantic view makes the analytical decisions explicit: grain, metrics, time semantics, filters, and supported relationships. Snowflake policies determine who may access the result, while our runtime contract check determines whether the data product is fit to use now.

It defines:

Logical tables, row grain, primary keys, and unique keys, so the meaning of one row is explicit.
Dimensions and time dimensions, such as customer segment and fiscal quarter.
Facts and governed metrics, including aggregation and distinct-count behavior.
Relationships, including bridge tables for many-to-many associations, so join paths are modeled rather than improvised.
Filters, synonyms, and question-handling instructions.
Verified queries, which pair important question patterns with reviewed SQL.

Snowflake recommends semantic views for new Cortex Analyst implementations. They are schema-level objects integrated with Snowflake privileges and metadata—not prompt text pretending to be governance.

A semantic view tells the assistant what the data means. A data contract tells it whether that data product is fit to use now.

For governed business questions in our preview, the assistant performs a mandatory contract check before using the analytical semantic view. It evaluates the target domain's contract health, freshness, latest validation result, and any failure reason.

The contract check is application logic around the agent; it is not performed automatically by the semantic view itself.

From question to governed answer

The flow is deliberately simple:

The Enterprise Data Assistant identifies the user's intent and target business domain.
A data-contract check evaluates contract status, freshness, quality, and the latest validation outcome before the business query runs.
The contract result determines the response path: proceed normally when trustworthy, or attach the appropriate warning when the data is degraded, stale, inactive, or failed.
Cortex Analyst interprets the question using the selected semantic view, which supplies the governed dimensions, metrics, relationships, filters, and examples needed to generate SQL.
The SQL runs inside the governed data platform. Snowflake privileges and data-protection policies still apply.
The response returns the result together with the SQL, definitions, scope, and any required warning.

The distinction matters. When the contract is active, fresh, and passing, the check stays behind the scenes and the user gets a clean answer. A stale or degraded contract adds a caution. An inactive or failed contract produces a prominent warning that the result is not guaranteed before the preview attempts the query. For a higher-risk domain, the same decision point can be implemented as a hard stop.

Cortex Agents can use Cortex Analyst for structured data and route across multiple semantic views. Cortex Analyst first attempts semantic SQL, where metrics, dimensions, and relationships come from the semantic view. When the modeled coverage cannot satisfy a request, Routing Mode can fall back to standard SQL on physical tables. That flexibility is useful, but for governed metrics the fallback should be treated as a different confidence path: expose it for review, request clarification, or block it for higher-risk questions rather than imply that it carries the same semantic guarantees. Snowflake documents this behavior as Routing Mode.

The answer should show its work

The experience should still feel conversational. The difference is that the answer exposes enough evidence for a user or engineer to challenge it.

Synthetic demonstration: no production interface or customer data is shown.

For the example above, the response contract is more important than the visual polish. It should make six things clear:

What the user sees	Why it matters
A concise answer	The user gets the result without reading SQL first.
Metric and dimension definitions	Business terms are not left open to interpretation.
Time range and filters	The scope of the answer is explicit.
Contract warning, when relevant	Users know when freshness or quality may affect trust.
Generated SQL	An engineer can inspect and reproduce the query.
Warnings or clarification	Ambiguity is surfaced instead of silently resolved.

That is a more useful standard than “the chatbot returned an answer.”

Accuracy is an engineering loop

Semantic metadata improves grounding, but it does not prove that every generated query is correct.

Important question patterns should be captured in a Verified Query Repository: a natural-language question paired with SQL whose logic and result have been validated by a qualified reviewer. The SQL should use the logical tables and columns defined by the semantic view rather than bypassing them for physical objects. Cortex Analyst can use relevant verified queries to guide similar requests. A verified example is guidance—not a blanket certification of every future answer. Snowflake's Verified Query Repository documentation makes that distinction concrete.

The next layer is repeatable evaluation. Current Cortex Analyst evaluations compare generated SQL results with selected verified queries, track regressions, and record latency. This turns semantic-layer tuning into a measurable build-test-run-improve cycle instead of a collection of impressive demos. Snowflake documents the evaluation workflow here.

We also learned to compare results, not only SQL. Reconciliation must use the same as-of timestamp, fiscal calendar, currency logic, exclusion rules, and entity scope. Two syntactically valid queries can still answer different business questions.

Our practical checklist is:

Start with a narrow, coherent business domain.
Define grain, keys, relationship cardinality, metrics, filters, and synonyms explicitly.
Add reviewed queries for high-value and high-risk questions.
Test ambiguous wording, invalid requests, boundary cases, and every contract state—not only the happy path.
Track accuracy, regression, latency, and workload cost with representative questions.
Show the generated SQL and scope when the audience needs traceability.

Governance does not disappear behind chat

A conversational interface must not become a shortcut around access controls.

Access control and data contracts answer different questions. Privileges determine whether this user may access the data. The contract check determines whether the selected data product should be trusted in its current state. A governed assistant needs both decisions; neither replaces the other.

Semantic views participate in Snowflake's privilege model. Row-access and masking policies applied to underlying tables can propagate to the semantic view and remain enforced. One subtle but important caution from Snowflake's guidance is that sample values stored as semantic metadata are not masked, so sensitive examples should not be embedded there. See Snowflake's development and deployment guidance.

There are product boundaries too. Cortex Analyst is designed for questions that can be resolved with SQL; it is not automatically a general business-insight engine. It also cannot refer to the results of a previous SQL query as if it had retained that result set. Those limits should shape both the user experience and the test suite. The current limitations are documented here.

What changed between Part 1 and Part 2

Part 1 recovered the knowledge needed to rebuild a data product: joins, filters, calculations, ownership, and dependencies.

Part 2 makes approved knowledge queryable:

Recovered legacy knowledge
          ↓
Governed data product + semantic view + contract
          ↓
Natural-language question
          ↓
Data-contract decision
          ↓
Semantic grounding + governed SQL
          ↓
Explained result

The language model is useful at the top of this stack because the hard decisions are represented beneath it.

That is the larger lesson from the preview. Natural-language analytics is not a replacement for engineering discipline. Done well, it is a new interface to that discipline—one that lets more people ask useful questions while keeping definitions, access, SQL, and validation visible.

Mirroring Snowflake Iceberg into Microsoft Fabric : The Gotchas - Part 2:

Krishna Tangudu — Thu, 10 Sep 2026 17:32:05 +0000

Previously: In Mirroring Snowflake Iceberg into Microsoft Fabric: The Gotchas, I documented connection issues, permissions, and the difference between healthy mirroring and a usable SQL analytics endpoint.

The goal remains the same: give analysts a shared serving layer in Fabric without making every downstream workflow query Snowflake independently.

Since that post, two support investigations have added to the picture. One concerns the cost of continuous mirroring when the source changes only a few times daily. The other reverses an earlier troubleshooting pattern: the SQL analytics endpoint reads the data, but Spark fails.

This is the checklist of what we have learned, which workarounds we have used, and the options we plan to evaluate next. Checked items describe experience or completed investigation. Unchecked items are future tests, not claimed results.

1. What we have done so far

[x] Established the Snowflake–Fabric integration. The first post covers the connection and permission fixes, including separate access to Iceberg storage.
[x] Separated replication health from query behavior. We validate actual results through the consuming engine, rather than treating a healthy status as sufficient.
[x] Compared SQL analytics endpoint and Spark behavior on the decimal issue. SQL succeeded where the affected Spark read failed.
[x] Validated the source, table, and file schemas. We submitted those findings to Microsoft support.
[x] Tested the non-vectorized Spark workaround. Reads succeeded; we subsequently reported performance and capacity impact.
[x] Investigated mirroring schedules, restart behavior, and Delta CDF. Support confirmed the limitations described below.
[ ] Measure whether Snowflake AWS-to-Azure replication improves the total economics. This is our next architecture experiment.

2. Continuous mirroring: the business clock matters

Our source data changes approximately three times per day. The mirroring support case described a Snowflake cloud Layer cost spike and asked a practical question: can we run replication around those updates instead of continuously?

At roughly 2,500 tables across the estate, maintaining an individual pipeline for every table would also be a substantial burden. That concern is about operational scale; it does not mean every batch design requires a separate pipeline per table.

What the investigation established:

[x] Scheduling: Microsoft documents no configurable mirroring schedules or replication windows today. Snowflake mirroring guidance
[x] Stop/start behavior: support confirmed a full reload rather than continuation from the previous CDC position, consistent with the public FAQ. A timer around stop/start is therefore not equivalent to a scheduled incremental load. Snowflake mirroring FAQ
[x] Disabling the service user: support explained that this is not a supported scheduling mechanism; connection retries and recovery become part of the problem.
[x] Alternative suggested: support recommended evaluating a Copy activity or Copy job, while acknowledging that this requires architectural work.

The case is closed, but the correspondence acknowledges a scheduling limitation. It does not record delivery of a scheduling fix.

3. The decimal issue: SQL worked, Spark did not

The second case captured this error, with the column anonymized:

Parquet column cannot be converted ...
Column: [AMOUNT], Expected: decimal(15,2), Found: INT32

Our diagnostics showed:

Layer	Reported representation
Snowflake source column	NUMBER(15,2)
Fabric table schema	decimal(15,2)
Examined Parquet file	decimal(9,2), backed by INT32
SQL analytics endpoint	Query succeeded
Affected Spark read	Failed with vectorization enabled
Spark with vectorization disabled	Read succeeded

The logical table schema and a file's physical encoding are different layers. The failing read path did not successfully reconcile the narrower file representation with the declared decimal type. The observed result was a read failure, not evidence of corrupted business values.

Microsoft's public guidance lists the type-width issue and the Spark workaround. OneLake Iceberg limitations

[x] Workaround used: disable the vectorized Parquet reader in the affected Spark session.
[x] Impact reported: performance and capacity concerns

Here is a scoped illustration of the workaround, using fictional table and column names:

# Fabric PySpark: use a small, known slice of an affected table.
setting = "spark.sql.parquet.enableVectorizedReader"
original = spark.conf.get(setting)

try:
    spark.conf.set(setting, "false")
    rows = spark.sql("""
        SELECT order_id, amount
        FROM dbo.orders_iceberg
        WHERE order_id IN (101, 102, 103)
    """).collect()
    print(rows)
finally:
    spark.conf.set(setting, original)

The action runs before the setting is restored. This example illustrates the configuration used; it is not a new benchmark or a reproduction executed for this article. A simple query showed a performance difference from 20 seconds to 2 minutes 26 seconds.

The lesson extends my previous post: SQL and Spark both need independent acceptance tests. Either engine can expose a limitation that the other does not.

4. Options on our checklist

Option	Our status	Why it is relevant	What still needs proof
Existing Snowflake mirroring / Iceberg integration	Used; support cases investigated	Shared Fabric serving layer	Sustainable cost and consistent engine behavior
SQL analytics endpoint for affected reads	Worked in the reported case	Keeps compatible SQL workloads usable	Coverage of workloads that currently require Spark
Spark non-vectorized reader	Tested workaround	Restores affected reads	Performance and capacity at production scale
Scheduled Copy job or reusable Copy pipeline	Official option; not benchmarked here	Direct control over movement frequency	CDC eligibility, deletes, schema changes, cost, and management at scale
Snowflake AWS → Snowflake Azure replication	Planned next experiment	Place a controlled replica nearer Azure consumers	Eligible objects, consumer access, freshness, and total cost
Direct Snowflake access	Tested workaround	Keeps users productive	Source load and total BI/query cost

5. What comes next: Snowflake AWS → Snowflake Azure

Our hypothesis: a scheduled Azure replica may be cheaper to operate when it replaces substantial repeated reads across the cloud boundary. We have not established savings yet.

The candidate flow is:

Snowflake AWS source
        |
        | scheduled Snowflake replication
        v
Snowflake Azure secondary + target-region storage where required
        |
        | consumer path to validate separately
        v
Fabric Iceberg access

Snowflake officially supports replication across AWS and Azure within an organization. Replication groups provide read-only secondaries and configurable refresh schedules. Database replication is available across editions; failover groups and various account-object capabilities have additional edition requirements. Replication overview, Replication schedules

We are evaluating replication for read access. Failover is a separate operating requirement and should not be introduced merely to make a read-only secondary writable for an integration.

6. The cost test that will decide the direction

Snowflake replication adds transfer and service-compute charges, billed to the target account, plus target storage costs. Refresh frequency and changed data volume affect the total. The initial seed must be measured separately from steady-state refreshes. Snowflake replication cost

Our comparison will be:

Current path:
  source integration activity + transfer/storage requests
  + Fabric serving cost + operating effort

Azure replica path:
  replication transfer + replication compute + target storage
  + Azure-side integration/query cost + Fabric serving cost
  + operating effort

These are measurement categories, not a savings estimate. For external Iceberg storage, include the applicable cloud-storage bill as well as Snowflake and Fabric usage.

My current direction is to test Azure replication. The former tests whether data location is driving unnecessary cost; the latter tests whether movement frequency is the larger problem. The decimal issue remains its own compatibility test.

That is what I want the next installment to report: which option we tested, what improved, what it cost, and what still required a workaround.

Moving Data Is Easier Than Moving Knowledge

Krishna Tangudu — Mon, 07 Sep 2026 04:15:25 +0000

This is Part 1 of a two-part technical series. Rupak's article on building AI-ready data products describes the larger destination: a machine-readable context layer, governed semantics, and engineering guardrails. This post tackles the migration problem that comes first—recovering legacy knowledge and turning it into grounded SQL. Part 2 will cover natural-language questions, governed execution, and explained answers.

All examples and names below are synthetic

Moving data is often the easier part of an SAP-to-cloud migration. Moving the knowledge is harder.

A pipeline can copy a table. It does not automatically carry forward why two tables are joined, which filter defines an open order, how technical statuses become business terms, or who owns the report.

Without that context, a new platform can produce a technically valid answer that is wrong for the business.

A question the copied tables cannot answer

Imagine an estate with more than 5,000 SAP HANA calculation views, hundreds of Qlik and Spotfire applications, and more than 2,500 cloud tables.

A data engineer receives a simple request:

Rebuild our sales-order backlog dashboard. What existing logic should I preserve?

The answer is scattered. A HANA calculation view may define the joins, filters, and open quantity. A Qlik application may map technical statuses and write a shared QVD. A Spotfire analysis may implement its own aging logic. Ownership and downstream usage may live in reporting metadata, while field-level dependencies live in a lineage graph.

Qlik and Spotfire are parallel reporting platforms over HANA. They are not sequential stages feeding one another; the same HANA model may support applications in either or both.

The migration task is therefore not just code conversion. It is evidence discovery.

How the assistant builds context

We call the user experience an Enterprise Data Discovery Assistant. Under the hood, a bounded Snowflake Cortex Agent selects retrieval and analysis tools. “Agent” describes the architecture; “assistant” describes its read-only responsibility. It does not change source systems, deploy code, or promote pipelines.

Engineer question
       |
       v
Enterprise Data Discovery Assistant
       |
       +-- HANA repository code
       +-- Parallel BI artifacts
       |      +-- Qlik scripts and QVDs
       |      +-- Spotfire queries and metadata
       +-- Reporting metadata catalog
       +-- Schema, object, and column lineage
       |
       v
Recovered logic + ownership + dependencies
       |
       v
Grounded SQL draft -> data-engineer validation

The four evidence layers play different roles:

HANA repository code provides the calculation-view definition. Parsing indexed .hdbcalculationview XML reveals sources, joins, filters, calculated columns, unions, and nested-view dependencies.
Parallel BI artifacts reveal reporting logic. Qlik contributes load queries, aliases, mappings, resident loads, and QVD flow. Spotfire contributes source queries, columns, and any calculations present in its indexed metadata or exports.
The reporting metadata catalog connects Qlik and Spotfire applications to HANA models, queries, fields, owners, paths, saved-query details, and comments. It adds operational context around the code; it does not replace the code.
Schema and lineage metadata traces upstream sources, downstream consumers, object dependencies, and column paths, then helps map recovered logic to approved target objects.

Together, these sources answer more than “What SQL should I write?” They can also answer: Who owns this dashboard? Which reports use this HANA view? Which QVDs feed the application? What could break if the model changes?

Search locates the artifacts. Deterministic parsers extract their structure. The language model relates the findings to the engineer's question.

From a question to grounded SQL

The experience should feel like a conversation, not a metadata report. The engineer asks one question; the assistant shows what it checked and returns an evidence-backed answer with SQL.

Synthetic chat demonstration: no production data or internal interface is shown.

The response separates recovered facts from the proposed target design. In this example, the assistant verifies the HANA joins and filter, recovers the Qlik status mapping and QVD flow, identifies the owner, and finds downstream Qlik and Spotfire consumers before drafting SQL.

Only then does the assistant propose a starting point:

SELECT
    i.order_id,
    i.ordered_qty - i.confirmed_qty AS open_quantity,
    s.business_status
FROM sales_order_item AS i
LEFT JOIN status_map AS s
    ON i.status_code = s.status_code
WHERE i.cancelled_flag = 'N';

The SQL is useful because its origin and limitations are visible—not because it appeared quickly.

Grounded SQL is a handoff

The draft is not a production pipeline. A data engineer still confirms the grain, compares results with the legacy report, maps the logic to canonical target names, and preserves null, fallback, and refresh behavior. Production work also requires incremental patterns such as CDC or watermarks, data-quality tests, security, documentation, owner approval, and CI/CD which was covered in Developer Toolkit.

Trust comes from a few explicit boundaries:

Prefer exact identifiers when available; use semantic search for discovery, then verify the selected objects.
Parse large HANA XML and Qlik scripts deterministically.
Separate verified evidence, proposed design, and missing evidence.
Keep generated code read-only until it passes normal engineering review and testing.

If an expected Qlik or Spotfire expression is absent from the indexed sources, the assistant says so instead of reconstructing it from assumptions.

The same human-in-the-loop pattern shaped the assistant itself. Snowflake CoCo,supported the build-and-tune cycle by helping refine tool instructions, test prompts, failure analysis, and response behavior. Engineers decided which changes to adopt and validated the results.

Moving data gives a new platform rows and columns. Moving knowledge preserves the meaning, ownership, and dependencies people rely on—and provides the context needed to build governed data products.

Part 2 will follow a natural-language question through governed SQL execution and show how the final answer explains its sources and limitations.

From Optimization to Protection: Adding a Security and Governance Agent to Your Snowflake Multi-Agent Team (Part 3)

Krishna Tangudu — Thu, 09 Jul 2026 21:26:01 +0000

From Optimization to Protection: Adding a Security and Governance Agent to Your Snowflake Multi-Agent Team (Part 3)

In Part 1, we built an Admin Agent for usage and cost visibility.
In Part 2, we added a Cost Optimizer Agent and an Orchestrator that routes questions to specialists.

Now we close the loop with the third specialist: a Security and Governance Agent.

This turns your assistant from "what happened" and "what to optimize" into a full team that also answers "what is risky right now".

By the end of this post, you will have:

A Security and Governance Agent with focused security tools
Security semantic views mapped to natural language
Orchestrator routing across Admin, Cost Optimizer, and Security agents
A practical triage workflow for failed logins, privilege risk, and unauthorized access

Why Add a Security Specialist?

The first two agents are strong for operations and spend, but security requires a different lens:

Access control and role hygiene
Failed login patterns and anomaly detection
Unauthorized access attempts
Inactive users with active privileges
Compliance-friendly audit summaries

Could one large agent do everything? Sometimes. But specialized agents are easier to maintain, safer to evolve, and easier to test.

Final Team Architecture

User Question (natural language)
        |
  Orchestrator Agent
   /      |        \
Admin   Cost     Security
Agent  Optimizer Governance
                 Agent
        \    |    /
      Unified Response

Role of each specialist

Admin Agent: usage, credits, storage, operational metrics
Cost Optimizer Agent: idle compute, rightsizing, optimization opportunities
Security and Governance Agent: roles, privileges, failed logins, unauthorized access, audits

The Security Pattern (Same Foundation as Parts 1 and 2)

Step 1: Base Views

Create security-focused views over SNOWFLAKE.ACCOUNT_USAGE, including:

Role hierarchy and privilege grants
Failed login attempts and anomaly severity
Excessive or unused privileged access
Unauthorized access attempts
User and role audit summaries
Network policy activity

Implementation file:

sql/08_create_security_governance_views.sql

Step 2: Semantic Views

Map these views into natural language dimensions, facts, and metrics so Cortex Analyst can reason over them.

Examples:

SV_LOGIN_ANOMALIES
SV_EXCESSIVE_PRIVILEGES
SV_UNAUTHORIZED_ACCESS_ATTEMPTS
SV_USER_ROLE_AUDIT

Implementation file:

sql/09_create_security_governance_semantic_views.sql

Step 3: Create the Security Agent

Define a dedicated agent with explicit analyst tools for each security domain:

CREATE OR REPLACE AGENT <APP_DB>.<APP_SCHEMA>.<SECURITY_AGENT_NAME>
  COMMENT = 'Security and Governance agent for access control and compliance'
  FROM SPECIFICATION
  $$
  instructions:
    response: "You are a Security and Governance assistant..."
    orchestration: "Route role hierarchy questions to RoleHierarchyAnalyst; \
                   privilege grants to PrivilegeGrantAnalyst; \
                   failed logins to FailedLoginAnalyst; \
                   login anomalies to LoginAnomalyDetector..."

  tools:
    - tool_spec: { type: "cortex_analyst_text_to_sql", name: "RoleHierarchyAnalyst" }
    - tool_spec: { type: "cortex_analyst_text_to_sql", name: "PrivilegeGrantAnalyst" }
    - tool_spec: { type: "cortex_analyst_text_to_sql", name: "FailedLoginAnalyst" }
    - tool_spec: { type: "cortex_analyst_text_to_sql", name: "LoginAnomalyDetector" }
    - tool_spec: { type: "cortex_analyst_text_to_sql", name: "ExcessivePrivilegeAnalyst" }
    - tool_spec: { type: "cortex_analyst_text_to_sql", name: "UnauthorizedAccessAnalyst" }
    - tool_spec: { type: "cortex_analyst_text_to_sql", name: "UserAuditAnalyst" }
  $$;

Implementation file:

sql/11_create_security_governance_agent.sql

Step 4: Route Through the Orchestrator

Your orchestrator now includes SecurityAgent as a first-class route target.

-- in sql/12_create_orchestrator_agent.sql
-- Questions about security, roles, privileges, failed logins, unauthorized access -> SecurityAgent

It can also fan out to multiple agents for blended questions.

Example 1: Security-Only Routing

Question:
"Are there suspicious login failures in the last 7 days?"

What happens:

Orchestrator classifies this as security monitoring
Routes to SecurityAgent
SecurityAgent uses SV_LOGIN_ANOMALIES
Returns severity-based findings and recommendations

Example response:

Login Anomaly Summary (Last 7 Days)

Critical: 2 users with >= 10 failed attempts/hour
High: 5 users with 5-9 failed attempts/hour
Pattern: Multiple failed attempts from distinct IPs for USER_X

Recommendation:
- Lock and verify impacted accounts
- Enforce MFA re-registration for affected users
- Review network policy and source IP ranges

Example 2: Privilege Governance

Question:
"Which users have ACCOUNTADMIN or SECURITYADMIN but low recent usage?"

What happens:

Routed to SecurityAgent
Uses SV_EXCESSIVE_PRIVILEGES
Identifies high-risk assignments with low usage

Example response:

Excessive Privilege Findings

Users flagged: 4
Critical: 2 users with no privileged role usage in 60+ days
High: 2 users with < 5 privileged queries in 90 days

Recommendation:
- Revoke unused privileged grants
- Replace standing privilege with just-in-time elevation
- Document business justification for remaining elevated users

Example 3: Cross-Agent Investigation

Question:
"Why are costs up and is there any security risk around this?"

What happens:

Orchestrator identifies multi-domain intent
Routes to AdminAgent + CostOptimizerAgent + SecurityAgent
Aggregates usage, optimization, and security posture into one answer

Example response:

Integrated Account Assessment

Operations (Admin Agent)
- Compute credits up 18% month-over-month
- Growth concentrated in two ETL warehouses

Optimization (Cost Optimizer Agent)
- Idle percentage > 60% on one ETL warehouse
- Suggested AUTO_SUSPEND change could reduce waste materially

Security (Security Agent)
- One privileged user inactive but still assigned elevated role
- Increased failed login attempts from multiple IPs for two accounts

Priority Actions
1) Apply warehouse auto-suspend tuning
2) Review elevated role assignments and revoke unused grants
3) Investigate failed-login anomalies and tighten network controls

Deployment Sequence

If you already deployed Parts 1 and 2:

# 1) Security views
sql/08_create_security_governance_views.sql

# 2) Security semantic layer
sql/09_create_security_governance_semantic_views.sql

# 3) Security agent
sql/11_create_security_governance_agent.sql

# 4) Orchestrator (includes SecurityAgent routing)
sql/12_create_orchestrator_agent.sql

Testing Queries

Direct test:

SELECT SNOWFLAKE.CORTEX.DATA_AGENT_RUN(
  '<APP_DB>.<APP_SCHEMA>.<SECURITY_AGENT_NAME>',
  '{"messages":[{"role":"user","content":[{"type":"text","text":"Show failed login anomalies by severity"}]}]}'
);

Orchestrated test:

SELECT SNOWFLAKE.CORTEX.DATA_AGENT_RUN(
  '<APP_DB>.<APP_SCHEMA>.<ORCHESTRATOR_AGENT_NAME>',
  '{"messages":[{"role":"user","content":[{"type":"text","text":"Analyze account risk: costs, idle warehouses, and failed logins"}]}]}'
);

Practical Notes

SNOWFLAKE.ACCOUNT_USAGE views have latency. For near-real-time incident response, combine this with event-driven telemetry.
Keep role and privilege reviews on a recurring schedule (monthly or quarterly depending on policy).
Use placeholders and environment-specific grants consistently:
- <APP_DB>.<APP_SCHEMA>
- <EXEC_WAREHOUSE>
- <ADMIN_ROLE>, <DEVELOPER_ROLE>

What We Have Now

After Part 3, you have a complete multi-agent Snowflake administration team:

Admin Agent for operational visibility
Cost Optimizer Agent for efficiency and savings
Security and Governance Agent for risk and compliance
Orchestrator Agent for seamless, natural-language routing across all three

This is where multi-agent design pays off: each specialist stays focused, and users still ask one simple question.

Code References

Security base views: sql/08_create_security_governance_views.sql
Security semantic views: sql/09_create_security_governance_semantic_views.sql
Security agent: sql/11_create_security_governance_agent.sql
Orchestrator routing: sql/12_create_orchestrator_agent.sql
Security skill guidance: skills/security-governance/SKILL.md

Code Repository

Complete implementation:

Conclusion

Part 1 gave us visibility.
Part 2 gave us optimization.
Part 3 gives us governance and risk control.

Same architecture pattern, broader coverage:
Views -> Semantic Views -> Specialist Agent -> Orchestrator

Next directions:

Add automated alerting and ticket creation workflows
Add remediation playbooks per risk severity
Add environment-level policy checks for continuous compliance

Questions or feedback? Drop a comment below.

Part 1: Ask Your Snowflake Account Anything - Build an AI Admin Agent
Part 2: From One Agent to Many - Building a Multi-Agent Team
Part 3: Security and Governance Agent (this post)

Mirroring Snowflake Iceberg into Microsoft Fabric: The Gotchas - Part 1:

Krishna Tangudu — Thu, 11 Jun 2026 22:53:42 +0000

Previously: In Power BI Connectivity with Azure AD, I walked through enabling end-to-end SSO from Power BI to Snowflake.

In this post: A hands-on implementation log for Snowflake mirroring (including Iceberg tables) in Microsoft Fabric — what official docs confirm, what broke in production, and the runbook that now works. If you're seeing differences between Mirroring Status and SQL Analytics Endpoint, this should help.

Our goal was to serve analysts from Fabric's shared OneLake layer instead of having every downstream workflow repeatedly query Snowflake directly.

TL;DR

Fabric Mirroring for Snowflake continuously replicates source data into OneLake.
For Iceberg tables, Fabric replicates metadata to OneLake via shortcuts to the underlying storage and OneLake exposes data in Delta format for Fabric workloads.
SQL Analytics Endpoint is read-only and can show type/metadata limitations even when mirroring is healthy.
New or changed tables can take time to appear consistently in SQL Analytics Endpoint because replication and SQL metadata sync are separate steps.
Connection configuration accuracy matters (especially warehouse identity and permissions).
Not all Snowflake table types are eligible for mirroring.

What I validated against official Microsoft docs

1. How mirroring works for Snowflake in Fabric

From Microsoft documentation, mirroring creates:

A mirrored database item in Fabric
A SQL Analytics Endpoint for querying the mirrored data

For Snowflake sources specifically:

Iceberg table metadata is replicated into OneLake using shortcuts to the storage that contains those Iceberg tables.
OneLake automatically converts those Iceberg tables to Delta Lake formatted tables for Fabric workloads.
Managed table data is replicated into OneLake and converted to analytics-ready format.

This means consumption in Fabric is against replicated data in OneLake, not a live pass-through to Snowflake. When users say "SQL endpoint looks wrong," I now split diagnosis into replication-plane checks vs SQL metadata-plane checks.

2. SQL Analytics Endpoint is read-only and has data type/materialization limits

Microsoft documents that SQL Analytics Endpoint uses the same engine family and constraints as Fabric Warehouse for persisted/object materialized types. In practice this explains why:

Mirroring can be healthy
But some columns can be missing/truncated/not surfaced as expected in SQL Analytics Endpoint

This is especially important when source types do not map cleanly to endpoint-supported persisted types.

3. Snowflake mirroring limitations that matter in design

Official Snowflake mirroring limitations include:

Regular/native table replication focus (not all table categories are supported)
A current mirrored table-count limit per mirrored database
Schema-change replication nuances (some schema changes need data changes to trigger propagation)
Backoff behavior when source tables are idle

4. Cost and serving model

From Microsoft guidance:

Fabric query workloads (SQL, Spark, BI) consume Fabric capacity.
Snowflake compute can be incurred for data-change reads during replication.

This is not a one-time load model; mirroring is continuous synchronization. Total cost depends on change volume, replication activity, and Fabric-side query demand.

Why we still chose it: it reduced repeated direct-query pressure from many downstream users and gave us a shared serving layer across SQL, Spark, and BI.

Incident Log: What broke and what fixed it

Simple-looking errors often had multiple causes: connection identity, permissions, metadata timing, and endpoint type handling. The key discipline was separating what docs explicitly confirm from support heuristics.

Ticket #1–2: Connection and warehouse errors

Symptom

Mirroring refresh failed with warehouse-not-found / not-authorized style errors.

What we found

The configured warehouse identity in Fabric connection did not align with the warehouse defined/accessible in Snowflake for the mirroring principal.

Fix

Stop replication
Recreate the Snowflake connection in Fabric
Re-enter server/warehouse carefully
Ensure the principal has warehouse usage + required mirroring privileges
Rebind mirrored database to the corrected connection
Restart replication and monitor

Important precision

Microsoft docs explicitly tell you to normalize server entry format (for example remove protocol and use lowercase server host format in setup guidance). For warehouse value handling, my operational practice is to treat it as an exact identifier and validate both existence and authorization before restart.

Ticket #3: Mirroring successful, but SQL Analytics Endpoint shows column issues

Symptom

Mirroring status looked healthy, but querying via SQL Analytics Endpoint showed warnings or incomplete column exposure for some tables.

Root cause pattern

Endpoint-level SQL metadata and type mapping did not always represent source columns as expected.

Why this happens

This behavior is aligned with documented SQL Analytics Endpoint limitations and data type mapping constraints. Mirroring into OneLake can still be successful while SQL endpoint representation is partially constrained.

Mitigations

Validate data in OneLake directly (for example via Spark) when endpoint output looks suspicious
Cast unsupported/problematic source columns to endpoint-friendly types in curated tables/views
Use SQL endpoint refresh and metadata sync checks where relevant
For critical workloads, create curated serving layers specifically for endpoint compatibility

Architecture model that helped me

Think about Snowflake mirroring in Fabric as three planes:

Replication plane

Movement/sync of data and metadata into OneLake
Includes Iceberg metadata handling + managed table replication

Storage/open format plane

Data persisted in OneLake with Delta interoperability for Fabric engines

Serving plane

SQL Analytics Endpoint as read-only, T-SQL-friendly surface
Useful, but not equivalent to full source-type fidelity

Most confusion in operations comes from mixing plane #1 health with plane #3 usability.

My production runbook now

Before enabling mirroring:

Validate Snowflake privileges for mirroring principal (CREATE STREAM, SELECT, SHOW TABLES, DESCRIBE TABLES per Microsoft/Snowflake guidance)
Validate server entry format and connection parameters
Validate warehouse existence and effective permissions for principal
Identify which tables are regular/native and eligible
For Iceberg, confirm object store (e.g. S3) permissions up front: the mirroring principal needs read access to the underlying bucket where Iceberg data files are stored, not just Snowflake-level privileges

After enabling mirroring:

Monitor table-level replication status and last refresh
Distinguish OneLake landing success from SQL endpoint metadata availability
Add smoke tests in both Spark and SQL endpoint
Track Snowflake compute impact during replication windows

When troubleshooting:

If connection/auth errors: recreate connection cleanly, verify identifiers and grants
If missing columns in endpoint: check SQL endpoint limitations and data type mappings, then curate/cast
If schema changes not reflected: trigger/verify data change propagation and refresh metadata
If large-table updates are massive: consider stop/start strategy per documented performance guidance
If newly added tables don't appear in SQL endpoint: wait for initial replication to complete, validate data landed in OneLake/Spark first, then trigger endpoint metadata refresh — endpoint availability is near-real-time, not instant

Common misconceptions I had (now corrected)

Misconception 1: "Mirroring success means SQL endpoint is guaranteed complete"

Not always true. Replication and endpoint representation are related but not identical concerns.

Misconception 2: "All Snowflake table variants mirror the same way"

No. Supported categories are explicitly constrained; design with documented limitations first.

Misconception 3: "If it says queried live, Fabric is directly querying Snowflake"

Operationally, Fabric mirroring is a replication model into OneLake with Fabric-side query consumption.

Final takeaways

Fabric mirroring for Snowflake Iceberg works well when you separate replication health from SQL endpoint behavior.
SQL Analytics Endpoint issues do not always mean replication failure.
Connection hygiene (especially warehouse + auth correctness) prevents a surprising number of incidents.
For production, design a compatibility layer for endpoint consumption instead of relying on raw source type fidelity.
This was genuinely painful to implement end-to-end, but the pain forced a better operating model and a sharper troubleshooting discipline.

This is one of those integrations where knowing the documented constraints upfront saves far more time than reactive firefighting.

Neutrality and scope disclosure

This post reflects my hands-on implementation experience and support-ticket learnings.
It is not sponsored content and not intended as a blanket endorsement of any product.
Platform behavior, latency, and cost outcomes can vary by workload shape, table volume, schema patterns, and operating model.
The intent is practical: help others avoid common mistakes by combining official documentation with real incident patterns.

Sources (official)

Mirroring Snowflake in Microsoft Fabric: https://learn.microsoft.com/en-us/fabric/mirroring/snowflake
Tutorial: Configure mirrored Snowflake: https://learn.microsoft.com/en-us/fabric/mirroring/snowflake-tutorial
Snowflake mirroring limitations: https://learn.microsoft.com/en-us/fabric/mirroring/snowflake-limitations
Troubleshoot mirrored databases: https://learn.microsoft.com/en-us/fabric/mirroring/troubleshooting
Snowflake mirroring FAQ: https://learn.microsoft.com/en-us/fabric/mirroring/snowflake-mirroring-faq
SQL Analytics Endpoint limitations: https://learn.microsoft.com/en-us/fabric/data-engineering/lakehouse-sql-analytics-endpoint#limitations
Data types in Fabric Data Warehouse / SQL endpoint mapping: https://learn.microsoft.com/en-us/fabric/data-warehouse/data-types

Evaluating Adaptive Warehouses for ETL: Why We Reverted to Standard

Krishna Tangudu — Wed, 27 May 2026 14:13:16 +0000

In our last post we showed how upsizing a Bronze ETL warehouse from XSmall to Small reduced daily credits by 11%. This post examines a different question: does Snowflake's Adaptive warehouse model (preview) improve on what a well-tuned Standard warehouse already delivers?

docs.snowflake.com

For our Silver layer workload, the answer was no.

Background

WH_ETL_SILVER_01 is our dedicated warehouse for Silver layer data loads — long-running transformation queries at a steady, predictable rate of ~2,000 queries/day. On May 12, 2026 we migrated it from Standard to Adaptive to evaluate the new model under production conditions.

Parameter	Standard Config	Adaptive Config
Type	STANDARD	ADAPTIVE
Size	Small	Max: Small
Concurrency/Burst	MAX_CLUSTER_COUNT = 3	QUERY_THROUGHPUT_MULTIPLIER = 3
Auto-Suspend	60 seconds	Managed by Snowflake

After 16 days of production data, we reverted.

The Problem: Queuing That Wouldn't Stop Growing

Within the first week on adaptive, we started seeing queries queue. By week three, the queue rate had grown to 3.4% — and still climbing.

Period	Config	Queue Rate	Avg Wait (queued)	Max Wait
Apr 1 – May 11	Standard	~0%	N/A	—
May 12–15	Adaptive Week 1	0.1%	~10s	102s
May 16–22	Adaptive Week 2	2.5%	80–98s	490s
May 23–27	Adaptive Week 3	3.4%	65–80s	508s

Under Standard with the same ~2,000 queries/day load, queueing was effectively zero. Two queries queued on the worst day, for 100ms each.

Fair Comparison: Matched Load Windows

To avoid skewing results by different query volumes, we compared periods with equivalent daily throughput.

Standard baseline: May 1–11 (11 days, avg 2,151 queries/day)
Adaptive comparison: May 12–14, 18–22, 25–26 (10 days, avg 2,060 queries/day)

Query Performance

Metric	Standard	Adaptive	Change
Avg Elapsed Time	19.9s	22.8s	+15% slower
Median Elapsed Time	256ms	267ms	+4%
P95 Elapsed Time	14.1s	20.4s	+45% slower
Avg Execution Time	19.6s	21.0s	+7% slower

Cost

Metric	Standard	Adaptive	Change
Avg Daily Credits	15.2	19.1	+26% more expensive
Credits Per Query	0.0071	0.0093	+31% more expensive

Queuing

Metric	Standard	Adaptive	Change
Queue Rate	0.3%	2.2% (growing)	+7x worse
Avg Queue Wait	3.3ms	1,543ms	+46,000% worse

How We Measured

We used the same ACCOUNT_USAGE views from the previous post.

Query Performance (Matched Windows)

SELECT
  CASE
    WHEN start_time::DATE BETWEEN '2026-05-01' AND '2026-05-11' THEN 'Standard'
    ELSE 'Adaptive'
  END AS config,
  COUNT(*) AS total_queries,
  ROUND(AVG(total_elapsed_time) / 1000, 2)      AS avg_elapsed_sec,
  ROUND(MEDIAN(total_elapsed_time) / 1000, 2)   AS median_elapsed_sec,
  ROUND(PERCENTILE_CONT(0.95) WITHIN GROUP
    (ORDER BY total_elapsed_time) / 1000, 2)    AS p95_elapsed_sec,
  ROUND(AVG(queued_overload_time) / 1000, 2)    AS avg_queued_sec,
  ROUND(COUNT_IF(queued_overload_time > 0)
    / COUNT(*) * 100, 2)                        AS queue_rate_pct
FROM SNOWFLAKE.ACCOUNT_USAGE.QUERY_HISTORY
WHERE warehouse_name = 'WH_ETL_SILVER_01'
  AND start_time::DATE >= '2026-05-01'
GROUP BY config
ORDER BY config;

Credit Consumption

SELECT
  CASE
    WHEN start_time::DATE BETWEEN '2026-05-01' AND '2026-05-11' THEN 'Standard'
    ELSE 'Adaptive'
  END AS config,
  ROUND(SUM(credits_used_compute) / COUNT(DISTINCT start_time::DATE), 2)  AS avg_daily_credits,
  ROUND(SUM(credits_used_compute), 2)                                      AS total_credits
FROM SNOWFLAKE.ACCOUNT_USAGE.WAREHOUSE_METERING_HISTORY
WHERE warehouse_name = 'WH_ETL_SILVER_01'
  AND start_time::DATE >= '2026-05-01'
GROUP BY config
ORDER BY config;

Why Adaptive Doesn't Fit This Workload

Adaptive warehouses are designed for variable, unpredictable workloads — dashboards, ad-hoc analytics, mixed query sizes. Our Silver pipeline is the opposite:

Steady ~2,000 queries/day
Predictable concurrency (same pipeline, same schedule)
Long-running transformation queries — not short, bursty ones
Consistent data volumes day-to-day

The QUERY_THROUGHPUT_MULTIPLIER = 3 cap was insufficient for peak concurrency windows. The adaptive model's dynamic scaling overhead — which adds value when workloads are unpredictable — just introduced latency without benefit here. The Standard config's fixed multi-cluster model handled this workload profile cleanly.

Recommendation: Revert to Standard

ALTER WAREHOUSE WH_ETL_SILVER_01 SET
  WAREHOUSE_TYPE    = 'STANDARD',
  WAREHOUSE_SIZE    = 'SMALL',
  MAX_CLUSTER_COUNT = 3,
  SCALING_POLICY    = 'STANDARD',
  AUTO_SUSPEND      = 60;

Expected impact after reverting:

Eliminate queuing (proven 0% queue rate at this load level)
Save ~26% on daily credits
Reduce P95 latency by 45%
Restore predictable, consistent performance

If You Must Stay on Adaptive

Snowflake's official guidance is clear: increase QUERY_THROUGHPUT_MULTIPLIER to reduce queuing.

ALTER WAREHOUSE WH_ETL_SILVER_01 SET QUERY_THROUGHPUT_MULTIPLIER = 6;

This would likely resolve the queuing issue. However, for a predictable, steady-state ETL workload, this approach adds operational complexity without strategic benefit:

You're now tuning a multiplier instead of configuring a fixed cluster count
Higher throughput capacity may increase peak concurrent spend, requiring new cost monitoring
For a pipeline that runs the same way every day, the adaptive model's flexibility adds overhead without value

Bottom line: Adaptive's tuning knobs are powerful for variable workloads. For predictable ETL, Standard's simplicity wins operationally — and financially.

Key Takeaways

Adaptive is not universally better — it adds overhead that only pays off for variable, unpredictable workloads.
Always compare at matched load — raw averages across different query volumes are misleading.
Queuing is the canary — a worsening queue rate on adaptive signals a workload mismatch, not just an under-tuned multiplier.
Standard multi-cluster remains the better fit for batch/ETL pipelines with consistent concurrency.
Reserve adaptive for variable workloads — ad-hoc analytics or environments where query volume swings 5–10x daily.

Closing Thought

The lesson from post was: in some workloads, bigger can be cheaper. The lesson here is the inverse: newer is not always better. The right warehouse type depends on workload shape — steady pipelines and dynamic workloads have fundamentally different resource patterns, and Snowflake's warehouse models reflect that.

Disclosure and Scope

Results reflect our specific environment and workload profile; outcomes may vary.
Analysis period: May 1–27, 2026 | Warehouse: WH_ETL_SILVER_01 | Workload: Silver layer production loads

This analysis used Snowflake built-in ACCOUNT_USAGE views only. No third-party monitoring stack required.