Previously: In Mirroring Snowflake Iceberg into Microsoft Fabric: The Gotchas, I documented connection issues, permissions, and the difference between healthy mirroring and a usable SQL analytics endpoint.
The goal remains the same: give analysts a shared serving layer in Fabric without making every downstream workflow query Snowflake independently.
Since that post, two support investigations have added to the picture. One concerns the cost of continuous mirroring when the source changes only a few times daily. The other reverses an earlier troubleshooting pattern: the SQL analytics endpoint reads the data, but Spark fails.
This is the checklist of what we have learned, which workarounds we have used, and the options we plan to evaluate next. Checked items describe experience or completed investigation. Unchecked items are future tests, not claimed results.
1. What we have done so far
- [x] Established the Snowflake–Fabric integration. The first post covers the connection and permission fixes, including separate access to Iceberg storage.
- [x] Separated replication health from query behavior. We validate actual results through the consuming engine, rather than treating a healthy status as sufficient.
- [x] Compared SQL analytics endpoint and Spark behavior on the decimal issue. SQL succeeded where the affected Spark read failed.
- [x] Validated the source, table, and file schemas. We submitted those findings to Microsoft support.
- [x] Tested the non-vectorized Spark workaround. Reads succeeded; we subsequently reported performance and capacity impact.
- [x] Investigated mirroring schedules, restart behavior, and Delta CDF. Support confirmed the limitations described below.
- [ ] Measure whether Snowflake AWS-to-Azure replication improves the total economics. This is our next architecture experiment.
2. Continuous mirroring: the business clock matters
Our source data changes approximately three times per day. The mirroring support case described a Snowflake cloud Layer cost spike and asked a practical question: can we run replication around those updates instead of continuously?
At roughly 2,500 tables across the estate, maintaining an individual pipeline for every table would also be a substantial burden. That concern is about operational scale; it does not mean every batch design requires a separate pipeline per table.
What the investigation established:
- [x] Scheduling: Microsoft documents no configurable mirroring schedules or replication windows today. Snowflake mirroring guidance
- [x] Stop/start behavior: support confirmed a full reload rather than continuation from the previous CDC position, consistent with the public FAQ. A timer around stop/start is therefore not equivalent to a scheduled incremental load. Snowflake mirroring FAQ
- [x] Disabling the service user: support explained that this is not a supported scheduling mechanism; connection retries and recovery become part of the problem.
- [x] Alternative suggested: support recommended evaluating a Copy activity or Copy job, while acknowledging that this requires architectural work.
The case is closed, but the correspondence acknowledges a scheduling limitation. It does not record delivery of a scheduling fix.
3. The decimal issue: SQL worked, Spark did not
The second case captured this error, with the column anonymized:
Parquet column cannot be converted ...
Column: [AMOUNT], Expected: decimal(15,2), Found: INT32
Our diagnostics showed:
| Layer | Reported representation |
|---|---|
| Snowflake source column | NUMBER(15,2) |
| Fabric table schema | decimal(15,2) |
| Examined Parquet file | decimal(9,2), backed by INT32 |
| SQL analytics endpoint | Query succeeded |
| Affected Spark read | Failed with vectorization enabled |
| Spark with vectorization disabled | Read succeeded |
The logical table schema and a file's physical encoding are different layers. The failing read path did not successfully reconcile the narrower file representation with the declared decimal type. The observed result was a read failure, not evidence of corrupted business values.
Microsoft's public guidance lists the type-width issue and the Spark workaround. OneLake Iceberg limitations
- [x] Workaround used: disable the vectorized Parquet reader in the affected Spark session.
- [x] Impact reported: performance and capacity concerns
Here is a scoped illustration of the workaround, using fictional table and column names:
# Fabric PySpark: use a small, known slice of an affected table.
setting = "spark.sql.parquet.enableVectorizedReader"
original = spark.conf.get(setting)
try:
spark.conf.set(setting, "false")
rows = spark.sql("""
SELECT order_id, amount
FROM dbo.orders_iceberg
WHERE order_id IN (101, 102, 103)
""").collect()
print(rows)
finally:
spark.conf.set(setting, original)
The action runs before the setting is restored. This example illustrates the configuration used; it is not a new benchmark or a reproduction executed for this article. A simple query showed a performance difference from 20 seconds to 2 minutes 26 seconds.
The lesson extends my previous post: SQL and Spark both need independent acceptance tests. Either engine can expose a limitation that the other does not.
4. Options on our checklist
| Option | Our status | Why it is relevant | What still needs proof |
|---|---|---|---|
| Existing Snowflake mirroring / Iceberg integration | Used; support cases investigated | Shared Fabric serving layer | Sustainable cost and consistent engine behavior |
| SQL analytics endpoint for affected reads | Worked in the reported case | Keeps compatible SQL workloads usable | Coverage of workloads that currently require Spark |
| Spark non-vectorized reader | Tested workaround | Restores affected reads | Performance and capacity at production scale |
| Scheduled Copy job or reusable Copy pipeline | Official option; not benchmarked here | Direct control over movement frequency | CDC eligibility, deletes, schema changes, cost, and management at scale |
| Snowflake AWS → Snowflake Azure replication | Planned next experiment | Place a controlled replica nearer Azure consumers | Eligible objects, consumer access, freshness, and total cost |
| Direct Snowflake access | Tested workaround | Keeps users productive | Source load and total BI/query cost |
5. What comes next: Snowflake AWS → Snowflake Azure
Our hypothesis: a scheduled Azure replica may be cheaper to operate when it replaces substantial repeated reads across the cloud boundary. We have not established savings yet.
The candidate flow is:
Snowflake AWS source
|
| scheduled Snowflake replication
v
Snowflake Azure secondary + target-region storage where required
|
| consumer path to validate separately
v
Fabric Iceberg access
Snowflake officially supports replication across AWS and Azure within an organization. Replication groups provide read-only secondaries and configurable refresh schedules. Database replication is available across editions; failover groups and various account-object capabilities have additional edition requirements. Replication overview, Replication schedules
We are evaluating replication for read access. Failover is a separate operating requirement and should not be introduced merely to make a read-only secondary writable for an integration.
6. The cost test that will decide the direction
Snowflake replication adds transfer and service-compute charges, billed to the target account, plus target storage costs. Refresh frequency and changed data volume affect the total. The initial seed must be measured separately from steady-state refreshes. Snowflake replication cost
Our comparison will be:
Current path:
source integration activity + transfer/storage requests
+ Fabric serving cost + operating effort
Azure replica path:
replication transfer + replication compute + target storage
+ Azure-side integration/query cost + Fabric serving cost
+ operating effort
These are measurement categories, not a savings estimate. For external Iceberg storage, include the applicable cloud-storage bill as well as Snowflake and Fabric usage.
My current direction is to test Azure replication. The former tests whether data location is driving unnecessary cost; the latter tests whether movement frequency is the larger problem. The decimal issue remains its own compatibility test.
That is what I want the next installment to report: which option we tested, what improved, what it cost, and what still required a workaround.
Top comments (0)