DEV Community: Shai Karmani

The Hidden Visual Pattern That Gives Power BI Copilot Better Context

Shai Karmani — Sun, 13 Sep 2026 23:44:30 +0000

Originally published at https://shai-kr.github.io/data-ninja-ai-lab/blog/2026-09-13-power-bi-copilot-hidden-visuals.html.

A Power BI report can contain useful analytical context that is not visible when the page first opens.

Teams already use bookmarks to reveal detail panels, alternate charts, and guided explanations without crowding the default canvas. The August 2026 Power BI update makes that pattern more relevant to Copilot: report summaries and answers can now consider visuals that are hidden by default and revealed through display-only report bookmarks.

This creates a useful design opportunity. A report author can keep the main page focused while making additional analytical views reachable to both the user and Copilot.

The important word is intentional. Hidden visuals should not become a place to store unexplained measures or abandoned chart experiments. They are part of the report's analytical contract now, and they need the same naming, permissions, validation, and ownership as visible visuals.

What changed

Microsoft documents two related experiences:

The Copilot report pane can summarize visuals across the report, including visuals hidden by default behind a display-only bookmark. It reads those visuals in place without changing the current bookmark state. RLS and OLS remain enforced.
The Copilot narrative visual lets an author turn on Show hidden bookmark visuals, review the eligible hidden visuals, and select which ones should contribute to the narrative.

The boundary matters. Microsoft limits this behavior to display-only report bookmarks, where the bookmark changes visibility but does not save data state. The bookmark must also be reachable through a bookmark button or bookmark navigator. Personal bookmarks are not supported. Visuals on hidden pages are excluded unless a report bookmark makes them visible.

This is more than a small Copilot enhancement. It changes how I would review the report canvas.

Treat hidden visuals as a second analytical surface

A clean executive page often needs fewer visuals, not more. That does not mean every useful analytical view must disappear.

A display-only bookmark can reveal a focused detail panel when a user asks for it. Examples include:

a regional breakdown behind a headline KPI;
a variance bridge behind a monthly result;
a product mix chart behind a total margin card;
a service-level detail view behind an operations summary;
a definition panel that explains how the KPI is calculated.

The default page remains readable. The extra context remains part of the report experience, and Copilot can use it when the documented conditions are met.

This is a better design pattern than placing twelve visuals on the opening page because each stakeholder may need one of them. It also avoids creating disconnected hidden pages that users cannot reach and authors forget to maintain.

But there is a catch: Copilot works from the report visuals and their metadata. A hidden chart called Visual 27 with a vague axis and an unexplained measure is poor context. Hiding it does not improve its meaning.

Use an authoring contract

I would add five rules to the report review checklist.

1. Make the bookmark display-only

Clear Data for the bookmark so it changes visibility rather than storing filter or slicer state. This keeps the hidden visual eligible under Microsoft's documented behavior and avoids coupling the context panel to an old saved selection.

2. Give users a real path to the visual

Use a bookmark button or bookmark navigator. A hidden visual that nobody can reach is not part of a usable report experience. Microsoft also excludes bookmarks without a user-facing affordance from this Copilot pattern.

3. Name the analytical view clearly

Use a title that describes the measure, comparison, and grain. Margin variance by product category gives Copilot and the reader more context than Variance chart.

Axis titles, units, date scope, and business terminology should be explicit. The hidden state is a layout choice, not permission to lower the metadata standard.

4. Keep one business question per hidden panel

A bookmark panel should answer a recognizable follow-up question. If it contains unrelated charts, Copilot may have more material but less coherent context.

A practical test is simple: can the author complete this sentence?

Open this panel when you need to understand ________.

If the answer needs three paragraphs, the panel probably needs to be split.

5. Review security and freshness like any visible visual

RLS and OLS remain enforced, which is the right security behavior. Still, test with representative roles. Also verify that the measure logic, visual filters, and refresh expectations are current. Hidden visuals are easy to miss during ordinary visual inspection.

Test the experience, not the checkbox

Do not validate this feature by confirming that the bookmark opens and Copilot returns text. Use a small acceptance test with a known result.

Pick one report page and one hidden analytical panel. Create a question that can be answered from the visible page, then a second question that requires the hidden visual. Record the expected values before running Copilot.

Run at least these four cases:

Case	Report state	What to verify
Visible baseline	Default page	Summary cites and reconciles with visible totals
Hidden context	Default page, hidden panel reachable	Answer can reference the eligible hidden visual without changing bookmark state
Filtered context	Apply a supported filter or slicer	Answer respects the current report context and cites the expected visual
Restricted role	Test as a representative RLS or OLS role	Response does not reveal data or metadata outside that role

For the narrative visual, also verify the author's selection. Turn on Show hidden bookmark visuals, include only the intended views, generate the narrative, and inspect the footnotes. Microsoft advises authors to read the generated summary for accuracy. That remains a required step, not a formality.

Keep the scope precise

This feature does not mean Copilot reads every hidden artifact in a report.

The documented pattern is narrower:

display-only report bookmarks;
a user-facing bookmark button or navigator;
supported visuals;
report pages and hidden visuals that meet the eligibility rules;
the current user's permissions, including RLS and OLS.

The Copilot report pane and the Copilot narrative visual also expose different controls. The report pane can use eligible hidden visuals in summaries and answers. The narrative author explicitly chooses hidden bookmark visuals through the selection experience.

That distinction belongs in documentation and QA. Otherwise, teams will assume a hidden visual is included because it exists, or assume it is excluded because the page starts with it hidden.

The practical payoff

This pattern lets report teams separate attention from availability.

The opening page can emphasize the few numbers that deserve immediate attention. Bookmark panels can preserve useful drill-in context. Copilot can work with those eligible views without forcing every chart onto the default canvas.

That is a better report design outcome than adding more visual density for the sake of AI. It also gives teams a concrete review artifact: a list of hidden context panels, the business question each one answers, and the four test results that prove the experience works for the intended users.

I would start with one report, one bookmark panel, and one known-answer test. If the pattern improves both human navigation and Copilot's cited answers, then standardize it.

Sources

Shai Karmani

Microsoft Data and AI practitioner

Connect with me on LinkedIn

Power BI Maps Can Finally Follow the Analysis

Shai Karmani — Fri, 11 Sep 2026 22:44:36 +0000

Originally published at https://shai-kr.github.io/data-ninja-ai-lab/blog/2026-09-11-power-bi-azure-maps-filter-context.html.

A map is useful when it stays aligned with the question the report user is asking.

That sounds obvious, but large geographic datasets have made it difficult. The Azure Maps visual in Power BI has a 30,000 data point rendering limit. Microsoft says that points outside the initial rendered set could previously remain absent even after a user filtered the report to a smaller area.

The August 2026 Power BI update changes that behavior. When the report is filtered to a smaller selection, the visual can reload the newly relevant points and automatically zoom to the visible area. Microsoft lists both improvements as generally available.

This is a small feature with a useful design consequence: a map can now follow the active analysis instead of behaving like a fixed overview.

My recommendation is to treat this as a report interaction change, not a formatting change. Test the filter path, the result count, and the viewport together.

What changed

Two behaviors matter.

First, the visual reloads against the filtered selection. If the initial broad view reaches the 30,000 point rendering limit, filtering to a region can bring in points that were relevant to that region but were not included in the initial rendered set.

Second, the map automatically zooms to fit the currently visible data. Filters, slicers, and cross-filtering from another visual can move the viewport to the selected area without making the user pan and zoom manually.

The important phrase is filtered selection. This does not remove the 30,000 point rendering limit, and it does not turn the map into an unlimited spatial query engine. It makes the visual recalculate which points matter for the current context.

Design the map as part of the report interaction

A map should answer a geographic question that the rest of the page helps define.

Consider a hypothetical service-operations report with thousands of work orders across North America. The page starts with a broad map, a date slicer, a service-category slicer, and a bar chart by region.

A user selects Ontario in the bar chart and chooses the last seven days. The expected experience is now testable:

The map receives the same filter context as the supporting visuals.
The relevant Ontario points load for that selection.
The viewport fits the visible data.
The totals in the map and supporting visuals can be reconciled.

The map is no longer an isolated picture. It is one step in the report's interaction contract.

That contract should be explicit. Decide which visuals filter the map, whether a selection should highlight or filter, and what happens when the user clears the selection. If automatic zoom is helpful for a regional drill, it may be distracting on a page where users need a stable national frame of reference.

Test data completeness separately from visual usefulness

A map that zooms correctly can still communicate the wrong thing.

Start with a known test slice. Use a region and date range where you can calculate the expected row or location count outside the visual. Then compare that result with the mapped selection.

Keep these questions separate:

Did the report apply the intended filter context?
Did the visual load the relevant points for that context?
Did the viewport fit the visible result?
Can the user still understand where they are?

The release changes rendering behavior, not data quality. Duplicate coordinates, missing latitude or longitude, ambiguous geocoding, and incorrect category fields still need their own checks.

If multiple rows share a coordinate, define what a point represents. Is it one work order, one customer site, or an aggregated location? The answer belongs in the model and report design, not in the user's guesswork.

Use a four-case acceptance test

I would validate the update with four repeatable cases before changing a production report.

1. Broad view

Open the report with its normal default filters. Record the visible extent, the supporting total, and whether the dataset is large enough to encounter the documented rendering limit.

2. Slicer filter

Choose one region or business segment from a slicer. Confirm that the map reloads the relevant selection and fits the selected geography.

3. Cross-filter

Select a category in another visual. Confirm that the map receives the intended interaction and that the displayed geography matches the filtered totals.

4. Clear and reset

Clear the selection or use the report's reset path. Confirm that the map returns to the expected default context. A smooth drill-in is only half the experience. Users also need a predictable way back.

Be careful with the 30,000 point boundary

Microsoft documents a 30,000 data point rendering limit for this behavior. The new filtered reload helps the visual represent a smaller selected area more completely than relying on the original broad-view set.

It does not mean that every filtered state will remain below the limit. A large city, dense device network, or long time window can still produce more points than the visual should render at once.

Use the report design to narrow the question. A required date range, region selector, category filter, aggregation, or drill path can be more useful than placing the entire operational estate on one map.

This is also where a supporting KPI or table earns its place. Give users a count they can reconcile with the map, especially when the map is intended for operational decisions rather than a high-level presentation.

Check the mobile experience

Automatic zoom reduces manual navigation, but it does not guarantee a readable phone layout.

Test the actual mobile report layout. Check whether the legend, zoom controls, tooltips, and neighboring visuals leave enough room for the selected geography. A narrow visual can technically fit the data while making individual points impossible to inspect.

If the map is central to the workflow, consider a focused mobile page rather than squeezing the desktop page into portrait orientation. Keep the filter path short and make the reset action visible.

The practical takeaway

This update makes Azure Maps more responsive to report context. The strongest use case is not a prettier map. It is a report where geographic detail arrives when the user narrows the question.

Build one controlled test page. Start broad, filter to a known subset, cross-filter from another visual, and reset. Compare the map against a known count at every step.

When the loaded selection, viewport, and supporting totals move together, the map becomes a reliable part of the analysis instead of a separate visual users have to manage.

Sources

Power BI August 2026 Feature Summary. Primary announcement for filtered-selection reload and automatic zoom in Azure Maps. The complete announcement was also verified through the official Power BI Updates RSS feed.
Azure Maps visual for Power BI. Product documentation for the Azure Maps visual, its configuration, and report interactions.
Change how visuals interact in a Power BI report. Supporting documentation for defining filter and highlight interactions between report visuals.
Create a mobile layout in Power BI. Supporting documentation for mobile-optimized report design.

Shai Karmani

Practical data engineering, Microsoft Fabric, Power BI, and AI.

Connect with me on LinkedIn.

Give Your Fabric AI Agent a Real Dependency Map

Shai Karmani — Fri, 11 Sep 2026 17:23:35 +0000

Originally published at https://shai-kr.github.io/data-ninja-ai-lab/blog/2026-09-10-fabric-lineage-review.html.

An AI agent can explain a proposed data-platform change much better when it can see what depends on the item being changed.

That is the useful opportunity in Microsoft's September 10 announcement of the Fabric item relations API, in preview. Two REST operations expose upstream and downstream item relationships as a graph that code can read. The response includes items, typed relations, and the workspaces referenced by those items.

My first use for this would be a read-only change brief. Pick an item, retrieve its relationships, and give the reviewer an evidence-backed summary of the dependencies to inspect before approving a change.

This is a proposed workflow, not a report of a production implementation or measured results. Microsoft explicitly describes the preview as intended for evaluation and development, and not recommended for production use. That boundary matters: build a small evaluation, not an automatic production deployment gate.

Start with a decision, not a chatbot

Consider a hypothetical team planning to replace a semantic model. The useful question is: which related items and teams should participate in the review?

The portal's lineage view already helps a person explore those dependencies. The API makes that metadata available to another tool. A script can turn the response into a change attachment. An agent can summarize it. A reviewer can inspect the original evidence without retracing every relationship manually.

I would keep the first version deliberately narrow: one familiar model, a read-only identity, one downstream request, and a short review brief. Once that output is understandable, add an upstream request to explain the model's source context.

The two directions answer different questions. Downstream helps identify consumers to investigate. Upstream helps explain dependencies that feed the item. Neither direction, by itself, proves that a proposed change is compatible.

What the preview actually exposes

The documented operations have these paths:

GET /v1/workspaces/{workspaceId}/items/{itemId}/relations/downstream?beta=true
GET /v1/workspaces/{workspaceId}/items/{itemId}/relations/upstream?beta=true

These are request shapes, not a complete executable authentication example. Use an approved identity and your normal protected authentication mechanism. Do not paste access tokens into a notebook, an agent prompt, or a review document.

The announcement says the caller needs read permission on the item and that both user and service principal identities are supported. The preview currently requires the beta query parameter. Check the linked API reference as the contract evolves.

Each response has three useful collections:

items: item identifiers, types, display names, and workspace identifiers. The queried item is included too.
relations: the edges connecting items, with a relation type.
workspaces: workspace context for the items returned in the graph.

Keep identifiers in the review evidence. Display names help a human read the result, but matching on names alone makes a poor join strategy. Two items can have similar names, and a friendly name is not a durable identity.

Preserve what the edges mean

A list of item names is less useful than a typed dependency map.

In Microsoft's association example, a report consumes a semantic model. The edge has the report as itemId and the model as dependentOnItemId. Read that as “the report depends on the model.” Keep that orientation intact when generating the explanation.

Other relationship types describe different behavior. A Shortcut references data through a shortcut. Orchestration describes an item running or managing another item. PushData describes writing or pushing data to a dependency item. Those are different reasons for bringing an owner into a change review.

The documented set of relationship types is extensible. My parser would preserve an unfamiliar type and label it for inspection rather than discard it. An unexpected relationship is useful evidence; silently dropping it makes the brief look more complete than it is.

The same principle applies to scope. Describe this as an item dependency map. Do not present it as column-level lineage, a measure dependency analyzer, or a complete inventory of every external consumer. Those are stronger claims than the item graph establishes.

Give the agent a bounded evidence package

The agent should summarize a structured record, not improvise a map from item names in the prompt.

For a prototype, I would store the target item ID, workspace ID, request direction, retrieval time, calling identity context, raw response, and any request errors. The summary can then reference the same item IDs as the evidence.

Treat names and metadata as data. An item description containing instructions is not permission for the agent to run a command, change access, or follow a link. The agent's job here is to explain the graph within the review task.

A useful output has a short answer to each of these questions:

What item are we reviewing, and when was this graph retrieved?
Which returned items depend on it, and through which relation types?
Which upstream dependencies help explain its context?
Which workspace owners or item owners still need to be identified?
What remains unverified before this change can be approved?

Owner assignment is a separate enrichment step. Do not imply the three response collections automatically provide an accountable business owner for every relationship.

Use a small acceptance test

Choose an item whose relationships your team already understands. Compare the returned graph with the portal view under the same identity context. Investigate discrepancies rather than asking the agent to explain them away.

Check that an association is described in the right direction. Confirm the queried item is not counted as one of its own consumers. Check that items from different workspaces retain their workspace context. Feed the summarizer a synthetic unfamiliar relation type and verify that it remains visible.

Then test failure behavior. If a request fails, the brief should say that evidence could not be retrieved. If no downstream items are returned, it should say exactly that. “No downstream items returned” is not the same conclusion as “safe to delete.”

Finally, review whether the brief is actually useful. Can a reviewer find the underlying item identifiers? Can they separate an observed relationship from a proposed action? Can they see what has not been checked? Those are better prototype success criteria than how confident the generated paragraph sounds.

Keep approval separate from explanation

A dependency tells you where to look. Compatibility testing tells you whether the proposed change works.

Changing a model may require report testing, refresh checks, security validation, and business sign-off even when the dependency graph is accurate. The item relations API adds useful context to that process. It does not replace it.

During this preview, I would use the brief as an evaluation artifact alongside an existing review process. No automatic deletion. No permission changes. No claim that an agent has certified a change as safe.

The practical win is straightforward: give the reviewer a grounded starting point, with the evidence attached. An agent that can explain what it knows, where it came from, and what still needs checking is far more useful than one that simply says “looks good.”

Sources and preview status

Verified September 10, 2026. The announcement was readable in the official Fabric RSS feed; the Community article URL returned an access restriction during this scan. Both API reference pages were accessible.

Shai Karmani writes about practical Microsoft Fabric, Power BI, data engineering, and AI implementation. Connect with me on LinkedIn.

Let Your Fabric Gateway Scale With the Workload

Shai Karmani — Wed, 05 Aug 2026 23:51:14 +0000

Originally published at https://shai-kr.github.io/data-ninja-ai-lab/blog/2026-08-05-vnet-gateway-autoscaling.html.

Private data access is one of those platform details that only becomes interesting when it fails.

A semantic model misses its refresh window. A Dataflow Gen2 job slows down. A pipeline that usually runs fine suddenly competes with every other workload in the same evening batch window. The business sees a report that is late. The platform team sees a connectivity layer that was treated as plumbing until it became the bottleneck.

That is why the new VNet data gateway autoscaling preview in Microsoft Fabric is worth paying attention to.

The feature itself is straightforward: Microsoft describes autoscaling as the ability to automatically adjust the number of active gateway nodes within admin-defined minimum and maximum limits based on workload demand. In plain English, the gateway can absorb spikes and scale back down when demand falls.

Useful feature. But the real opportunity is bigger than adding more gateway nodes.

It is a chance to treat private data access as an operating model.

Why this matters in Fabric and Power BI

The VNet data gateway sits in a sensitive part of the architecture. Microsoft documents it as a managed way to connect Microsoft Fabric and Power Platform services to Azure and other data services inside a virtual network, without using an on-premises data gateway. It can support Fabric Dataflow Gen2, Fabric data pipelines, Copy Job, Mirroring, Power BI semantic models, and Power BI paginated reports.

That means one gateway can sit behind a lot of important work.

A platform team might use it for:

Power BI semantic model refreshes against private Azure SQL or storage;
Dataflow Gen2 ingestion into a Lakehouse;
Fabric pipelines and Copy Job activity;
Mirroring patterns;
paginated reports that still depend on secured source systems.

When demand grows, the gateway becomes more than a network object. It becomes shared platform capacity.

Autoscaling helps with that. But it should not be enabled as a blind performance switch.

The wrong way to think about autoscaling

The weak version of the story is simple:

"Refresh is slow, turn on autoscaling."

That misses the point.

Autoscaling can reduce pressure during spikes, but it will not fix every bad design around the gateway. It will not clean up duplicate refresh schedules. It will not decide which workloads deserve priority. It will not turn an overloaded semantic model into a clean one. It will not make weak ownership disappear.

If the gateway has become a mystery box, scaling it may only make the mystery box more expensive.

The better approach is to use autoscaling as a forcing function for platform review.

The review I would run first

Before changing gateway scale settings, I would build a simple inventory.

Not a six-month governance program. Just enough information to make the decision responsibly.

1. Map the workloads

List every important workload using the gateway:

semantic models;
Dataflow Gen2 items;
Fabric pipelines;
Copy Jobs;
Mirroring flows;
paginated reports.

For each one, capture the owner, business criticality, schedule, expected runtime, and source system.

This is not busywork. Without it, you cannot tell the difference between a real scaling need and a messy batch window.

2. Baseline the demand pattern

Autoscaling is useful when demand changes. So measure demand first.

Look for:

refresh windows where many semantic models start together;
pipelines that overlap with executive reporting deadlines;
repeated gateway-related failures;
latency spikes that match predictable business cycles;
low-usage periods where scaled-down capacity would make sense.

The goal is not perfect telemetry. The goal is enough evidence to avoid guessing.

3. Define the scale policy

Autoscaling still needs boundaries.

At minimum, I would document:

minimum gateway node count;
maximum gateway node count;
who owns the setting;
what cost guardrails apply;
when the maximum should be revisited;
what incident pattern justifies a temporary change.

This is where the feature becomes an admin policy instead of a one-time UI change.

4. Clean the workload before scaling the gateway

If ten semantic models refresh at the same minute because nobody reviewed the schedule, the gateway is not the root problem.

Fix the obvious scheduling and ownership issues first:

stagger refreshes;
remove abandoned reports and models;
separate critical workloads from noisy ones where possible;
check whether a dataflow, warehouse, or semantic model is doing work in the wrong layer;
make owners accountable for refresh design.

Then use autoscaling for the demand that remains.

A better mental model

I would frame the gateway like this:

The VNet data gateway is not only the path to private data. It is the shared access layer between Fabric workloads and protected systems.

That access layer needs capacity, policy, monitoring, and ownership.

Autoscaling makes that layer more elastic. Good. But elastic does not mean unmanaged.

A healthy setup should be able to answer four questions:

Which workloads depend on this gateway?
What is normal demand?
What happens during a spike?
Who changes the limits when the pattern changes?

If those answers are clear, autoscaling becomes a useful production feature.

If those answers are vague, autoscaling becomes another place where platform cost and reliability drift quietly.

Where this fits in the Fabric architecture story

A lot of Fabric updates are pointing in the same direction: more managed platform capabilities, more API surface, more private connectivity, more operational responsibility inside the analytics estate.

That is the right direction.

But every managed capability still needs an owner.

For BI and data teams, the practical takeaway is simple:

Do not wait until the gateway becomes a refresh incident before you treat it as part of the platform.

Use autoscaling as the trigger to build a cleaner gateway operating model:

inventory the workloads;
baseline the peaks;
set guardrails;
review the batch window;
document the runbook.

That is not glamorous. It is the kind of work that makes Fabric and Power BI estates reliable enough for real business use.

Sources

Written by Shai Karmani. If you work with Microsoft Fabric, Power BI, data platforms, or practical AI systems, connect with me on LinkedIn.

Make Power BI Semantic Model Settings Easier to Govern

Shai Karmani — Sat, 01 Aug 2026 23:04:37 +0000

Originally published at Data Ninja AI Lab.

Power BI semantic model settings are becoming less of a separate admin page and more of an in-context workflow.

That is a good change.

In the July 2026 Power BI update, Microsoft says the semantic model settings pane becomes the default starting in August. Instead of jumping away to the full settings page, admins and model owners can open settings beside the content they are working with and update refresh schedules, credentials, and other model options with less context switching.

On paper, that sounds like a UI improvement.

In practice, it is a governance opportunity.

A lot of Power BI estates do not fail because teams lack features. They fail because semantic model administration is scattered across too many people, too many workspaces, and too many undocumented decisions. Refresh settings are copied from old models. Credentials belong to whoever built the first version. RLS roles are rarely reviewed after release. Sensitivity labels, endorsements, descriptions, and workspace access drift quietly.

A better settings pane will not fix that by itself.

But it can make the right checks easier to run.

Why this matters now

Semantic models are no longer just datasets behind reports.

They are shared business assets. They feed reports, apps, Excel, embedded experiences, Copilot experiences, and sometimes downstream automation. When the model is wrong, stale, insecure, or poorly described, the problem spreads quickly.

The new settings pane matters because it brings administration closer to the model owner workflow.

That changes the question from:

Where do I find the old settings page?

To:

What should we verify every time we touch a critical model?

That second question is where senior BI teams can get value.

The mistake to avoid

The easy reaction is to treat the new pane as a convenience feature.

Open model. Change setting. Move on.

That is fine for small personal reports. It is not enough for shared semantic models that executives, finance teams, operations teams, or customer-facing systems depend on.

For production models, the pane should become part of a short review workflow.

Not a huge governance ceremony. Just enough discipline to prevent the common failures.

The review I would run first

If I inherited a Power BI workspace today, I would use the settings pane as the entry point for a 30 minute semantic model review.

I would not start with a beautiful architecture diagram.

I would start with four checks.

1. Who owns this model?

Every important semantic model needs two owners.

One business owner who knows whether the numbers are trusted.

One technical owner who knows how the model refreshes, where the data comes from, how measures are maintained, and what breaks when a source changes.

If the answer is "the person who built it three years ago", that model is already carrying operational risk.

The settings pane cannot assign accountability by itself, but it gives the team a practical place to start the conversation.

For each important model, capture:

business owner;
technical owner;
workspace;
source systems;
refresh schedule;
audience or org app dependency;
criticality;
support contact.

A spreadsheet is enough for the first pass.

2. Does the refresh schedule match the business need?

Refresh settings are often inherited rather than designed.

A daily finance model, a near-real-time operations model, and a low-usage management report should not all follow the same pattern just because someone copied a previous dataset.

For each model, check:

refresh frequency;
refresh window;
dependency order;
gateway or cloud connection path;
last failure pattern;
whether the business still needs that refresh frequency.

The practical win is not always faster refresh.

Sometimes the win is fewer unnecessary refreshes, less capacity pressure, and fewer false support incidents.

3. Are credentials and connections safe to operate?

Credentials are one of the easiest places for Power BI estates to drift.

Personal credentials work until the person leaves, changes roles, rotates a password, or loses access. Gateway mappings work until a source moves. Cloud connections work until nobody remembers why a service principal was created.

For production models, I would check:

whether credentials belong to a person or an approved service identity;
whether the connection is documented;
whether the gateway path is still the right one;
whether the model uses sources that require special privacy or network rules;
whether connection failures are routed to someone who can fix them.

This is not glamorous work. It is the work that keeps trusted reports from turning into Monday morning escalations.

4. Is access aligned with how the model is consumed?

Power BI access is rarely only one thing.

A user might reach a report through a workspace, a shared link, an app, an org app audience, Excel, or a direct semantic model connection. That means model access deserves its own review, not just report access.

For each important model, verify:

workspace roles;
build permissions;
RLS roles and members;
sensitivity labels;
endorsement or certification status;
downstream reports and apps;
whether external sharing is relevant;
whether Copilot or AI experiences may rely on the model metadata.

The model is the contract. Reports are only one consumer of that contract.

A practical operating model

The settings pane is useful because it puts more of this work closer to where model owners already work.

But UI proximity is not the same as operational discipline.

I would define a simple operating model around critical semantic models.

For low-risk models

A lightweight review is enough.

named owner;
refresh schedule checked;
credentials confirmed;
access reviewed when someone asks for build permission.

For shared departmental models

Add a little more control.

owner register;
refresh failure routing;
quarterly access review;
measure descriptions for core KPIs;
endorsement policy;
documented source systems.

For executive or operational models

Treat the semantic model like production infrastructure.

business and technical ownership;
change review for major model updates;
tested refresh dependencies;
approved credential pattern;
RLS and access review;
monitoring for refresh failures;
support path;
documentation that a new admin can understand.

That sounds heavier, but it is usually lighter than explaining why the CEO dashboard is stale on the morning of a board meeting.

The checklist I would keep beside the pane

Here is the short version I would use with a BI team.

Check	Question
Ownership	Who owns the business meaning and who owns the technical model?
Refresh	Does the schedule match the actual business need?
Dependencies	What upstream jobs, gateways, or sources does this model depend on?
Credentials	Are credentials tied to a safe identity pattern?
Access	Who can view, build, share, or manage the model?
RLS	Are roles still correct and tested?
Labels	Is the sensitivity label appropriate for the data?
Endorsement	Should this be promoted, certified, or left unendorsed?
Documentation	Can a new admin understand how this model works?
Support	Who gets called when refresh fails?

This is not bureaucracy. It is basic model hygiene.

Where this fits in the bigger Power BI shift

Power BI is slowly moving more model work into the service.

Web modeling, model options in the service, TMDL view on the web, measure descriptions from triple-slash comments, org apps, audience management, and richer APIs all point in the same direction.

The browser is becoming a serious place to operate BI assets, not just consume reports.

That creates a useful tension.

More people can make changes.

More changes can happen closer to the workspace.

More model administration can happen without opening Desktop.

That is good for speed, but it also means teams need clearer standards.

The semantic model settings pane is a small feature if you only look at the screen.

It is a bigger feature if you use it as the front door to model governance.

My take

I like this update because it makes the right admin work easier to reach.

But I would not sell it internally as "new settings UI".

I would sell it as a chance to clean up semantic model ownership.

Pick the top ten models in the tenant. Open the settings pane. Check owners, refresh, credentials, access, labels, endorsement, and support routing.

You will probably find at least one risk worth fixing before it becomes visible to the business.

That is the real value.

Sources

About the author

Shai Karmani is a senior data and AI practitioner focused on Microsoft Fabric, Power BI, analytics engineering, automation, and practical AI systems.

Connect with Shai on LinkedIn: https://www.linkedin.com/in/shai-kr

OneLake Security Just Got Easier to Trust Across Fabric

Shai Karmani — Tue, 28 Jul 2026 22:35:54 +0000

Originally published at Data Ninja AI Lab.

Microsoft Fabric keeps moving toward a more useful idea of governance: define access close to the data, then let more of the platform respect it.

The latest OneLake security updates are worth paying attention to because they are not only about another admin setting. They make Fabric security fit better with how real enterprises already operate: groups, nested groups, service principals, shortcuts, shared lakehouses, SQL endpoints, Eventhouse, and Graph Database.

That is the part that matters.

A data platform does not become trusted because it has a nice diagram. It becomes trusted when the same user, group, workload identity, or agent gets the right answer through the actual path they use to reach the data.

The useful shift

OneLake Security already points Fabric in the right direction. Instead of duplicating access rules in every tool, data owners can define roles and policies on the lakehouse, including row-level security, column-level security, and object-level security.

The SQL analytics endpoint can then pass the signed-in user identity through to OneLake, so the endpoint is not a separate permission universe.

The new updates make that model more practical.

Microsoft called out improvements for SQL analytics endpoints, nested group support, shortcuts, service principals, and hub-and-spoke lakehouse patterns. In a separate OneLake security update, Microsoft also described broader engine coverage, including Eventhouse in preview and Graph Database generally available, plus a simplified column-level security experience.

That combination is important because most Fabric estates are not tidy.

They have central lakehouses and team-owned workspaces. They have security groups managed by IT. They have shortcuts. They have service principals. They have BI users querying through SQL endpoints. They increasingly have agents and automated workloads that need governed access too.

If OneLake security works only for the clean demo path, it is not enough.

Why SQL analytics endpoints matter so much

The SQL analytics endpoint is one of the most important adoption bridges in Fabric.

Data engineers may think in Lakehouse tables, Spark, Delta, or notebooks. BI teams and SQL operators often still need a SQL surface they can query, inspect, troubleshoot, and connect to downstream tools.

That bridge is useful only if users can trust the security boundary.

If a table is protected in OneLake but the SQL endpoint behaves differently, teams get nervous. If a nested group works in Entra ID but not in the data access path, admins create manual exceptions. If service principals behave differently from user identities, automation becomes fragile.

The latest update matters because it addresses exactly that kind of enterprise friction.

It is not glamorous. It is the work that makes the platform believable.

The governance point for BI and data teams

The instinct will be to treat this as a security team update.

I would not.

This affects how analytics teams design the estate.

A central OneLake security model changes the questions teams should ask before they build yet another copy of the data:

Can this dataset stay in one governed lakehouse instead of being copied into team workspaces?
Should access be managed through existing Entra groups rather than local workspace exceptions?
Which columns should be governed at the data layer instead of hidden in a report?
Which workloads need service principal access, and how will that be reviewed?
Which query engines must be validated before the data owner calls the policy complete?

That last question is the one teams often miss.

A policy is not complete when it exists in the admin screen. It is complete when the expected users and workload identities get the expected result through the actual engines they use.

A practical rollout pattern

If I were rolling this out, I would not start with a massive policy rebuild.

I would start with one shared lakehouse, one sensitive table, and three access paths:

A normal BI user through the SQL analytics endpoint.
A user who inherits access through a nested group.
A workload identity or service principal used by automation.

Then I would validate the result against the OneLake policy.

The goal is not only to prove that the feature works. The goal is to find the places where your current access model is messy.

That usually means finding old groups nobody owns, shortcuts that were created for convenience, columns that should have been classified earlier, and reports that quietly depended on copied data.

The AI agent angle

There is also an agent story here.

Fabric data agents, Foundry integration, MCP endpoints, and AI-assisted data workflows are all moving in the same direction: agents need access to governed business data.

That does not work if the governance model is mostly report-level hiding or workspace-level habit.

Agents need a data access model that is explicit, testable, and close to the source. OneLake security improvements are part of that foundation.

The point is not to give agents more data.

The point is to make sure an agent sees only the data it is supposed to see, through the same governed paths a human or application would use.

What I would watch next

The feature direction is good. The operating model is the hard part.

For Fabric admins and BI leaders, I would watch five things:

how consistently OneLake security applies across the engines your teams actually use;
whether nested group support matches your Entra ID design;
how service principal access is requested, reviewed, and audited;
how shortcuts behave in hub-and-spoke lakehouse designs;
whether column-level security becomes a reusable data contract, not a one-off workaround.

That is where this update becomes more than a product announcement.

It becomes a chance to reduce duplicated security logic across the analytics estate.

Bottom line

OneLake security is getting closer to how enterprise Fabric environments actually work.

That is good news for BI teams, data engineers, platform admins, and anyone trying to make AI over enterprise data safer.

The practical next step is simple: pick one important lakehouse and test the real access paths. SQL endpoint. Groups. Shortcuts. Service principals. Eventhouse or Graph if they are part of your architecture.

If the results are consistent, you have a stronger governance foundation.

If they are not, you just found the work that was already waiting for you.

Sources

Written by Shai Karmani, senior data and analytics engineering leader focused on Microsoft Fabric, Power BI, governance, and practical AI implementation.

Build Fabric Dataflows From Chat and Still Keep Control

Shai Karmani — Mon, 27 Jul 2026 22:18:32 +0000

Originally published at Data Ninja AI Lab.

Microsoft Fabric is getting closer to a very practical version of AI-assisted data engineering.

The interesting part is not that an assistant can write M code or help create a Dataflow Gen2 item. We have had code generation for a while. The useful shift is that Microsoft now has a Data Factory MCP server that exposes Fabric resource discovery and Data Factory operations through a standard assistant interface.

That changes the conversation.

A data engineer can ask an AI assistant to inspect available workspaces, look at connections, create or update Dataflows, work with pipelines, run jobs, monitor status, and manage schedules. That is not a demo feature. That is the start of an operational surface for AI-assisted data work.

The benefit is obvious: less manual setup, fewer repeated portal steps, faster iteration, and a more natural way to build Fabric assets from intent.

The risk is also obvious: if teams treat this as a shortcut around engineering discipline, they will create faster mess.

What Data Factory MCP actually gives teams

Microsoft's Data Factory MCP project describes an MCP server for Microsoft Fabric resource discovery and information retrieval, with tool coverage across authentication, gateways, connections, workspaces, dataflows, pipelines, copy jobs, Apache Airflow jobs, and capacities.

That tool surface matters.

It means an assistant is not only answering questions about Fabric. It can interact with the same operational objects that determine whether data work is reliable: connections, gateways, schedules, definitions, run status, and pipeline execution.

Used well, this can reduce the boring friction in Data Factory work:

create a draft Dataflow Gen2 structure;
add or update queries;
connect it to known Fabric connections;
create a pipeline around the flow;
run a test refresh;
inspect status;
iterate from the same chat loop.

That is a good direction. Data engineering has too much manual plumbing. Repeated setup work should be automated.

But this is also exactly why the control model matters.

The control point moves from authoring to review

In the old portal-first workflow, the friction was in creation. You clicked through items, settings, connections, gateway choices, schedules, destinations, and refresh behavior.

In an AI-assisted workflow, creation becomes cheaper.

That does not remove the need for review. It moves the review closer to the generated definition.

For a production team, the important questions are not only:

Can the assistant create the dataflow?
Can it add a query?
Can it run a pipeline?

The better questions are:

Which identity is the assistant using?
Which workspace can it touch?
Which connections can it discover or create?
Can a human review the changed definition before it runs?
Is the output destination explicit?
Is the schedule owned by a team or by whoever tried the demo first?
Is there a rollback path if the generated change is wrong?
What gets logged when it runs?

This is where experienced data teams will separate themselves from teams chasing the shiny part.

AI-assisted Fabric work should feel like a pull request, not like an invisible macro.

A practical first pilot

The first pilot should be intentionally small.

Pick one workspace, one source system, one Dataflow Gen2 item, and one pipeline. Do not give an assistant broad access to the estate just because the setup is exciting.

A good pilot has four gates.

1. Scope gate

Limit the assistant to a non-critical workspace and a known source. The goal is to prove the workflow, not automate the most sensitive pipeline first.

The output should be a small Fabric asset that the team can inspect by hand.

2. Identity gate

Use a deliberate identity model. Interactive authentication is useful for exploration, but production patterns need service principals, managed identities, or tightly controlled user access depending on the scenario.

The point is simple: the item should not depend on the wrong person being logged in at the wrong time.

3. Review gate

Every generated change needs a reviewable definition. That can be a Dataflow definition, pipeline definition, schedule configuration, or connection mapping.

The review should check:

source and destination;
credential and connection choice;
transformation logic;
refresh behavior;
failure path;
naming and ownership;
whether the change is safe to repeat.

This is not bureaucracy. It is how teams avoid turning AI speed into production uncertainty.

4. Operations gate

A created pipeline is not done when it runs once.

It needs a run history, failure notification path, schedule owner, cost expectation, and cleanup rule. If the pilot creates an asset nobody owns after the demo, the pilot failed.

The opportunity for Fabric teams

The real opportunity is not replacing Data Factory engineers. It is removing low-value mechanical work so engineers can spend more time on design, testing, data contracts, and operational quality.

That is where this starts to matter for analytics engineering teams.

Imagine a workflow where an assistant can draft the dataflow, wire the connection, create the first pipeline, and run the test, while the engineer reviews the definition and decides what gets promoted.

That is useful.

It is also realistic.

The teams that get the most value will not be the ones that give the assistant the broadest access. They will be the ones that give it a clear lane:

known workspaces;
approved connections;
reusable query patterns;
naming rules;
test data;
review steps;
promotion gates.

That is how AI-assisted Data Factory work becomes an engineering workflow instead of another pile of generated assets.

My take

Data Factory MCP is one of the more practical Fabric AI signals because it sits where work actually happens: connections, dataflows, pipelines, copy jobs, schedules, and run status.

That makes it more useful than a generic chat demo.

It also means Fabric admins and data teams should think about it early. Not because every team needs to adopt it immediately, but because the control model needs to be designed before assistants start changing production assets.

If I were piloting this, I would start with one question:

Can we let an assistant build the first draft while keeping identity, review, testing, and operations fully human-owned?

If the answer is yes, this is worth testing.

If the answer is no, the assistant is not the problem. The platform workflow is.

Sources

Written by Shai Karmani.

Connect with me on LinkedIn: https://www.linkedin.com/in/shai-kr

Bring Power BI Answers Into the Flow of Work With Fabric IQ

Shai Karmani — Fri, 10 Jul 2026 03:15:08 +0000

Originally published at Data Ninja AI Lab.

Power BI answers are starting to move closer to where decisions already happen.

That is the useful part of Fabric IQ in Microsoft 365 Copilot Chat.

The feature lets users ask Microsoft 365 Copilot questions grounded in Power BI reports and semantic models. Instead of switching to Power BI, finding the right report, applying filters, and interpreting the visual, a user can ask a data question in Copilot Chat and bring the answer into the same place where they are already working with files, chats, emails, and meetings.

That sounds small if you look at it as another Copilot surface.

It is bigger if you look at the workflow.

For years, BI teams have tried to pull users into dashboards. This pattern starts pulling governed BI answers into the user’s normal decision path.

That is a real opportunity. It also raises the bar for semantic model quality.

When a Power BI report is opened by a trained analyst, the analyst brings context. They know which measure to trust, which filter matters, which visual is old, and which field name is misleading.

When a question is asked through Copilot Chat, much of that implicit human context has to be built into the model, the report, the access model, and the operating process.

That is the part teams should prepare for now.

What changed

Microsoft’s Fabric IQ connector for Microsoft 365 Copilot Chat is currently described as a Frontier capability. It allows eligible users to ask questions grounded in Power BI reports and semantic models from Copilot Chat.

The important mechanics are straightforward:

Copilot uses the user’s existing permissions to access relevant Power BI content.
Users can reference reports by pasting a report link, using the attachment menu when available, or naming the report in the prompt.
The answer is grounded in Power BI data, then can be reconciled with broader Microsoft 365 context.
Users need the right Microsoft 365 Copilot licensing and access to the relevant Power BI content.

That last point matters.

This does not remove BI governance. It exposes BI governance in a new place.

If your semantic model is well named, secure, documented, refreshed, and owned, this can become a very useful decision layer.

If the model is messy, the mess now has a new audience.

The architecture pattern

The clean mental model is not “Copilot answers everything.”

The better model is this:

The user asks a business question in Microsoft 365 Copilot Chat.
Fabric IQ helps connect that question to relevant Power BI content.
The semantic model provides measures, relationships, security, business names, and definitions.
Copilot brings the answer back into the user’s work context.

That makes the semantic model the contract.

The report still matters. The visuals still matter. But for conversational answering, the semantic model becomes even more important because it carries the business logic.

The model needs to answer questions without relying on a report author standing next to the user.

That means the basics become production requirements:

measures need clear names
field descriptions need to exist
hidden technical fields should stay hidden
certified models need to be obvious
RLS and OLS need to be tested with real user scenarios
refresh expectations need to be documented
ownership needs to be visible

None of this is glamorous. That is why it matters.

Most AI failure modes in BI will not come from spectacular model hallucinations. They will come from ordinary BI hygiene gaps that were tolerable inside a dashboard and painful inside a chat answer.

A readiness checklist for BI teams

Before I would promote this broadly, I would run a readiness check across four areas.

1. Model quality

Start with the semantic model.

A conversational answer depends on the model being understandable without a human translator.

Check:

Are the key measures named in business language?
Do important fields have descriptions?
Are technical columns hidden from the user experience?
Are calculation groups, relationships, and measure folders organized enough to support discovery?
Is the model certified or promoted when it should be?
Are duplicate or outdated models still competing for attention?

This is where many teams will need cleanup.

If five models all claim to represent “sales,” Copilot is not the root problem. The estate is.

2. Security and permissions

Copilot uses the user’s existing access, so the permission model has to be correct before the feature becomes trusted.

Check:

Is RLS tested with the same roles real users have?
Is OLS used where sensitive fields should not be exposed?
Are workspace permissions tighter than “everyone can view everything”?
Are sensitivity labels aligned with the data’s real business meaning?
Are report and semantic model permissions reviewed together?

The key test is simple:

If this user asks a question in chat, would we be comfortable with the same answer appearing in a meeting recap or shared work thread?

If not, fix access before expanding usage.

3. Operations

A chat answer feels immediate. That makes stale data more dangerous.

Inside a report, users sometimes notice context clues: refresh timestamps, page titles, filters, bookmarks, or report notes. In chat, the answer may feel more direct and more final.

That means the operating model needs to be explicit.

Check:

What is the refresh SLA for the model?
Who owns failed refreshes?
Who owns bad answers?
How are model changes reviewed?
Where do users report issues?
How often are usage and failures reviewed?

A good Copilot experience is not only a good prompt experience. It is a supportable data product.

4. Answer design

Not every question should be answered the same way.

Some questions need a number. Some need a trend. Some need a filtered slice. Some need a warning that the model does not contain the right context.

Create a small answer design guide for the first pilot:

which questions are supported
which questions are out of scope
which report or model should be referenced
how users should phrase common questions
what answer quality looks like
when the user should open the report instead of relying on chat

That guidance does not have to be heavy. One page is enough for a pilot.

But it should exist.

A practical pilot path

I would not roll this out across the whole Power BI estate first.

I would choose one high-value scenario and make it boringly reliable.

Step 1: Choose one recurring business question

Pick a question people already ask every week.

Good candidates:

“How are sales tracking this month?”
“Which region is behind target?”
“What changed in pipeline since last week?”
“Which customers are driving the variance?”
“What should I know before the forecast meeting?”

Avoid vague pilots like “ask anything about revenue.”

That invites noise.

Step 2: Pick the trusted report and semantic model

Choose one report and one model as the source of truth for the pilot.

Do not let the first pilot search across ten similar assets.

The point is to prove the answer path, not to test every governance edge case at once.

Step 3: Prepare the model for questions

This is the cleanup sprint.

Focus on:

business-friendly measure names
descriptions for the most important fields
certified status where appropriate
hidden fields that should not appear in answers
tested RLS and OLS
refresh visibility
clear ownership

If the model cannot explain itself, the chat experience will struggle.

Step 4: Test with real prompts

Use the questions people actually ask.

For each one, compare the Copilot answer to the Power BI report and the semantic model logic.

Capture:

correct answers
incomplete answers
confusing wording
unsupported questions
security or access surprises
cases where opening the report is still better

This becomes the pilot’s improvement backlog.

Step 5: Publish a small user guide

Users need guardrails, not a training course.

Give them:

three example prompts that work well
two examples that are out of scope
the trusted report name
the owner or support channel
a reminder that governed data still depends on refresh and model design

That is enough to start.

Step 6: Review after launch

After the pilot goes live, review what happened.

Look for:

repeated questions
confusing answers
reports users keep referencing manually
model fields that need better names
missing measures
access issues
opportunities to add a second scenario

This is where the value compounds.

Every good question teaches the BI team what the model needs to support next.

What I would tell BI teams to do now

Do not wait for every Copilot surface to be fully mature before cleaning up the model layer.

The preparation is useful either way.

A semantic model with clear names, tested security, good descriptions, certified ownership, and a refresh SLA is better for Power BI, Fabric Apps, AI skills, and Copilot Chat.

The same work improves the whole estate.

My short checklist would be:

For each candidate semantic model:

1. Confirm business owner and technical owner.
2. Confirm the model is the trusted source for a real decision path.
3. Review measure names and descriptions.
4. Hide technical fields from user-facing experiences.
5. Test RLS and OLS with real user roles.
6. Confirm refresh SLA and failure ownership.
7. Create 10 approved example questions.
8. Test answers against the report and source logic.
9. Publish a one-page user guide.
10. Review usage and misses after launch.

That is the work that turns a Copilot feature into a reliable business capability.

The bigger shift

For a long time, the center of gravity in BI was the report.

Then the semantic model became more important as teams standardized measures, lineage, security, and reusable business logic.

Now conversational interfaces are pushing that model layer into more places.

Microsoft 365 Copilot Chat is one of the most important places because it sits close to the work: meetings, files, messages, decisions, and follow-ups.

That does not make Power BI less important.

It makes Power BI governance more visible.

The teams that win here will not be the teams with the most dashboards. They will be the teams with the most trustworthy models and the clearest ownership.

That is a good direction.

It rewards the BI work that already should have mattered: definitions, security, freshness, ownership, and practical trust.

Sources

Want to discuss Power BI, Microsoft Fabric, or practical AI implementation? Connect with me on LinkedIn.

Make Fabric AI Agents Smarter With Labels You Already Own

Shai Karmani — Mon, 06 Jul 2026 22:22:24 +0000

Originally published at Data Ninja AI Lab.

Microsoft Fabric just gave sensitivity labels a more interesting job.

Most teams think about sensitivity labels as governance metadata: General, Confidential, Highly Confidential, custom labels, protection policies, access rules, and audit expectations.

That work still matters.

But the new AI angle is better: those labels can also help Fabric AI agents decide which data belongs in an answer.

That turns sensitivity labels from a control layer into a context layer.

And that is a very useful shift.

If an AI agent can access several reports, semantic models, lakehouses, or other Fabric items, the hard question is not only “does the user have permission?”

The harder question is:

Which sources should the agent consider for this question?

That is where labels become valuable. They give the agent a signal your organization already understands.

General data can support broader analysis. Confidential data may require tighter answer rules. Highly Confidential data may need explicit clearance, summary-only responses, escalation, or a full audit trail.

The practical win is simple: better answers with less noise and clearer governance.

The opportunity

AI agents do not fail only because they lack access.

They also fail because they have too much undifferentiated context.

A reporting skill that can read every report equally may produce an answer that is technically grounded, but still poorly scoped. It might mix public sales summaries with confidential forecast material. It might use executive planning data in a broad operational answer. It might answer with more detail than the scenario deserves.

That is not a model problem. It is a context design problem.

Sensitivity labels can help solve it.

Microsoft’s update describes a pattern where labels guide how an AI skill or agent selects and prioritizes data. The agent still respects protection and permissions, but labels also become a relevance signal.

In plain language:

The label tells the agent how the organization thinks about that data.

That is useful because organizations already invest time in classification. They already know which information is broadly shareable, which information needs care, and which information belongs in specific business contexts.

The next step is to stop treating that classification as something only humans and compliance tools can use.

Let the agent use it too.

The architecture pattern I would use

I would not start with a complex policy framework.

I would start with a small behavior map.

For each label category, define what the agent is allowed to do with that content.

A simple first version might look like this:

General

Use it freely for normal analysis.

The agent can summarize it, compare it, cite it, and use it as the primary context for broad business questions.

Good fit for:

public sales summaries
operational KPI reports
broadly shared semantic models
documentation intended for many teams

Confidential

Use it only when relevant and when the user is authorized.

The agent can summarize, but it should avoid pulling confidential detail into broad answers unless the question clearly requires it.

Good fit for:

budget forecasts
customer-specific analysis
margin or pricing reports
internal planning material

Highly Confidential

Use it only for explicit, cleared scenarios.

The agent may need to decline, escalate, or provide a high-level briefing instead of a detailed answer.

Good fit for:

executive strategy
M&A planning
legal-sensitive reporting
restricted financial planning

Your labels will not look exactly like this. They should not.

The important part is the translation layer: label to agent behavior.

Without that translation, the label exists, but the AI skill does not know what to do with it.

A practical example

Imagine a Fabric AI skill that analyzes Power BI reports and answers questions about business performance.

A user asks:

What are our Q3 projections?

The skill can see several possible sources:

a General-labeled sales performance report
a Confidential finance forecast semantic model
a Highly Confidential executive planning report
a department-specific Lakehouse table

A normal permission check answers only part of the question.

The user may be allowed to access more than one of those assets. But permission alone does not tell the agent which source should shape the answer.

A label-aware skill can apply a better decision model:

If the question is broad and operational:
  prefer General-labeled sources
  use Confidential sources only when the question explicitly needs them
  exclude Highly Confidential sources unless the user and scenario are cleared

If Confidential sources are used:
  summarize first
  avoid unnecessary row-level detail
  record the source and rationale

If Highly Confidential sources are needed:
  require explicit clearance
  return a briefing or escalate
  log the question and source selection

That is not heavy governance. That is practical answer design.

The agent becomes more useful because it stops treating every accessible source as equally appropriate.

The pilot playbook

The safest way to use this pattern is to pilot it with one skill and one business scenario.

Here is the sequence I would use.

1. Pick one skill

Do not start across the whole estate.

Pick one focused AI skill:

report analysis
KPI explanation
budget Q&A
customer summary generation
operations briefing

The skill should have a clear business owner and a clear answer pattern.

If nobody owns the answers, nobody will own the rules.

2. Inventory the sources

List the Fabric items the skill can use:

reports
semantic models
lakehouses
warehouses
Eventhouse tables
notebooks or generated outputs
supporting documents, if they are part of the workflow

For each source, capture owner, label, workspace, refresh pattern, and business purpose.

This is where many teams find the first gap: important sources are unlabeled, inconsistently labeled, or owned by the wrong team.

That is useful to know before the agent starts answering real questions.

3. Check label coverage

A label-aware skill is only as good as the labels behind it.

Before building rules, ask four questions:

Are the key sources labeled?
Are labels applied consistently across reports, semantic models, and data items?
Does each label have a clear business meaning?
Are there exceptions the agent needs to know about?

If the answer is no, fix the labeling pattern first.

Otherwise the skill will learn from inconsistent signals.

4. Write agent behavior rules

This is the part most teams will be tempted to skip.

Do not skip it.

For each label, define what the skill should do:

use normally
use only when clearly relevant
summarize instead of showing detail
exclude from broad answers
require a higher clearance path
log usage
ask a clarifying question
decline and explain why

Keep the rules short enough for a product owner, data owner, and compliance stakeholder to read together.

If the rules are too complex for review, they will be too complex to operate.

5. Test answer quality

Do not test only whether the agent blocks the right things.

Test whether the answer gets better.

Use a small evaluation set:

broad business questions
sensitive finance questions
ambiguous questions
questions that should trigger clarifying prompts
questions that should avoid sensitive sources
questions that should escalate

Then compare the agent before and after label guidance.

Look for four outcomes:

less irrelevant context
fewer mixed-context answers
clearer explanation of source choice
stronger auditability

That is the point of this pattern.

What improves when labels guide context

The most obvious benefit is governance, but the more interesting benefit is answer quality.

A label-aware skill can avoid a common AI problem: over-answering.

If a user asks a broad question, the agent does not need to pull the most sensitive source just because it can. It can start with the broadly appropriate source and only move into restricted context when the question, permission, and scenario justify it.

That makes the answer easier to trust.

It also makes the system easier to explain.

When someone asks why the agent used a source, the answer is not vague. It can point to a rule:

this source was General-labeled and appropriate for broad analysis
this Confidential source was used because the question explicitly asked for forecast detail
this Highly Confidential source was excluded because the user did not have the required scenario clearance

That is the kind of explanation real organizations need if AI agents are going to touch business data.

The checklist

Before using sensitivity labels as AI guidance, I would want these items in place.

Label foundation

Key Fabric items are labeled.
Labels are defined in Microsoft Purview.
Labels have business meaning, not only compliance meaning.
Custom labels are documented where they exist.
Owners know which assets they are responsible for labeling.

Agent design

The skill has a defined business scenario.
The allowed source list is explicit.
Label-to-behavior rules are documented.
The skill has a fallback path for ambiguous questions.
The skill can explain why it used or avoided a source.

Governance and operations

Confidential and highly sensitive usage is logged.
Rules are reviewed when labels change.
Output behavior is tested with real example questions.
There is a process for correcting mislabeled sources.
The business owner can review sample answers before rollout.

Quality review

Answers use fewer irrelevant sources.
Sensitive context does not leak into broad answers.
Refusals are understandable.
Summaries preserve meaning without exposing unnecessary detail.
Audit logs show which sources shaped the answer.

This is the difference between “we added AI” and “we designed how AI should behave around our data.”

What I would avoid

I would avoid three traps.

First, do not assume access control is enough.

Access control tells you what a user can reach. It does not always tell you what an AI answer should include.

Second, do not build label rules only for the most sensitive case.

The everyday value is often in the middle: helping agents choose between broad operational context and confidential planning context.

Third, do not let every skill invent its own interpretation of labels.

If one agent treats Confidential as “summary only” and another treats it as “full detail if authorized,” users will not understand the system. Create a shared pattern, then adjust only where the scenario requires it.

The practical takeaway

This update is bigger than it looks.

Sensitivity labels are already part of Fabric and Power BI governance. Microsoft is now showing how those same labels can help AI agents produce more relevant and context-aware answers.

That is exactly the kind of pattern teams need as agents move from demos to real data work.

Start small:

pick one skill
inventory the sources
check label coverage
define label-to-behavior rules
test whether answers improve

The best version of this is not AI with a compliance sticker on top.

It is AI that understands the context your organization already uses to manage data.

That is how Fabric AI agents become smarter, safer, and more useful.

Source

Microsoft Fabric Updates Blog: Use sensitivity labels to improve AI Agents accuracy and organizational alignment

Microsoft Learn: Information protection in Microsoft Fabric

Microsoft Learn: Apply sensitivity labels in Microsoft Fabric

Shai Karmani is a data engineering, Microsoft Fabric, Power BI, and AI practitioner focused on building practical data systems people can trust.

Fabric Warehouse Brings AI Enrichment Into T-SQL. Here’s the Practical Guide.

Shai Karmani — Sun, 28 Jun 2026 00:23:05 +0000

Originally published at Data Ninja AI Lab.

Fabric Data Warehouse now has preview AI functions that let you classify, summarize, translate, extract, and improve text directly from T-SQL.

That is a bigger shift than it first looks.

For years, a lot of text enrichment work has lived outside the warehouse. It gets handled in notebooks, one-off Python scripts, Power Query steps, spreadsheets, application code, or manual cleanup queues. Sometimes that is the right place. Often, it creates another disconnected transformation layer that nobody governs properly.

The interesting part of these functions is not that Fabric can call AI from SQL. The interesting part is that common text intelligence tasks can now sit closer to governed warehouse workflows.

That gives data teams a practical option:

enrich support tickets before they reach a semantic model
classify messy feedback into controlled categories
extract structured fields from free text
translate multilingual comments for analysis
summarize long operational notes
clean user-entered text before reporting
generate controlled response drafts from trusted data

Used well, this can reduce friction between AI experiments and production analytics.

Used badly, it can bury expensive, non-deterministic logic inside report queries.

The difference is architecture.

What Microsoft added

Microsoft documents seven preview AI functions for Fabric Data Warehouse and the SQL analytics endpoint.

Function	What it does	Practical use
`AI_ANALYZE_SENTIMENT(text)`	Returns `positive`, `negative`, `mixed`, or `neutral`	Review analysis, support triage, survey feedback
`AI_CLASSIFY(text, class1, class2, ...)`	Classifies text into labels you provide	Ticket routing, complaint categories, product issue groups
`AI_EXTRACT(text, field1, field2, ...)`	Extracts fields as JSON	Pulling problem, date, sentiment, location, or entity values from text
`AI_SUMMARIZE(text)`	Produces a shorter summary	Condensing long notes for analysts or dashboards
`AI_GENERATE_RESPONSE(prompt, data)`	Generates a response from a prompt and optional data	Response drafts, internal summaries, controlled explanation text
`AI_TRANSLATE(text, lang_code)`	Translates text into supported languages	Multilingual support and feedback analysis
`AI_FIX_GRAMMAR(text)`	Corrects grammar in text	Cleaning user-entered comments or notes

These are not replacements for data modeling, governance, or review. They are transformation tools.

That distinction matters.

A simple example

Imagine a support_cases table with a free-text case_notes column.

A useful enrichment table might include sentiment, a business category, a short summary, and structured fields extracted from the notes.

CREATE TABLE curated.support_case_ai_enrichment AS
SELECT
    case_id,
    case_notes,
    AI_ANALYZE_SENTIMENT(case_notes) AS sentiment,
    AI_CLASSIFY(
        case_notes,
        'billing',
        'delivery',
        'technical issue',
        'account access',
        'other'
    ) AS case_category,
    AI_SUMMARIZE(case_notes) AS case_summary,
    AI_EXTRACT(
        case_notes,
        'problem',
        'product',
        'urgency',
        'time_reported'
    ) AS extracted_json
FROM staging.support_cases;

That output can then feed Power BI, a semantic model, an operations dashboard, or a downstream process.

But I would not put this directly inside a report-facing query that runs every time a user opens a dashboard.

Microsoft’s documentation calls out two practical constraints:

AI functions can return NULL if the model cannot process the text.
Typical processing speed is around 20 to 100 rows per second.

That points to the correct pattern: precompute and materialize repeated transformations.

The function guide

1. Use `AI_ANALYZE_SENTIMENT` for directional signals

Sentiment is useful when you need a rough business signal from text.

Examples:

customer review sentiment
employee survey comments
support ticket tone
partner feedback
product complaint notes

Good use:

SELECT
    review_id,
    AI_ANALYZE_SENTIMENT(review_text) AS review_sentiment
FROM staging.customer_reviews;

What I would not do: treat sentiment as absolute truth. It should support analysis, not replace review for high-impact cases.

2. Use `AI_CLASSIFY` when you control the categories

AI_CLASSIFY is strongest when the business already knows the target categories.

Example:

SELECT
    case_id,
    AI_CLASSIFY(
        case_notes,
        'billing',
        'service',
        'technical issue',
        'contract',
        'other'
    ) AS case_type
FROM staging.support_cases;

The governance point is simple: the labels are part of the data contract. If the business changes the categories, the transformation logic changed too.

Track that change like you would track a schema change.

3. Use `AI_EXTRACT` to turn text into structured fields

This is the most interesting function for analytics engineering.

Free text often contains useful structure, but parsing it with regular expressions gets brittle fast. AI_EXTRACT lets you ask for fields and returns JSON.

Example:

SELECT
    case_id,
    AI_EXTRACT(
        case_notes,
        'problem',
        'affected_system',
        'urgency'
    ) AS extracted_case_details
FROM staging.support_cases;

For reporting, I would normally parse the JSON into typed columns in a curated table.

SELECT
    s.case_id,
    j.problem,
    j.affected_system,
    j.urgency
FROM staging.support_cases s
CROSS APPLY OPENJSON(
    AI_EXTRACT(s.case_notes, 'problem', 'affected_system', 'urgency')
)
WITH (
    problem VARCHAR(1000),
    affected_system VARCHAR(200),
    urgency VARCHAR(100)
) AS j;

This is where validation matters. Sample the output. Review wrong extractions. Keep the original text.

4. Use `AI_SUMMARIZE` for readability, not evidence

Summaries are useful when analysts need context without reading a full comment field.

Example:

SELECT
    incident_id,
    AI_SUMMARIZE(incident_notes) AS incident_summary
FROM staging.incidents;

The summary should not become the only version of the record. Keep the original text beside it or one click away.

A summary is a reading aid. It is not the source of truth.

5. Use `AI_TRANSLATE` when language blocks analysis

AI_TRANSLATE can help standardize multilingual text for analysis.

Example:

SELECT
    feedback_id,
    feedback_text,
    AI_TRANSLATE(feedback_text, 'en') AS feedback_text_en
FROM staging.customer_feedback;

Microsoft lists supported language codes including en, de, fr, it, es, el, pl, sv, fi, and cs.

For global reporting, this can make feedback analysis easier. Still, translation can change nuance, especially in complaints, legal text, or regulated workflows. Keep the original language value.

6. Use `AI_FIX_GRAMMAR` carefully

Grammar correction is useful for presentation and readability.

Example:

UPDATE curated.customer_feedback
SET cleaned_comment = ISNULL(
    AI_FIX_GRAMMAR(raw_comment),
    raw_comment
);

The ISNULL pattern matters. Microsoft’s docs note that AI functions can return NULL, so avoid overwriting useful source text with a blank result.

I would use this for cleaned display fields, not for replacing the original source record.

7. Use `AI_GENERATE_RESPONSE` with the most discipline

This function can generate text from a prompt and data.

Example:

SELECT
    case_id,
    AI_GENERATE_RESPONSE(
        'Write a concise internal summary for a support manager:',
        case_notes
    ) AS manager_summary
FROM staging.support_cases;

This is powerful, but it is also where teams need the most control.

If generated text will be sent to customers, used in decisions, or shown in operational workflows, add human review, prompt ownership, audit fields, and clear usage rules.

Generated text should not quietly become an automated business action.

The production pattern I would use

I would treat each AI function output as a governed transformation.

That means:

Profile the source text first.
Choose the function based on the business task.
Materialize the output in a staging or enrichment table.
Keep the original text.
Add fallback behavior for NULL results.
Validate a sample of outputs.
Track prompt or label changes.
Monitor refresh time and cost.
Separate experimental outputs from certified reporting fields.

The practical table design might include:

CREATE TABLE curated.support_case_ai_enrichment
(
    case_id BIGINT,
    source_text_hash VARCHAR(200),
    ai_function_used VARCHAR(100),
    ai_labels_or_prompt VARCHAR(4000),
    ai_output VARCHAR(4000),
    ai_output_status VARCHAR(50),
    validated_flag BIT,
    validation_sample_group VARCHAR(100),
    created_at DATETIME2(6),
    created_by_pipeline VARCHAR(200)
);

This may look heavy for a demo. It is not heavy for production.

If the output will influence reporting, routing, prioritization, or an AI agent, someone needs to know where it came from and how it was produced.

Where this fits in a Fabric architecture

The strongest use cases are not generic AI demos.

They are specific enrichment steps inside a real data workflow:

classify customer feedback before semantic model refresh
extract product issue fields from support notes
summarize long incident comments for operations dashboards
translate multilingual survey comments for regional comparison
clean messy user-entered text in curated reporting tables
generate internal response drafts for review queues

That is the sweet spot.

The warehouse becomes a controlled place to enrich text, not just store it.

What I would avoid

I would avoid four patterns:

Calling AI functions repeatedly in interactive report queries.
Treating model output as deterministic truth.
Replacing original text with AI-cleaned text.
Using generated responses without review, ownership, and audit.

Preview features are for learning and controlled adoption. The right move is to test the pattern, measure the cost, validate outputs, and decide where it belongs in the pipeline.

Final take

This is a useful direction for Fabric.

Not because it makes every warehouse query smarter by default. That would be the wrong framing.

It is useful because it gives data teams a practical way to move common text enrichment closer to the governed data layer.

If your team already has free-text feedback, support cases, notes, reviews, comments, or multilingual text sitting in the warehouse, these functions are worth testing.

Just test them like production data transformations, not like magic buttons.

Start with one table, one business use case, one materialized output, and one validation sample.

That is enough to learn quickly without turning AI-in-SQL into another unmanaged layer.

Fabric Lakehouse Health Checks Make Optimization Practical. Here’s the Runbook.

Shai Karmani — Wed, 24 Jun 2026 22:50:45 +0000

Originally published at Data Ninja AI Lab.

Microsoft Fabric just added a small feature that can change how teams maintain Lakehouse tables.

The new sp_get_table_health_metrics stored procedure gives SQL analytics endpoint users a T-SQL way to inspect Lakehouse table health before deciding whether Spark maintenance is needed.

That sounds narrow. It is not.

For teams serving Power BI, SQL users, downstream data products, or AI workflows from Lakehouse tables, this closes an annoying operational gap: the place where users feel the slowdown is often SQL, but the maintenance action usually happens in Spark.

Until now, a lot of teams handled that gap with guesswork.

Run OPTIMIZE every night. Compact everything on a schedule. Wait until dashboards get slow. Open a notebook. Inspect Delta files. Ask support. Hope the maintenance job was worth the compute.

The better pattern is simple:

Check table health first. Optimize only when the evidence says to.

That is the practical win here.

The problem this solves

Lakehouse tables can look fine logically while becoming less efficient physically.

The schema is still valid. The row counts still make sense. The reports still refresh. But over time the physical layout can drift:

too many small files
too many deleted rows
stale or missing checkpoints
uneven row distribution
invalid or weak file statistics
fragmented table layout after frequent writes, deletes, or merges

Users usually experience this as slower SQL queries or lagging Power BI reports.

Data engineers experience it as a vague support problem:

The dashboard is slow. Can you check the Lakehouse?

That is a bad starting point. It pushes the team into reactive troubleshooting instead of evidence-based maintenance.

sp_get_table_health_metrics gives the SQL side a diagnostic step. It does not replace Spark maintenance, but it gives teams a better way to decide when Spark maintenance is actually justified.

What the stored procedure gives you

Microsoft’s announcement describes a built-in stored procedure for the SQL analytics endpoint that returns table health signals for Lakehouse tables.

The useful part is not just one metric. It is the mix of signals:

PotentialAnomalyType
PotentialAnomalyDescription
snapshot and checkpoint versions
physical row counts
deleted row counts
file size distribution
row count distribution
deleted row distribution

That gives you a much better conversation than “the table feels slow”.

You can ask more specific questions:

Is this a small-file problem?
Are deleted rows accumulating?
Is the table missing a recent checkpoint?
Are file statistics valid?
Is this table actually healthy and the issue is somewhere else?

That last point matters.

A health check can save capacity by proving that a maintenance job is not needed.

The runbook I would use

I would not treat this as a one-off troubleshooting command. I would turn it into a small operational runbook.

1. Start with the critical tables

Do not begin by checking every table in the Lakehouse.

Start with tables that actually matter:

tables behind important Power BI semantic models
tables queried heavily through the SQL analytics endpoint
fact tables with frequent incremental writes
tables touched by merge, delete, or update patterns
tables used as context for AI agents or downstream apps
tables with known performance complaints

This keeps the first version focused.

A table nobody queries does not need the same operational attention as the table behind the CFO dashboard.

2. Run the health check before maintenance

The basic pattern is straightforward.

EXEC sp_get_table_health_metrics @table_name = 'schema.YourTable';

In a real runbook, I would capture the output instead of only looking at it manually.

For example, create a control table that stores:

run timestamp
workspace or environment
Lakehouse name
table name
anomaly type
anomaly description
selected file distribution metrics
maintenance decision
action taken
post-check result

The goal is not to create bureaucracy. The goal is to make maintenance reviewable.

If someone asks why a table was optimized yesterday, the answer should not be “because the schedule said so”.

The answer should be tied to the health output.

3. Classify the result

Use the anomaly fields as the first decision point.

A simple classification model can work well:

None
  No maintenance action by default.
  Log the result and continue monitoring.

Too many small files
  Candidate for compaction or OPTIMIZE.
  Check whether this table is written frequently in small batches.

Too many deleted rows
  Candidate for maintenance.
  Also review the upstream write, delete, or merge pattern.

No recent checkpoint
  Review checkpoint behavior and table activity.
  Decide whether maintenance should include checkpoint handling.

Invalid file statistics
  Investigate before routine optimization.
  Do not assume compaction is the only answer.

The exact action should depend on your workload, table size, freshness needs, and Fabric capacity behavior. The important change is that the action comes after diagnosis.

4. Decide, then act

The SQL analytics endpoint can diagnose table health. It is read-only, so it cannot perform the maintenance itself.

That means the runbook needs a handoff:

SQL health check identifies the condition
orchestration layer records the result
if action is needed, a Spark notebook or Lakehouse maintenance process runs the fix
a post-check confirms the result
the control table records what happened

This is the bridge that matters.

SQL sees the pain. Spark applies the fix. The pipeline connects the two.

A practical pipeline pattern

I would implement the first version with a small scheduled pipeline.

Not fancy. Just useful.

Step 1: Table list

Maintain a small configuration table:

CREATE TABLE dbo.LakehouseMaintenanceTargets
(
    TableName varchar(256) NOT NULL,
    Priority varchar(20) NOT NULL,
    Enabled bit NOT NULL,
    Owner varchar(100) NULL,
    Notes varchar(1000) NULL
);

Start with five to ten important tables. Add more only after the pattern works.

Step 2: Health check activity

For each enabled table, run the stored procedure and capture the result.

The exact mechanics will depend on how you orchestrate SQL activity in your Fabric environment, but the operating idea is the same: store the health output, not just the final action.

A simple log table might look like this:

CREATE TABLE dbo.LakehouseTableHealthLog
(
    RunId varchar(64) NOT NULL,
    CheckedAt datetime2 NOT NULL,
    TableName varchar(256) NOT NULL,
    PotentialAnomalyType int NULL,
    PotentialAnomalyDescription varchar(4000) NULL,
    Decision varchar(50) NOT NULL,
    ActionTaken varchar(100) NULL,
    Notes varchar(4000) NULL
);

The specific metric columns can be expanded after you inspect the procedure output in your environment.

Step 3: Decision rule

Keep the first decision rule conservative.

Example:

If no anomaly is detected:
  Log HEALTHY
  Skip Spark maintenance

If a known maintenance anomaly is detected:
  Log ACTION_REQUIRED
  Trigger Spark maintenance for that table

If the anomaly is unclear:
  Log REVIEW_REQUIRED
  Notify the owner instead of running automatic maintenance

That last branch is important.

Automation should not turn every warning into a compute job. Some signals need human review, especially early in the rollout.

Step 4: Spark maintenance

When the decision is ACTION_REQUIRED, run the maintenance action through Spark or the Lakehouse engine.

For tables with a clear small-file problem, that may mean running OPTIMIZE through a notebook.

I would keep this notebook parameterized:

# Parameters supplied by the pipeline
lakehouse_table = "schema.YourTable"

spark.sql(f"OPTIMIZE {lakehouse_table}")

Do not hard-code table names in five different notebooks. Pass the table name in, log the run, and keep the action traceable.

Step 5: Post-check

After maintenance, run the health check again.

This is the part teams often skip.

If a job consumed capacity, it should produce evidence that the table health improved or at least that the expected action completed.

The post-check does three useful things:

proves the maintenance job had an effect
catches cases where optimization did not solve the issue
gives you history for future threshold decisions

What I would measure

I would track both technical health and operational impact.

Technical health:

number of tables checked
number of anomalies detected
anomaly type frequency
maintenance actions triggered
post-check status
repeated anomalies on the same table

Operational impact:

query duration before and after maintenance for key SQL queries
Power BI refresh or report interaction patterns where available
Fabric capacity consumed by maintenance jobs
skipped maintenance runs because tables were healthy
incidents or user complaints tied to Lakehouse table performance

The skipped jobs are easy to overlook, but they matter.

If the health check prevents unnecessary optimization work, that is a real platform win.

Where this fits in a Fabric operating model

This feature belongs in the same conversation as monitoring, FinOps, data product ownership, and semantic model reliability.

A Lakehouse table that feeds several important reports is not just storage. It is part of the production data path.

For those tables, I would define:

owner
expected freshness
expected query pattern
health check frequency
maintenance decision rule
escalation path
last known health status

That sounds heavier than “run optimize nightly”, but it is actually lighter over time.

The team stops paying for blind maintenance and stops waiting for users to discover performance problems.

What I would avoid

I would avoid four traps.

Trap 1: Optimizing everything because you can

A health check is useful because it lets you avoid unnecessary work.

If every signal still leads to OPTIMIZE, the runbook has failed.

Trap 2: Treating the anomaly description as the whole diagnosis

The anomaly is a starting point. Pair it with workload knowledge.

A table written every few minutes will behave differently from a monthly snapshot table. A small-file pattern may be expected in one stage and unacceptable in another.

Trap 3: Ignoring the upstream write pattern

If a table keeps accumulating small files, compaction is only part of the answer.

Look upstream:

batch size
write frequency
partitioning choices
merge patterns
delete patterns
source system behavior

Maintenance cleans up the symptom. The upstream pattern often explains why it keeps coming back.

Trap 4: Not logging the decision

If the runbook cannot explain what it did, it is not a runbook. It is another black box.

Keep a small audit trail.

Table health, decision, action, result.

That is enough for a first version.

A simple first-week rollout

If I were adding this to a Fabric environment, I would do it in this order.

Day 1: Identify targets

Pick five important Lakehouse tables.

For each one, document the owner, main consumers, refresh pattern, and why it matters.

Day 2: Run manual checks

Run sp_get_table_health_metrics manually and review the output.

Do not automate yet. First understand what healthy and unhealthy look like in your own environment.

Day 3: Create the log table

Create the health log table and start storing results.

Even if the first version is manual, logging gives you history.

Day 4: Add a conservative pipeline

Automate the health check. Let the first version notify or log, not automatically optimize every table.

Day 5: Add one maintenance action

Choose one clear condition, such as a small-file anomaly on a high-value table, and trigger a parameterized Spark maintenance notebook.

Then run the post-check.

That is enough for a useful pilot.

The practical takeaway

This is a good Fabric update because it moves Lakehouse maintenance closer to how teams actually operate.

SQL users and Power BI users usually feel the performance issue first. Spark usually fixes the physical layout. sp_get_table_health_metrics gives teams a diagnostic bridge between those two worlds.

The feature is useful on its own. It becomes much more valuable when you turn it into a runbook:

pick critical tables
check health before maintenance
classify the anomaly
act only when needed
log the decision
run a post-check
adjust upstream write patterns when the same issue returns

That is the difference between scheduled guesswork and data engineering operations.

Good maintenance starts with evidence.

Source

Microsoft Fabric Updates Blog: Know before you optimize: Diagnose Lakehouse table health with a single T-SQL command (Generally Available)

Fabric Data Factory Makes Multi-Cloud Integration Practical. Here’s the Architecture Checklist.

Shai Karmani — Tue, 23 Jun 2026 22:30:56 +0000

Originally published at Data Ninja AI Lab.

Microsoft Fabric Data Factory just made a useful multi-cloud pattern generally available.

That matters because most companies do not live in one clean cloud. They have Azure, AWS, Google Cloud, SaaS platforms, vendor drops, legacy databases, and business-critical files that somehow need to become reliable analytics data.

The exciting part is not only that Fabric can connect across clouds. The practical win is that teams can now treat multi-cloud data movement as part of a governed Fabric architecture instead of another side integration project.

That is the angle I would focus on.

Use Fabric Data Factory to make multi-cloud integration easier, but build the ownership model around it from day one.

Microsoft’s update positions Fabric Data Factory as a way to make multi-cloud data integration and transformation easier. I agree with the direction. The question for data teams is how to turn that capability into a pattern they can operate safely.

The opportunity

Multi-cloud data work usually starts with a simple request:

bring S3 files into the analytics platform
combine Azure SQL data with Google Cloud Storage exports
pull SaaS data into OneLake
standardize vendor feeds before they hit Power BI
create one governed data product from systems that live in different places

The first pipeline is rarely the problem.

The problem appears when the fifth, tenth, or fiftieth pipeline shows up. Suddenly nobody is sure who owns the raw copy, where schema changes are detected, which capacity pays for the workload, what happens after a failed run, or whether downstream teams trust the curated output.

Fabric Data Factory helps with the integration layer. Architecture still has to handle the operating model.

That is where this update becomes useful for real teams.

The architecture pattern I would use

I would keep the pattern simple and explicit.

For every multi-cloud data flow, define six things before you scale it.

1. Landing zone

Decide where each source lands in Fabric.

I like separating the flow into clear zones:

raw copy from the external source
standardized data with basic type and naming cleanup
curated data that is ready for shared use
published data products used by reports, semantic models, AI agents, or downstream systems

This sounds obvious, but it prevents a lot of future pain.

If raw S3 files, curated tables, and report-ready outputs all land in the same place, every downstream consumer starts depending on internal pipeline details. That makes change management harder than it needs to be.

A clean landing model lets teams change ingestion logic without breaking everything that consumes the output.

2. Identity path

Multi-cloud does not remove identity design. It makes it more important.

For each source, document which identity accesses the source, which Fabric connection is used, where secrets or credentials are managed, and how access is reviewed.

The key question is simple:

Can you explain the identity path from the external source to the Fabric output?

If the answer is no, the pipeline is not ready for production.

This is especially important when the output feeds Power BI or an AI workflow. Users may only see a friendly report or agent response, but the data path behind it still needs to be governed.

3. Transform boundary

Fabric gives teams several places to transform data: Data Factory pipelines, Dataflows Gen2, notebooks, Lakehouse SQL, Warehouse SQL, and semantic model logic.

That flexibility is useful, but it can become messy fast.

Before building the pipeline, decide what belongs where.

My default rule:

use Data Factory for orchestration and movement
use Dataflows Gen2 for repeatable shaping where the team benefits from a visual transformation layer
use Warehouse or Lakehouse logic for shared data products and reusable business rules
use notebooks when code-based transformation is genuinely the better fit
keep report-specific logic out of the ingestion layer unless it is truly only for that report

The point is not to create a perfect rulebook. The point is to avoid spreading the same business logic across four different tools.

4. Cost model

Multi-cloud pipelines can hide cost in several places.

There is source-side cost, Fabric capacity usage, storage growth in OneLake, refresh frequency, retry behavior, and sometimes network movement. A pipeline that looks small in development can become expensive when it runs every 15 minutes across several regions or business units.

Before promoting a flow, define:

how often it runs
what triggers it
which Fabric capacity it uses
how retries behave
how much data is expected per run
who owns the cost if the workload grows

This is not finance theater. It is architecture hygiene.

If the business wants fresher data, the cost conversation should be attached to the value of that freshness.

5. Failure contract

A multi-cloud flow needs a failure contract.

Not a vague “monitor the pipeline” statement. A real contract.

For example:

what counts as failed, delayed, or degraded
which failures retry automatically
which failures require human review
who gets notified
where failed records are stored
how replay is handled
what downstream consumers see when the latest load is incomplete

This is where many analytics pipelines become fragile. They work until they do not, then the business finds out from a stale dashboard.

A failure contract turns the pipeline into something the team can operate.

6. Business output

Do not end the design at ingestion.

The real test is whether the pipeline produces a useful, trusted output:

a Warehouse table used by several teams
a Lakehouse data product
a Power BI semantic model
a Real-Time Dashboard
an AI agent context source
an operational extract for another system

If nobody owns the output, the pipeline is just movement.

The strongest Fabric Data Factory use cases will be the ones where a multi-cloud flow lands as a governed data product, not as another pile of copied files.

A low-risk rollout sequence

I would not start by trying to standardize every cloud source in the company.

Start with one valuable flow.

Pick a use case where the business value is clear and the source complexity is manageable. Then build the operating pattern around it.

A good first candidate has:

one or two external sources
a clear business owner
a visible reporting or operational outcome
enough pain that the current process is worth replacing
limited blast radius if the first version needs adjustment

Examples:

daily vendor files from cloud storage into a curated Power BI model
SaaS operational exports into a Fabric Warehouse table
cross-cloud product usage data into OneLake for customer analytics
finance or planning extracts into a governed reporting layer

Build the first pipeline. Document the contract. Prove the output. Then reuse the pattern.

That is how multi-cloud architecture becomes repeatable instead of heroic.

What I would avoid

I would avoid three traps.

First, do not let every team create its own connector pattern. You will get speed for a month and cleanup work for a year.

Second, do not treat OneLake as a dumping ground. Landing everything is not the same as governing anything.

Third, do not move business rules into whichever tool the first developer prefers. Decide where shared logic belongs and keep it reviewable.

Fabric makes the technical path easier. That should give teams more room to design the operating model, not less.

The practical takeaway

The GA update is good news for teams building analytics across messy real-world estates.

Fabric Data Factory can make multi-cloud integration more approachable. The win is bigger when teams pair it with a clear architecture checklist:

define the landing zone
trace the identity path
choose the transform boundary
model the cost
write the failure contract
publish a trusted business output

That is the version of multi-cloud data architecture I want to see more often.

Not a collection of connectors.

A repeatable Fabric pattern the team can actually operate.

Source

Microsoft Fabric Updates Blog: Multi-cloud data architecture patterns using Fabric Data Factory (Generally Available)