DEV Community: Shai Karmani

Fabric Warehouse Brings AI Enrichment Into T-SQL. Here’s the Practical Guide.

Shai Karmani — Sun, 28 Jun 2026 00:23:05 +0000

Originally published at Data Ninja AI Lab.

Fabric Data Warehouse now has preview AI functions that let you classify, summarize, translate, extract, and improve text directly from T-SQL.

That is a bigger shift than it first looks.

For years, a lot of text enrichment work has lived outside the warehouse. It gets handled in notebooks, one-off Python scripts, Power Query steps, spreadsheets, application code, or manual cleanup queues. Sometimes that is the right place. Often, it creates another disconnected transformation layer that nobody governs properly.

The interesting part of these functions is not that Fabric can call AI from SQL. The interesting part is that common text intelligence tasks can now sit closer to governed warehouse workflows.

That gives data teams a practical option:

enrich support tickets before they reach a semantic model
classify messy feedback into controlled categories
extract structured fields from free text
translate multilingual comments for analysis
summarize long operational notes
clean user-entered text before reporting
generate controlled response drafts from trusted data

Used well, this can reduce friction between AI experiments and production analytics.

Used badly, it can bury expensive, non-deterministic logic inside report queries.

The difference is architecture.

What Microsoft added

Microsoft documents seven preview AI functions for Fabric Data Warehouse and the SQL analytics endpoint.

Function	What it does	Practical use
`AI_ANALYZE_SENTIMENT(text)`	Returns `positive`, `negative`, `mixed`, or `neutral`	Review analysis, support triage, survey feedback
`AI_CLASSIFY(text, class1, class2, ...)`	Classifies text into labels you provide	Ticket routing, complaint categories, product issue groups
`AI_EXTRACT(text, field1, field2, ...)`	Extracts fields as JSON	Pulling problem, date, sentiment, location, or entity values from text
`AI_SUMMARIZE(text)`	Produces a shorter summary	Condensing long notes for analysts or dashboards
`AI_GENERATE_RESPONSE(prompt, data)`	Generates a response from a prompt and optional data	Response drafts, internal summaries, controlled explanation text
`AI_TRANSLATE(text, lang_code)`	Translates text into supported languages	Multilingual support and feedback analysis
`AI_FIX_GRAMMAR(text)`	Corrects grammar in text	Cleaning user-entered comments or notes

These are not replacements for data modeling, governance, or review. They are transformation tools.

That distinction matters.

A simple example

Imagine a support_cases table with a free-text case_notes column.

A useful enrichment table might include sentiment, a business category, a short summary, and structured fields extracted from the notes.

CREATE TABLE curated.support_case_ai_enrichment AS
SELECT
    case_id,
    case_notes,
    AI_ANALYZE_SENTIMENT(case_notes) AS sentiment,
    AI_CLASSIFY(
        case_notes,
        'billing',
        'delivery',
        'technical issue',
        'account access',
        'other'
    ) AS case_category,
    AI_SUMMARIZE(case_notes) AS case_summary,
    AI_EXTRACT(
        case_notes,
        'problem',
        'product',
        'urgency',
        'time_reported'
    ) AS extracted_json
FROM staging.support_cases;

That output can then feed Power BI, a semantic model, an operations dashboard, or a downstream process.

But I would not put this directly inside a report-facing query that runs every time a user opens a dashboard.

Microsoft’s documentation calls out two practical constraints:

AI functions can return NULL if the model cannot process the text.
Typical processing speed is around 20 to 100 rows per second.

That points to the correct pattern: precompute and materialize repeated transformations.

The function guide

1. Use `AI_ANALYZE_SENTIMENT` for directional signals

Sentiment is useful when you need a rough business signal from text.

Examples:

customer review sentiment
employee survey comments
support ticket tone
partner feedback
product complaint notes

Good use:

SELECT
    review_id,
    AI_ANALYZE_SENTIMENT(review_text) AS review_sentiment
FROM staging.customer_reviews;

What I would not do: treat sentiment as absolute truth. It should support analysis, not replace review for high-impact cases.

2. Use `AI_CLASSIFY` when you control the categories

AI_CLASSIFY is strongest when the business already knows the target categories.

Example:

SELECT
    case_id,
    AI_CLASSIFY(
        case_notes,
        'billing',
        'service',
        'technical issue',
        'contract',
        'other'
    ) AS case_type
FROM staging.support_cases;

The governance point is simple: the labels are part of the data contract. If the business changes the categories, the transformation logic changed too.

Track that change like you would track a schema change.

3. Use `AI_EXTRACT` to turn text into structured fields

This is the most interesting function for analytics engineering.

Free text often contains useful structure, but parsing it with regular expressions gets brittle fast. AI_EXTRACT lets you ask for fields and returns JSON.

Example:

SELECT
    case_id,
    AI_EXTRACT(
        case_notes,
        'problem',
        'affected_system',
        'urgency'
    ) AS extracted_case_details
FROM staging.support_cases;

For reporting, I would normally parse the JSON into typed columns in a curated table.

SELECT
    s.case_id,
    j.problem,
    j.affected_system,
    j.urgency
FROM staging.support_cases s
CROSS APPLY OPENJSON(
    AI_EXTRACT(s.case_notes, 'problem', 'affected_system', 'urgency')
)
WITH (
    problem VARCHAR(1000),
    affected_system VARCHAR(200),
    urgency VARCHAR(100)
) AS j;

This is where validation matters. Sample the output. Review wrong extractions. Keep the original text.

4. Use `AI_SUMMARIZE` for readability, not evidence

Summaries are useful when analysts need context without reading a full comment field.

Example:

SELECT
    incident_id,
    AI_SUMMARIZE(incident_notes) AS incident_summary
FROM staging.incidents;

The summary should not become the only version of the record. Keep the original text beside it or one click away.

A summary is a reading aid. It is not the source of truth.

5. Use `AI_TRANSLATE` when language blocks analysis

AI_TRANSLATE can help standardize multilingual text for analysis.

Example:

SELECT
    feedback_id,
    feedback_text,
    AI_TRANSLATE(feedback_text, 'en') AS feedback_text_en
FROM staging.customer_feedback;

Microsoft lists supported language codes including en, de, fr, it, es, el, pl, sv, fi, and cs.

For global reporting, this can make feedback analysis easier. Still, translation can change nuance, especially in complaints, legal text, or regulated workflows. Keep the original language value.

6. Use `AI_FIX_GRAMMAR` carefully

Grammar correction is useful for presentation and readability.

Example:

UPDATE curated.customer_feedback
SET cleaned_comment = ISNULL(
    AI_FIX_GRAMMAR(raw_comment),
    raw_comment
);

The ISNULL pattern matters. Microsoft’s docs note that AI functions can return NULL, so avoid overwriting useful source text with a blank result.

I would use this for cleaned display fields, not for replacing the original source record.

7. Use `AI_GENERATE_RESPONSE` with the most discipline

This function can generate text from a prompt and data.

Example:

SELECT
    case_id,
    AI_GENERATE_RESPONSE(
        'Write a concise internal summary for a support manager:',
        case_notes
    ) AS manager_summary
FROM staging.support_cases;

This is powerful, but it is also where teams need the most control.

If generated text will be sent to customers, used in decisions, or shown in operational workflows, add human review, prompt ownership, audit fields, and clear usage rules.

Generated text should not quietly become an automated business action.

The production pattern I would use

I would treat each AI function output as a governed transformation.

That means:

Profile the source text first.
Choose the function based on the business task.
Materialize the output in a staging or enrichment table.
Keep the original text.
Add fallback behavior for NULL results.
Validate a sample of outputs.
Track prompt or label changes.
Monitor refresh time and cost.
Separate experimental outputs from certified reporting fields.

The practical table design might include:

CREATE TABLE curated.support_case_ai_enrichment
(
    case_id BIGINT,
    source_text_hash VARCHAR(200),
    ai_function_used VARCHAR(100),
    ai_labels_or_prompt VARCHAR(4000),
    ai_output VARCHAR(4000),
    ai_output_status VARCHAR(50),
    validated_flag BIT,
    validation_sample_group VARCHAR(100),
    created_at DATETIME2(6),
    created_by_pipeline VARCHAR(200)
);

This may look heavy for a demo. It is not heavy for production.

If the output will influence reporting, routing, prioritization, or an AI agent, someone needs to know where it came from and how it was produced.

Where this fits in a Fabric architecture

The strongest use cases are not generic AI demos.

They are specific enrichment steps inside a real data workflow:

classify customer feedback before semantic model refresh
extract product issue fields from support notes
summarize long incident comments for operations dashboards
translate multilingual survey comments for regional comparison
clean messy user-entered text in curated reporting tables
generate internal response drafts for review queues

That is the sweet spot.

The warehouse becomes a controlled place to enrich text, not just store it.

What I would avoid

I would avoid four patterns:

Calling AI functions repeatedly in interactive report queries.
Treating model output as deterministic truth.
Replacing original text with AI-cleaned text.
Using generated responses without review, ownership, and audit.

Preview features are for learning and controlled adoption. The right move is to test the pattern, measure the cost, validate outputs, and decide where it belongs in the pipeline.

Final take

This is a useful direction for Fabric.

Not because it makes every warehouse query smarter by default. That would be the wrong framing.

It is useful because it gives data teams a practical way to move common text enrichment closer to the governed data layer.

If your team already has free-text feedback, support cases, notes, reviews, comments, or multilingual text sitting in the warehouse, these functions are worth testing.

Just test them like production data transformations, not like magic buttons.

Start with one table, one business use case, one materialized output, and one validation sample.

That is enough to learn quickly without turning AI-in-SQL into another unmanaged layer.

Fabric Lakehouse Health Checks Make Optimization Practical. Here’s the Runbook.

Shai Karmani — Wed, 24 Jun 2026 22:50:45 +0000

Originally published at Data Ninja AI Lab.

Microsoft Fabric just added a small feature that can change how teams maintain Lakehouse tables.

The new sp_get_table_health_metrics stored procedure gives SQL analytics endpoint users a T-SQL way to inspect Lakehouse table health before deciding whether Spark maintenance is needed.

That sounds narrow. It is not.

For teams serving Power BI, SQL users, downstream data products, or AI workflows from Lakehouse tables, this closes an annoying operational gap: the place where users feel the slowdown is often SQL, but the maintenance action usually happens in Spark.

Until now, a lot of teams handled that gap with guesswork.

Run OPTIMIZE every night. Compact everything on a schedule. Wait until dashboards get slow. Open a notebook. Inspect Delta files. Ask support. Hope the maintenance job was worth the compute.

The better pattern is simple:

Check table health first. Optimize only when the evidence says to.

That is the practical win here.

The problem this solves

Lakehouse tables can look fine logically while becoming less efficient physically.

The schema is still valid. The row counts still make sense. The reports still refresh. But over time the physical layout can drift:

too many small files
too many deleted rows
stale or missing checkpoints
uneven row distribution
invalid or weak file statistics
fragmented table layout after frequent writes, deletes, or merges

Users usually experience this as slower SQL queries or lagging Power BI reports.

Data engineers experience it as a vague support problem:

The dashboard is slow. Can you check the Lakehouse?

That is a bad starting point. It pushes the team into reactive troubleshooting instead of evidence-based maintenance.

sp_get_table_health_metrics gives the SQL side a diagnostic step. It does not replace Spark maintenance, but it gives teams a better way to decide when Spark maintenance is actually justified.

What the stored procedure gives you

Microsoft’s announcement describes a built-in stored procedure for the SQL analytics endpoint that returns table health signals for Lakehouse tables.

The useful part is not just one metric. It is the mix of signals:

PotentialAnomalyType
PotentialAnomalyDescription
snapshot and checkpoint versions
physical row counts
deleted row counts
file size distribution
row count distribution
deleted row distribution

That gives you a much better conversation than “the table feels slow”.

You can ask more specific questions:

Is this a small-file problem?
Are deleted rows accumulating?
Is the table missing a recent checkpoint?
Are file statistics valid?
Is this table actually healthy and the issue is somewhere else?

That last point matters.

A health check can save capacity by proving that a maintenance job is not needed.

The runbook I would use

I would not treat this as a one-off troubleshooting command. I would turn it into a small operational runbook.

1. Start with the critical tables

Do not begin by checking every table in the Lakehouse.

Start with tables that actually matter:

tables behind important Power BI semantic models
tables queried heavily through the SQL analytics endpoint
fact tables with frequent incremental writes
tables touched by merge, delete, or update patterns
tables used as context for AI agents or downstream apps
tables with known performance complaints

This keeps the first version focused.

A table nobody queries does not need the same operational attention as the table behind the CFO dashboard.

2. Run the health check before maintenance

The basic pattern is straightforward.

EXEC sp_get_table_health_metrics @table_name = 'schema.YourTable';

In a real runbook, I would capture the output instead of only looking at it manually.

For example, create a control table that stores:

run timestamp
workspace or environment
Lakehouse name
table name
anomaly type
anomaly description
selected file distribution metrics
maintenance decision
action taken
post-check result

The goal is not to create bureaucracy. The goal is to make maintenance reviewable.

If someone asks why a table was optimized yesterday, the answer should not be “because the schedule said so”.

The answer should be tied to the health output.

3. Classify the result

Use the anomaly fields as the first decision point.

A simple classification model can work well:

None
  No maintenance action by default.
  Log the result and continue monitoring.

Too many small files
  Candidate for compaction or OPTIMIZE.
  Check whether this table is written frequently in small batches.

Too many deleted rows
  Candidate for maintenance.
  Also review the upstream write, delete, or merge pattern.

No recent checkpoint
  Review checkpoint behavior and table activity.
  Decide whether maintenance should include checkpoint handling.

Invalid file statistics
  Investigate before routine optimization.
  Do not assume compaction is the only answer.

The exact action should depend on your workload, table size, freshness needs, and Fabric capacity behavior. The important change is that the action comes after diagnosis.

4. Decide, then act

The SQL analytics endpoint can diagnose table health. It is read-only, so it cannot perform the maintenance itself.

That means the runbook needs a handoff:

SQL health check identifies the condition
orchestration layer records the result
if action is needed, a Spark notebook or Lakehouse maintenance process runs the fix
a post-check confirms the result
the control table records what happened

This is the bridge that matters.

SQL sees the pain. Spark applies the fix. The pipeline connects the two.

A practical pipeline pattern

I would implement the first version with a small scheduled pipeline.

Not fancy. Just useful.

Step 1: Table list

Maintain a small configuration table:

CREATE TABLE dbo.LakehouseMaintenanceTargets
(
    TableName varchar(256) NOT NULL,
    Priority varchar(20) NOT NULL,
    Enabled bit NOT NULL,
    Owner varchar(100) NULL,
    Notes varchar(1000) NULL
);

Start with five to ten important tables. Add more only after the pattern works.

Step 2: Health check activity

For each enabled table, run the stored procedure and capture the result.

The exact mechanics will depend on how you orchestrate SQL activity in your Fabric environment, but the operating idea is the same: store the health output, not just the final action.

A simple log table might look like this:

CREATE TABLE dbo.LakehouseTableHealthLog
(
    RunId varchar(64) NOT NULL,
    CheckedAt datetime2 NOT NULL,
    TableName varchar(256) NOT NULL,
    PotentialAnomalyType int NULL,
    PotentialAnomalyDescription varchar(4000) NULL,
    Decision varchar(50) NOT NULL,
    ActionTaken varchar(100) NULL,
    Notes varchar(4000) NULL
);

The specific metric columns can be expanded after you inspect the procedure output in your environment.

Step 3: Decision rule

Keep the first decision rule conservative.

Example:

If no anomaly is detected:
  Log HEALTHY
  Skip Spark maintenance

If a known maintenance anomaly is detected:
  Log ACTION_REQUIRED
  Trigger Spark maintenance for that table

If the anomaly is unclear:
  Log REVIEW_REQUIRED
  Notify the owner instead of running automatic maintenance

That last branch is important.

Automation should not turn every warning into a compute job. Some signals need human review, especially early in the rollout.

Step 4: Spark maintenance

When the decision is ACTION_REQUIRED, run the maintenance action through Spark or the Lakehouse engine.

For tables with a clear small-file problem, that may mean running OPTIMIZE through a notebook.

I would keep this notebook parameterized:

# Parameters supplied by the pipeline
lakehouse_table = "schema.YourTable"

spark.sql(f"OPTIMIZE {lakehouse_table}")

Do not hard-code table names in five different notebooks. Pass the table name in, log the run, and keep the action traceable.

Step 5: Post-check

After maintenance, run the health check again.

This is the part teams often skip.

If a job consumed capacity, it should produce evidence that the table health improved or at least that the expected action completed.

The post-check does three useful things:

proves the maintenance job had an effect
catches cases where optimization did not solve the issue
gives you history for future threshold decisions

What I would measure

I would track both technical health and operational impact.

Technical health:

number of tables checked
number of anomalies detected
anomaly type frequency
maintenance actions triggered
post-check status
repeated anomalies on the same table

Operational impact:

query duration before and after maintenance for key SQL queries
Power BI refresh or report interaction patterns where available
Fabric capacity consumed by maintenance jobs
skipped maintenance runs because tables were healthy
incidents or user complaints tied to Lakehouse table performance

The skipped jobs are easy to overlook, but they matter.

If the health check prevents unnecessary optimization work, that is a real platform win.

Where this fits in a Fabric operating model

This feature belongs in the same conversation as monitoring, FinOps, data product ownership, and semantic model reliability.

A Lakehouse table that feeds several important reports is not just storage. It is part of the production data path.

For those tables, I would define:

owner
expected freshness
expected query pattern
health check frequency
maintenance decision rule
escalation path
last known health status

That sounds heavier than “run optimize nightly”, but it is actually lighter over time.

The team stops paying for blind maintenance and stops waiting for users to discover performance problems.

What I would avoid

I would avoid four traps.

Trap 1: Optimizing everything because you can

A health check is useful because it lets you avoid unnecessary work.

If every signal still leads to OPTIMIZE, the runbook has failed.

Trap 2: Treating the anomaly description as the whole diagnosis

The anomaly is a starting point. Pair it with workload knowledge.

A table written every few minutes will behave differently from a monthly snapshot table. A small-file pattern may be expected in one stage and unacceptable in another.

Trap 3: Ignoring the upstream write pattern

If a table keeps accumulating small files, compaction is only part of the answer.

Look upstream:

batch size
write frequency
partitioning choices
merge patterns
delete patterns
source system behavior

Maintenance cleans up the symptom. The upstream pattern often explains why it keeps coming back.

Trap 4: Not logging the decision

If the runbook cannot explain what it did, it is not a runbook. It is another black box.

Keep a small audit trail.

Table health, decision, action, result.

That is enough for a first version.

A simple first-week rollout

If I were adding this to a Fabric environment, I would do it in this order.

Day 1: Identify targets

Pick five important Lakehouse tables.

For each one, document the owner, main consumers, refresh pattern, and why it matters.

Day 2: Run manual checks

Run sp_get_table_health_metrics manually and review the output.

Do not automate yet. First understand what healthy and unhealthy look like in your own environment.

Day 3: Create the log table

Create the health log table and start storing results.

Even if the first version is manual, logging gives you history.

Day 4: Add a conservative pipeline

Automate the health check. Let the first version notify or log, not automatically optimize every table.

Day 5: Add one maintenance action

Choose one clear condition, such as a small-file anomaly on a high-value table, and trigger a parameterized Spark maintenance notebook.

Then run the post-check.

That is enough for a useful pilot.

The practical takeaway

This is a good Fabric update because it moves Lakehouse maintenance closer to how teams actually operate.

SQL users and Power BI users usually feel the performance issue first. Spark usually fixes the physical layout. sp_get_table_health_metrics gives teams a diagnostic bridge between those two worlds.

The feature is useful on its own. It becomes much more valuable when you turn it into a runbook:

pick critical tables
check health before maintenance
classify the anomaly
act only when needed
log the decision
run a post-check
adjust upstream write patterns when the same issue returns

That is the difference between scheduled guesswork and data engineering operations.

Good maintenance starts with evidence.

Source

Microsoft Fabric Updates Blog: Know before you optimize: Diagnose Lakehouse table health with a single T-SQL command (Generally Available)

Fabric Data Factory Makes Multi-Cloud Integration Practical. Here’s the Architecture Checklist.

Shai Karmani — Tue, 23 Jun 2026 22:30:56 +0000

Originally published at Data Ninja AI Lab.

Microsoft Fabric Data Factory just made a useful multi-cloud pattern generally available.

That matters because most companies do not live in one clean cloud. They have Azure, AWS, Google Cloud, SaaS platforms, vendor drops, legacy databases, and business-critical files that somehow need to become reliable analytics data.

The exciting part is not only that Fabric can connect across clouds. The practical win is that teams can now treat multi-cloud data movement as part of a governed Fabric architecture instead of another side integration project.

That is the angle I would focus on.

Use Fabric Data Factory to make multi-cloud integration easier, but build the ownership model around it from day one.

Microsoft’s update positions Fabric Data Factory as a way to make multi-cloud data integration and transformation easier. I agree with the direction. The question for data teams is how to turn that capability into a pattern they can operate safely.

The opportunity

Multi-cloud data work usually starts with a simple request:

bring S3 files into the analytics platform
combine Azure SQL data with Google Cloud Storage exports
pull SaaS data into OneLake
standardize vendor feeds before they hit Power BI
create one governed data product from systems that live in different places

The first pipeline is rarely the problem.

The problem appears when the fifth, tenth, or fiftieth pipeline shows up. Suddenly nobody is sure who owns the raw copy, where schema changes are detected, which capacity pays for the workload, what happens after a failed run, or whether downstream teams trust the curated output.

Fabric Data Factory helps with the integration layer. Architecture still has to handle the operating model.

That is where this update becomes useful for real teams.

The architecture pattern I would use

I would keep the pattern simple and explicit.

For every multi-cloud data flow, define six things before you scale it.

1. Landing zone

Decide where each source lands in Fabric.

I like separating the flow into clear zones:

raw copy from the external source
standardized data with basic type and naming cleanup
curated data that is ready for shared use
published data products used by reports, semantic models, AI agents, or downstream systems

This sounds obvious, but it prevents a lot of future pain.

If raw S3 files, curated tables, and report-ready outputs all land in the same place, every downstream consumer starts depending on internal pipeline details. That makes change management harder than it needs to be.

A clean landing model lets teams change ingestion logic without breaking everything that consumes the output.

2. Identity path

Multi-cloud does not remove identity design. It makes it more important.

For each source, document which identity accesses the source, which Fabric connection is used, where secrets or credentials are managed, and how access is reviewed.

The key question is simple:

Can you explain the identity path from the external source to the Fabric output?

If the answer is no, the pipeline is not ready for production.

This is especially important when the output feeds Power BI or an AI workflow. Users may only see a friendly report or agent response, but the data path behind it still needs to be governed.

3. Transform boundary

Fabric gives teams several places to transform data: Data Factory pipelines, Dataflows Gen2, notebooks, Lakehouse SQL, Warehouse SQL, and semantic model logic.

That flexibility is useful, but it can become messy fast.

Before building the pipeline, decide what belongs where.

My default rule:

use Data Factory for orchestration and movement
use Dataflows Gen2 for repeatable shaping where the team benefits from a visual transformation layer
use Warehouse or Lakehouse logic for shared data products and reusable business rules
use notebooks when code-based transformation is genuinely the better fit
keep report-specific logic out of the ingestion layer unless it is truly only for that report

The point is not to create a perfect rulebook. The point is to avoid spreading the same business logic across four different tools.

4. Cost model

Multi-cloud pipelines can hide cost in several places.

There is source-side cost, Fabric capacity usage, storage growth in OneLake, refresh frequency, retry behavior, and sometimes network movement. A pipeline that looks small in development can become expensive when it runs every 15 minutes across several regions or business units.

Before promoting a flow, define:

how often it runs
what triggers it
which Fabric capacity it uses
how retries behave
how much data is expected per run
who owns the cost if the workload grows

This is not finance theater. It is architecture hygiene.

If the business wants fresher data, the cost conversation should be attached to the value of that freshness.

5. Failure contract

A multi-cloud flow needs a failure contract.

Not a vague “monitor the pipeline” statement. A real contract.

For example:

what counts as failed, delayed, or degraded
which failures retry automatically
which failures require human review
who gets notified
where failed records are stored
how replay is handled
what downstream consumers see when the latest load is incomplete

This is where many analytics pipelines become fragile. They work until they do not, then the business finds out from a stale dashboard.

A failure contract turns the pipeline into something the team can operate.

6. Business output

Do not end the design at ingestion.

The real test is whether the pipeline produces a useful, trusted output:

a Warehouse table used by several teams
a Lakehouse data product
a Power BI semantic model
a Real-Time Dashboard
an AI agent context source
an operational extract for another system

If nobody owns the output, the pipeline is just movement.

The strongest Fabric Data Factory use cases will be the ones where a multi-cloud flow lands as a governed data product, not as another pile of copied files.

A low-risk rollout sequence

I would not start by trying to standardize every cloud source in the company.

Start with one valuable flow.

Pick a use case where the business value is clear and the source complexity is manageable. Then build the operating pattern around it.

A good first candidate has:

one or two external sources
a clear business owner
a visible reporting or operational outcome
enough pain that the current process is worth replacing
limited blast radius if the first version needs adjustment

Examples:

daily vendor files from cloud storage into a curated Power BI model
SaaS operational exports into a Fabric Warehouse table
cross-cloud product usage data into OneLake for customer analytics
finance or planning extracts into a governed reporting layer

Build the first pipeline. Document the contract. Prove the output. Then reuse the pattern.

That is how multi-cloud architecture becomes repeatable instead of heroic.

What I would avoid

I would avoid three traps.

First, do not let every team create its own connector pattern. You will get speed for a month and cleanup work for a year.

Second, do not treat OneLake as a dumping ground. Landing everything is not the same as governing anything.

Third, do not move business rules into whichever tool the first developer prefers. Decide where shared logic belongs and keep it reviewable.

Fabric makes the technical path easier. That should give teams more room to design the operating model, not less.

The practical takeaway

The GA update is good news for teams building analytics across messy real-world estates.

Fabric Data Factory can make multi-cloud integration more approachable. The win is bigger when teams pair it with a clear architecture checklist:

define the landing zone
trace the identity path
choose the transform boundary
model the cost
write the failure contract
publish a trusted business output

That is the version of multi-cloud data architecture I want to see more often.

Not a collection of connectors.

A repeatable Fabric pattern the team can actually operate.

Source

Microsoft Fabric Updates Blog: Multi-cloud data architecture patterns using Fabric Data Factory (Generally Available)

Fabric Warehouse Can Clean Messy Text Now. Here’s the Data Quality Playbook.

Shai Karmani — Fri, 19 Jun 2026 02:02:47 +0000

Originally published at Data Ninja AI Lab.

Microsoft Fabric Data Warehouse just got a set of preview string-processing capabilities that sound small until you think about where data quality work usually gets stuck.

Approximate string matching. Modern string functions. ANSI-style string concatenation. Better ways to validate and compare messy text directly in T-SQL.

The practical win is that a painful class of data quality work can move closer to the warehouse, where the data already lives and where the review trail can be governed.

I like this update because it solves a real problem that shows up in almost every data estate:

the same customer appears under five slightly different names
product descriptions arrive with inconsistent punctuation
supplier records use different naming conventions
free-text fields hide useful matching signals
email, phone, SKU, and reference fields need validation before reporting
deduplication logic lives in a notebook, spreadsheet, or one-off script nobody owns

The opportunity is simple: use these functions to build a repeatable text quality workflow, not a clever SQL trick.

What changed

Microsoft’s Fabric update describes new preview capabilities in Fabric Data Warehouse for approximate string matching, modern string-processing functions, and string operators. The stated goal is to make everyday string processing easier in T-SQL, improve query clarity, and improve portability.

That includes EDIT_DISTANCE, EDIT_DISTANCE_SIMILARITY, JARO_WINKLER_DISTANCE, and JARO_WINKLER_SIMILARITY for approximate matching. It also includes ||, ||=, and UNISTR for clearer string composition and Unicode handling.

For teams working with messy warehouse data, this matters because string cleanup is often treated as side work. It happens in Power Query, in a notebook, in a source-system export, in a temporary SQL script, or manually in Excel.

That works once. It does not scale as a governed data process.

A better model is to make text quality part of the warehouse pipeline.

The pattern I would use

I would not start by throwing fuzzy matching at every column.

That creates false positives fast.

Start with a controlled workflow.

The pattern is:

Profile the raw text.
Normalize the obvious variation.
Compose readable clean keys and audit labels.
Score likely matches.
Review uncertain cases.
Write trusted output with audit fields.

That sounds heavier than just writing a query, but it is what makes the result usable by business teams.

If a report says two customer records were merged, someone needs to know why.

If a pipeline says a supplier name is invalid, someone needs to know which rule failed.

If an executive dashboard depends on a cleaned product hierarchy, someone needs to know whether that hierarchy came from deterministic rules, similarity scoring, or manual approval.

Step 1: Profile the messy field first

Before cleaning a text field, profile it.

For example, if you are working with customer names, start with basic shape checks:

SELECT
    COUNT(*) AS row_count,
    COUNT(NULLIF(TRIM(CustomerName), '')) AS populated_count,
    MIN(LEN(CustomerName)) AS min_length,
    MAX(LEN(CustomerName)) AS max_length,
    COUNT(DISTINCT CustomerName) AS distinct_raw_names,
    COUNT(DISTINCT LOWER(TRIM(CustomerName))) AS distinct_normalized_names
FROM staging.Customer;

This tells you whether the issue is mostly blanks, inconsistent casing, punctuation, real duplication, or something stranger.

Do not skip this step. The right matching rule depends on the kind of mess you have.

Step 2: Normalize before you match

Approximate matching works better after basic normalization.

If you compare raw strings, you waste effort on differences that do not matter:

leading and trailing spaces
casing
dots and commas
legal suffixes like Ltd, Inc, Corp, LLC
repeated whitespace
known spelling variants

A simple normalized projection might look like this:

WITH normalized AS (
    SELECT
        CustomerId,
        CustomerName,
        LOWER(
            REPLACE(
                REPLACE(
                    REPLACE(TRIM(CustomerName), '.', ''),
                ',', ''),
            ' inc', '')
        ) AS CustomerNameClean
    FROM staging.Customer
)
SELECT *
FROM normalized;

This is not perfect. It is not meant to be.

The goal is to remove noise before using more expensive or less deterministic matching.

Also, keep the raw value. Always.

A cleaned value without the original value is hard to audit later.

Step 3: Compose clean keys and audit labels clearly

String quality work usually produces more than one output column.

You often need a normalized match key, a display value, a failure reason, a rule label, and a review note. This is where the new string operators matter. They make the SQL easier to read, especially when you are composing values that will be inspected by another human.

Example pattern:

SELECT
    CustomerId,
    CustomerName,
    LOWER(TRIM(CustomerName)) AS CustomerNameClean,
    SourceSystem || ':' || CAST(CustomerId AS VARCHAR(50)) AS SourceRecordKey,
    'customer_name_normalization_v1' AS QualityRule
FROM staging.Customer;

The || operator is not the headline feature, but readability matters in data quality code. If a steward or another engineer needs to review the logic, clear composition beats a long chain of string handling that hides intent.

UNISTR is useful when your cleanup or labeling work needs explicit Unicode characters. That matters in international data, symbols, and cases where the literal value should be reviewable in SQL instead of being hidden in application code.

Step 4: Use similarity scoring for likely duplicates

This is where approximate string matching becomes useful.

EDIT_DISTANCE can help identify records that are close enough to review.

Example:

WITH candidates AS (
    SELECT
        a.CustomerId AS CustomerIdA,
        b.CustomerId AS CustomerIdB,
        a.CustomerNameClean AS NameA,
        b.CustomerNameClean AS NameB,
        EDIT_DISTANCE(a.CustomerNameClean, b.CustomerNameClean, 4) AS NameDistance
    FROM curated.CustomerNameNormalized a
    JOIN curated.CustomerNameNormalized b
        ON a.CustomerId < b.CustomerId
       AND LEFT(a.CustomerNameClean, 1) = LEFT(b.CustomerNameClean, 1)
)
SELECT *
FROM candidates
WHERE NameDistance BETWEEN 1 AND 4
ORDER BY NameDistance;

A few practical points matter here.

First, avoid comparing every row to every other row unless the dataset is tiny. Use blocking rules, such as same first character, same country, same postal prefix, same source system group, or same cleaned token.

Second, thresholds should be field-specific. A distance of 2 may be meaningful for a short product code and meaningless for a long company name.

Third, do not auto-merge everything with a close score. Similarity is a signal. It is not a business decision.

Step 5: Create a review queue for uncertain matches

The biggest mistake in fuzzy matching is pretending the function knows the business meaning.

It does not.

It can tell you two strings are close. It cannot tell you whether two customers should be merged, whether two suppliers are legally the same entity, or whether a product alias is approved.

So I would create a review table.

CREATE TABLE dq.CustomerMatchReview
(
    ReviewId INT,
    CustomerIdA INT,
    CustomerIdB INT,
    NameA VARCHAR(300),
    NameB VARCHAR(300),
    MatchScore INT,
    MatchRule VARCHAR(100),
    ReviewStatus VARCHAR(30),
    ReviewedBy VARCHAR(200),
    ReviewedAt DATETIME2(6),
    ReviewerNotes VARCHAR(1000)
);

Then route borderline matches into that table:

INSERT INTO dq.CustomerMatchReview
(
    ReviewId,
    CustomerIdA,
    CustomerIdB,
    NameA,
    NameB,
    MatchScore,
    MatchRule,
    ReviewStatus
)
SELECT
    ROW_NUMBER() OVER (ORDER BY NameDistance, CustomerIdA, CustomerIdB) AS ReviewId,
    CustomerIdA,
    CustomerIdB,
    NameA,
    NameB,
    NameDistance,
    'customer_name_edit_distance_v1' AS MatchRule,
    'Pending' AS ReviewStatus
FROM dq.CustomerNameMatchCandidates
WHERE NameDistance BETWEEN 2 AND 4;

That one table changes the conversation.

Now the quality process has ownership, status, and history. The warehouse produces cleaned data with evidence behind it.

Step 6: Write the trusted result with audit fields

The final clean output should not be a mystery.

A practical trusted dimension or quality output should include fields like:

raw value
normalized value
approved value
quality status
match rule
match score
reviewer
reviewed timestamp
source system
rule version

That gives report owners and downstream users a way to understand what happened.

Example:

SELECT
    c.CustomerId,
    c.CustomerName AS RawCustomerName,
    n.CustomerNameClean,
    COALESCE(r.ApprovedCustomerName, n.CustomerNameClean) AS TrustedCustomerName,
    CASE
        WHEN r.ReviewStatus = 'Approved' THEN 'Reviewed match'
        WHEN n.CustomerNameClean IS NULL THEN 'Missing name'
        ELSE 'Rule-based clean'
    END AS CustomerNameQualityStatus,
    r.MatchRule,
    r.MatchScore,
    r.ReviewedBy,
    r.ReviewedAt
FROM staging.Customer c
LEFT JOIN curated.CustomerNameNormalized n
    ON c.CustomerId = n.CustomerId
LEFT JOIN dq.CustomerNameApprovedMatch r
    ON c.CustomerId = r.CustomerId;

This is the part that makes the approach safe for analytics.

The business gets more than a prettier name. It gets a traceable decision.

A simple checklist before production

Before I would let a text matching process feed a production semantic model, I would check these items:

Have we profiled the raw field and measured the real problem?
Are normalization rules stored in SQL, source control, or another reviewable place?
Do we keep raw values beside cleaned values?
Are match and cleanup rules named and versioned?
Are similarity thresholds different by field type?
Do uncertain matches go to human review?
Do approved matches produce audit fields?
Can a report owner explain why a value changed?
Can we measure false positives and false negatives?
Is this preview capability acceptable for the workload and release stage?

That last one matters.

These new capabilities are preview. Preview is fine for exploration, pilots, internal quality workflows, and controlled adoption. I would be more careful before using them as the only control behind a critical production merge process.

Where this is useful immediately

I would look at these use cases first:

Customer and supplier deduplication

Use normalized names, geography, tax IDs, domains, and similarity scoring to find likely duplicates. Send uncertain records to review.

Product catalog cleanup

Clean product labels, remove punctuation noise, compare descriptions, and flag suspicious near-duplicates.

Reference data validation

Check whether codes, IDs, email fields, phone fields, and source-system references follow expected patterns.

Migration cleanup

Before moving data into Fabric, profile messy source text and create a visible cleanup backlog.

Semantic model trust

Feed Power BI with fields that carry quality status alongside cleaned labels. That lets report builders expose quality context where needed.

The bigger point

The best part of this update is not that Fabric Warehouse can do more string work.

The best part is that teams can bring more of the text quality process into the same governed place where they already model, query, and serve analytical data.

That is a cleaner operating model than hiding business-critical cleanup logic in a spreadsheet, one notebook, or one person’s local script.

My recommended starting point:

Pick one painful text field. Profile it. Normalize it. Add one validation rule. Add one similarity rule. Create one review table. Publish one trusted output with audit fields.

Small enough to finish.

Structured enough to scale.

That is how this update turns from “new SQL syntax” into a real data quality improvement.

Sources

Fabric Skills Turn AI Prompts Into Platform Standards

Shai Karmani — Mon, 15 Jun 2026 00:50:57 +0000

Originally published at Data Ninja AI Lab.

Microsoft Fabric Skills look like a prompt library at first glance.

That is the least interesting way to read the announcement.

The more useful interpretation is this: Microsoft is starting to formalize how AI coding agents should work with Fabric. Not by giving them a vague “be helpful with data” instruction, but by packaging workload-specific guidance, API patterns, authentication context, MCP setup, and operational playbooks into reusable skills.

That changes the conversation.

For most teams, the AI problem is no longer “can the model answer a question?” The harder problem is “can the model follow the way our platform is supposed to be used?”

Fabric Skills are a step toward answering that second question.

What Microsoft released

Microsoft published Skills for Fabric, a public GitHub repository described as reusable AI assistant instructions for working with Microsoft Fabric.

The repo is designed for GitHub Copilot CLI and compatible AI coding tools. Microsoft also includes root-level configuration files for tools such as Claude Code, Cursor, Windsurf, Codex, Jules, and OpenCode.

The install path is straightforward.

/plugin marketplace add microsoft/skills-for-fabric
/plugin install fabric-skills@fabric-collection

Teams can also install focused bundles instead of the full package:

/plugin install fabric-authoring@fabric-collection
/plugin install fabric-consumption@fabric-collection
/plugin install fabric-operations@fabric-collection

There are workload filters too:

/plugin install fabric-skills@fabric-collection --filter "sqldw-*"
/plugin install fabric-skills@fabric-collection --filter "spark-*"
/plugin install fabric-skills@fabric-collection --filter "eventhouse-*"

That bundle structure is the important part.

It means Fabric agent work can be scoped. A developer can install authoring guidance when the agent needs to create or change artifacts. An analyst can use consumption guidance when the agent should inspect and query. An admin can use operations guidance when the task is diagnostic.

That is a better pattern than giving every AI tool broad instructions and hoping the user remembers the boundaries.

Why this is not just prompt engineering

Prompt engineering is usually personal.

One person finds a good wording pattern. Another person saves it in a note. A third person rewrites it for a slightly different tool. Six weeks later, nobody knows which version produced the working result.

That is fine for experimentation. It is not good enough for platform work.

Fabric work touches real assets:

Warehouse objects
Lakehouse files and tables
Eventstreams
Eventhouse and KQL databases
Dataflows Gen2
Semantic models
Power BI reports
Workspace items
REST API calls
Authentication and deployment steps

If AI agents are going to help with that work, the instructions need to become inspectable team assets.

A skill can define:

what workload the agent is working with
which API patterns are valid
which authentication flow is expected
what commands or MCP servers are relevant
which artifacts should be created or changed
what should be checked before the result is trusted

That is closer to an operating standard than a prompt trick.

The real value: shared instructions for repeatable work

The best use case is not asking an agent to “build something in Fabric.”

That prompt is too broad.

A better workflow sounds like this:

Use Fabric authoring skills to create a medallion architecture plan for this ingestion scenario. Produce the item list, workspace assumptions, API steps, and review checklist before any implementation work.

Or:

Use Fabric operations skills to investigate slow Warehouse queries. Summarize the evidence, likely bottlenecks, and the next safe actions. Do not change production objects.

Those are different jobs. They need different boundaries.

The skill layer helps put those boundaries where the agent can see them.

For a senior data team, that matters more than speed. Speed without repeatability creates a support problem. Repeatability creates a playbook.

Skills plus MCP is where this gets practical

The repository separates two ideas that should stay separate.

Skills provide guidance and patterns. MCP servers provide live tool access to data sources and APIs.

That separation is healthy.

A skill can teach the agent how to approach a Fabric task. An MCP server can give the agent a controlled way to inspect metadata, query data, or call a tool. The skill should not be treated as permission. The tool layer should still enforce what the agent can actually do.

This gives teams a cleaner mental model:

Skills are the instructions.
MCP servers are the tools.
Authentication is the gate.
Git is the history.
Pull requests or review checklists are the human control point.
Fabric is the target platform.

That model is much easier to govern than a chat window with a powerful account behind it.

Where I would start

I would not roll this out broadly on day one.

I would start with one narrow workflow where the output is useful but low-risk.

Good candidates:

Document a workspace

Let the agent inspect workspace structure and produce a readable inventory. This is useful, easy to review, and unlikely to damage anything if access is read-only.

Generate a medallion architecture plan

Use the skills to produce a proposed Fabric item design, naming convention, ingestion path, and validation checklist. Review the plan before any build work.

Investigate Warehouse performance

Use operations skills to collect evidence and recommend next steps. Keep the first pilot advisory only.

Create a first draft of implementation steps

Ask the agent to produce API calls, CLI commands, or notebooks as draft artifacts. Review before execution.

The pattern is the same in every case: advisory first, execution later.

That is how teams learn where the skills help, where the agent guesses, and what needs a local team standard layered on top.

The governance checklist I would use

Before I let agent skills near a real Fabric workspace, I would want a small checklist.

1. Pin the version

Record which skill bundle and version were used. The Skills for Fabric repo has a public changelog, and the skills are moving quickly. That is good, but it also means results can change.

If the agent produced a useful pattern, capture the version that produced it.

2. Separate read-only from authoring

Consumption and authoring are different risk levels.

A read-only agent that documents a workspace is not the same as an agent that creates or updates Fabric items. Treat those as different permission profiles.

3. Keep prompts with the output

If the output matters, the prompt matters.

Save the prompt, the skill used, the tool used, the workspace context, and the result. Otherwise the team cannot reproduce or audit the work.

4. Review changed artifacts, not just the summary

AI summaries can sound confident while hiding bad implementation details.

Review the actual output: notebooks, PBIP or PBIR files, semantic model changes, API payloads, KQL, T-SQL, Dataflow definitions, deployment commands, and workspace item changes.

5. Add local standards on top

Microsoft can provide the Fabric patterns. Your team still owns naming conventions, workspace design, security rules, deployment process, cost controls, and rollback paths.

The skill should be the starting point. The internal platform standard is the finished version.

What this says about the direction of Fabric

This release fits a broader pattern.

Fabric is becoming more agent-addressable.

We already see the pieces forming:

APIs for workspace and item operations
OneLake and catalog discovery
semantic model and Power BI artifact formats
MCP servers for controlled tool access
skills that teach agents how to work with Fabric workloads
Git-friendly artifacts and reviewable project structures

That combination is more important than any single AI demo.

The long-term shift is not “AI can write a query.” That is already normal.

The shift is that AI agents are being given a more structured way to participate in platform work: understand the workload, use the right API, produce reviewable artifacts, and operate inside a workflow a team can govern.

That is where the value is.

The risk

The risk is teams treating skills as a shortcut around engineering discipline.

They are not.

A Fabric skill can make an agent more useful. It does not automatically make the agent safe, correct, or aligned with your environment.

The same old questions still apply:

Which account is the agent using?
What workspace can it access?
Can it create or delete items?
Are generated artifacts reviewed?
Are API calls logged?
Is there a rollback path?
Who owns the result after the agent is done?

If those questions are missing, skills will only make bad automation faster.

My take

Fabric Skills are worth paying attention to because they move AI work closer to how serious platform teams actually operate.

Not one-off prompts.

Reusable instructions. Scoped bundles. MCP-aware workflows. API patterns. Versioned guidance. Reviewable artifacts.

That is the shape enterprise AI agents need.

The teams that get value from this will not be the ones that ask the biggest prompt. They will be the ones that turn useful agent behavior into small, governed, repeatable platform standards.

That is where Fabric Skills become interesting.

Sources

If this was useful, you can also connect with me on LinkedIn.

Fabric Real-Time Dashboards Just Became Much More Useful for Live Operations

Shai Karmani — Fri, 12 Jun 2026 23:00:28 +0000

Originally published at Data Ninja AI Lab.

Microsoft just shipped a set of Real-Time Dashboard updates that are easy to underestimate.

On their own, each feature sounds useful:

a redesigned tile editing experience with AI-assisted authoring
a dedicated Time Series visual
Live Refresh becoming generally available

Together, they point to something bigger.

Real-Time Dashboards in Fabric are starting to look less like report pages and more like operational screens. Not because the visuals are nicer. Because the loop is getting tighter between live data, visual authoring, time-based analysis, and refresh behavior.

That matters for teams that monitor systems, processes, events, queues, machines, applications, capacity, business events, or anything else where the question is not “what happened last month?”

The question is “what is happening now, and what should someone do about it?”

The practical shift

Most dashboards fail as operating tools for one of three reasons.

First, the visual is hard to build, so only a few technical people can create or maintain it.

Second, the time dimension is treated like a normal chart axis, even though real-time data needs zooming, comparison, entity selection, and synchronized timelines.

Third, refresh is either manual or interval-based. That means the dashboard can be stale, noisy, expensive, or all three.

The new Fabric updates attack those three problems directly.

The important point is not “Fabric has more dashboard features.”

The important point is that Real-Time Dashboards are becoming easier to build, easier to inspect, and easier to keep current without hammering the backend unnecessarily.

That is what moves a dashboard from reporting into operations.

1. Faster visual authoring with AI and KQL still in the loop

Microsoft’s new Real-Time Dashboard tile editing experience adds a cleaner authoring flow with AI-assisted visual creation, a larger preview area, and more flexible editing, as shown in Microsoft’s official update post.

You can start from a visual type, describe what you need in a prompt, review what Copilot generates, refine the output, and still work directly with KQL when needed.

This is the part I like: it does not remove the technical workflow.

For business users, the prompt path lowers the barrier to creating a first useful visual. For technical users, the editor still supports KQL, preview, schema inspection, parameters, and iterative refinement.

That is the right model.

Real-time dashboards need speed, but they also need control. A generated visual is only useful if the query, filters, time window, grouping, and labels are correct.

A good workflow looks like this:

Start with the operational question.
Let Copilot help produce the first visual or query shape.
Review the generated KQL and visual behavior.
Test parameters and edge cases.
Apply only when the visual answers the actual operating question.

The win is not that AI creates the dashboard for you.

The win is that AI can shorten the first-draft loop while the builder still owns the logic.

2. Time Series visualization makes real-time data easier to investigate

The new Time Series visual is more important than it looks.

A normal line chart is fine when you have one or two clean measures. Real operational data is rarely that polite.

You may have many sensors, services, queues, regions, SKUs, applications, machines, or business event types. You need to search through series, hide and show entities, compare measures, zoom into a time range, and keep the timeline aligned across views.

That is exactly the gap the Time Series visual is trying to close. Microsoft’s official preview post shows the visual with entity navigation, measures, synchronized timelines, and a focused time range.

Microsoft describes capabilities such as:

legend search for specific data series
entity and measure panels
hierarchical grouping
synchronized time sliders
separate charts for multiple measures
flexible Y-axis scaling
color assignment
zoom controls
linear and logarithmic axis options

That sounds like UI detail, but it changes the practical use case.

If I am monitoring application latency, I do not only want one average response time line. I want to compare services, regions, endpoints, time windows, and outliers.

If I am monitoring equipment, I do not only want a single sensor value. I want to isolate one machine, compare it to a peer group, zoom into the suspicious interval, and see whether the pattern repeats.

If I am monitoring business events, I do not only want a count. I want to see event volume, error patterns, processing lag, and unusual spikes in the same time context.

That is where a dedicated Time Series visual becomes useful.

It helps the viewer investigate without changing the underlying query every time.

3. Live Refresh changes the refresh contract

Live Refresh is now generally available for Real-Time Dashboards, with Microsoft documenting both the refresh status behavior and the settings pane for dashboard editors.

This is the feature that makes the operational story much stronger.

Traditional refresh creates a tradeoff. Short intervals keep the dashboard fresher, but they create more query load. Longer intervals reduce cost, but the screen can lag behind the event stream.

Live Refresh uses a different model. It detects when new data has been ingested and refreshes the dashboard visuals when there is something new to show. If no new data arrived, it avoids unnecessary visual refresh work.

That is a better fit for real monitoring.

Dashboards should update when the underlying state changes, not just because a timer fired again.

Microsoft also includes useful operational controls, such as pausing Live Refresh while investigating a data point and configuring dashboard refresh behavior from the settings pane.

For production use, I would treat Live Refresh as a contract, not a checkbox.

Before enabling it everywhere, answer these questions:

What is the acceptable delay between ingestion and visual update?
Which visuals support ingestion detection cleanly?
What fallback refresh interval is acceptable?
When should users pause refresh during investigation?
Which dashboard owner watches capacity impact?
What happens when the dashboard changes state?

That last question is the one teams often skip.

If a screen turns red and nobody owns the next action, it is not an operational dashboard. It is expensive wallpaper.

A simple architecture pattern

Here is the pattern I would use for a real implementation.

Start with the event source.

That might be an application log, IoT stream, business event, capacity event, queue, support workflow, or operational system. Land the data in Eventhouse where KQL can give you both current-state queries and historical context.

Then design the Real-Time Dashboard around one operating question.

Not ten questions. One.

Examples:

Are any order processing events stuck right now?
Which service tier is producing abnormal latency?
Which machines are drifting out of tolerance?
Is capacity pressure rising before users complain?
Which business event type needs human review?

Then choose the right dashboard capability:

Use AI-assisted visual authoring when you need a faster first visual.
Use Time Series when the investigation depends on comparing entities over time.
Use Live Refresh when new data should update the screen quickly and efficiently.

Finally, attach the operational layer:

owner
threshold
runbook
escalation path
known false positives
audit or investigation link

That is the difference between a dashboard people glance at and a screen people can actually operate from.

Example KQL shape

A Real-Time Dashboard visual usually lives or dies by the query shape. Here is a simplified example pattern for monitoring events by status over a short rolling window:

BusinessEvents
| where Timestamp > ago(30m)
| summarize EventCount = count() by bin(Timestamp, 1m), EventType, Status
| order by Timestamp asc

For a Time Series visual, I would usually make the entity and measure decisions explicit:

ServiceTelemetry
| where Timestamp > ago(2h)
| summarize
    AvgLatencyMs = avg(DurationMs),
    ErrorCount = countif(StatusCode >= 500)
  by bin(Timestamp, 1m), ServiceName, Region
| order by Timestamp asc

The query should tell the visual what the viewer is allowed to compare.

If the dashboard depends on a business concept like “delayed,” “failed,” “healthy,” or “at risk,” define that logic in the query or upstream model. Do not leave the viewer to infer it from raw lines.

My recommended build checklist

If I were building a Real-Time Dashboard for a real team, I would use this checklist before calling it done.

1. Define the operating question

Write the dashboard’s job in one sentence.

If the sentence is “monitor everything,” the dashboard is already too broad.

2. Define freshness

Decide how current the screen needs to be.

A factory safety signal, application outage, and executive sales trend do not need the same refresh behavior.

3. Design the time model

Pick the time window, bin size, timezone behavior, and comparison pattern.

This is where Time Series visualization can help, but it cannot fix a weak time model.

4. Validate the query

Review the KQL. Test empty data, late-arriving events, duplicate events, high-volume spikes, and unusual entity names.

5. Configure Live Refresh deliberately

Use Live Refresh where event-driven updates are useful. Set fallback behavior. Document when manual refresh or pause behavior is expected.

6. Add the action layer

Every important state needs an owner and next step.

If nobody knows what to do after the visual changes, the dashboard is unfinished.

The positive read

I like this direction because it makes Real-Time Dashboards more practical for real teams.

AI-assisted authoring helps more people get started.

Time Series visualization helps users investigate data that changes over time.

Live Refresh helps the dashboard stay current without turning refresh into a constant polling tax.

That combination is exactly what operational analytics needs.

The opportunity now is to stop treating Real-Time Dashboards as prettier report pages and start designing them as live operating surfaces.

That means better questions, better queries, better refresh contracts, and better runbooks.

The feature set is getting there.

The implementation discipline still matters.

Sources

Microsoft Fabric Updates Blog: A new way to create visuals on Real-Time Dashboards
Microsoft Fabric Updates Blog: Time Series Visualization in Real-Time Dashboard
Microsoft Fabric Updates Blog: Live refresh for Real-Time Dashboards
Microsoft Learn: Create Real-Time Dashboards
Microsoft Learn: Customize Real-Time Dashboard visuals

Shai Karmani

Senior data, BI, and AI practitioner focused on Microsoft Fabric, Power BI, analytics engineering, and practical AI systems.

Connect with me on LinkedIn

AI Can Build Power BI Reports Now. Here’s the Playbook I’d Use First.

Shai Karmani — Mon, 08 Jun 2026 23:33:57 +0000

Originally published at Data Ninja AI Lab.

Microsoft just opened a very interesting door for Power BI teams.

AI-powered Power BI reporting with agent skills is now in preview, and this is one of the most practical AI announcements in the Power BI space right now.

The reason is simple: this is not only chat over a report. This is AI helping with the actual report-building workflow.

Design pages. Generate PBIR files. Work inside a PBIP project. Reload Power BI Desktop. Capture screenshots. Improve the report based on what was actually rendered. Publish to Fabric when the report is ready.

That is a very different thing from asking Copilot to summarize a visual.

This is closer to giving an AI agent a real Power BI workbench.

What Microsoft released

Microsoft announced AI-powered Power BI reporting: From design to deployment with agent skills as part of the Power BI authoring plugin in Skills for Fabric.

The core idea: install a first-party Power BI authoring plugin, then use compatible AI tools, currently optimized for GitHub Copilot CLI, to build and modify Power BI reports through natural language.

The plugin can help an agent:

create report pages from a prompt
write schema-correct PBIR files
work with PBIP projects
reload an open Power BI Desktop report
capture screenshots from the rendered report
improve the report based on the screenshot output
coordinate with semantic model authoring and Modeling MCP
publish or manage reports in Fabric through companion skills

That last part is the important shift.

A lot of AI reporting demos stop at “generate a report.” This one is being designed around the artifacts Power BI developers already care about: PBIR, PBIP, semantic models, Desktop rendering, and Fabric publishing.

The repository behind this is public: microsoft/skills-for-fabric.

At the time I checked it, the repo was created on February 17, 2026, had 425 stars, 94 forks, and was still active, with a latest main-branch commit on June 7, 2026. The Power BI Authoring plugin manifest in the repo is at version 0.3.3.

This matters because it shows the direction clearly: Microsoft is not treating these as a throwaway demo prompt pack. This is a first-party skills catalog that can be installed, versioned, inspected, improved, and contributed to.

The Power BI Authoring plugin

The Power BI Authoring plugin lives under:

plugins/powerbi-authoring

The plugin currently includes these skills:

check-updates
semantic-model-authoring
powerbi-report-planning
powerbi-report-design
powerbi-report-authoring
powerbi-report-management

That split is smart.

Report building is not one task. It is a chain of decisions and artifacts.

You plan the report. You design the experience. You create or connect the semantic model. You author the PBIR files. You reload and inspect the report. You manage the Fabric item.

The plugin structure reflects that workflow instead of pretending one mega-prompt can do everything well.

The plugin also declares a local MCP server for Power BI Modeling:

"powerbi-modeling-mcp": {
  "type": "local",
  "command": "npx",
  "args": [
    "-y",
    "@microsoft/powerbi-modeling-mcp@latest",
    "--start"
  ]
}

That is where the ecosystem starts to become powerful.

Skills provide the operating instructions. MCP gives the agent live tool access. PBIP and PBIR give the work a file-based shape. Git gives the work history. Power BI Desktop gives the rendered output. Fabric gives the deployment target.

Put together, this becomes a real authoring loop.

How to install it

The install flow from Microsoft is short.

First, register the Skills for Fabric marketplace in GitHub Copilot CLI:

/plugin marketplace add microsoft/skills-for-fabric

Then install the Power BI Authoring plugin:

/plugin install powerbi-authoring@fabric-collection

If you want the broader Fabric bundle, Microsoft also documents:

/plugin install fabric-skills@fabric-collection

For a focused Power BI report pilot, I would start with powerbi-authoring@fabric-collection first. Keep the test narrow, prove the loop, then expand.

What this can actually do

Microsoft showed three practical examples in the announcement.

1. Create a report from scratch

You can ask the agent to create report pages with KPIs, slicers, tables, branding, and page structure.

For example, Microsoft’s demo prompt asks for an Opportunities page with revenue KPIs, slicers, and a table, then a Collabs page with offer status KPIs and filters.

The agent uses the powerbi-report-authoring skill to create Power BI report definitions in PBIR format.

This is a strong use case for the first draft of a report.

Not the final report. The first structured draft.

That alone can save a lot of time. Page scaffolding, KPI placement, slicer setup, table layout, and basic branding are not usually the highest-value part of BI work. They are necessary, but repetitive.

If an agent can get the first 60 percent into a usable PBIR structure, the developer can spend more time on business logic, model quality, visual clarity, and stakeholder feedback.

2. Modify an existing report from a prompt or reference image

The announcement also shows the agent updating an existing report based on a reference image and logo.

That means the workflow is not limited to greenfield reports.

You can point the agent at an existing PBIP project, describe the visual change, provide a reference image, and let it apply the style to the report pages.

This is where I see a lot of practical value.

Every BI team has reports that are useful but visually inconsistent. Different fonts. Random colors. Misaligned objects. Slicers in five different places. KPI cards that grew organically over time.

A good AI report assistant can help normalize those reports faster.

3. Modernize a messy report

Microsoft’s third example is the one that will probably resonate with the most Power BI teams: modernize a report with better design.

The prompt asks the agent to create a cleaner landing page, improve navigation, apply a consistent theme, reduce clutter, and make insights easier to scan.

Behind the scenes, Microsoft says the agent uses the powerbi-report-design skill to create a structured design brief, then passes that to the authoring skill for implementation.

This is exactly the kind of work where agent skills make sense.

The work has patterns. The output is visible. The files are structured. The result can be reloaded and checked. The agent can iterate.

That is a much better fit than asking an AI model to “make a dashboard better” with no real access to the report definition or rendered page.

The part that makes this different: screenshots in the loop

The feature I like most is the Desktop bridge.

Microsoft describes a loop where the agent can reload the report in an already-open Power BI Desktop instance, capture screenshots of the latest report pages, inspect the rendered output, and make another pass.

That changes the quality of the workflow.

Without screenshots, an agent is editing JSON and hoping the report looks right.

With screenshots, the agent can see the actual page.

That matters for:

overlapping visuals
bad alignment
poor spacing
unreadable labels
broken image placement
inconsistent card sizes
visual clutter
theme mismatch
navigation layout

This is the same reason designers do not approve a report by reading JSON. They look at the rendered page.

Giving the agent access to that rendered page is a big practical step.

Top use cases that can save real time

Here are the use cases I would prioritize first.

1. First draft report generation

Give the agent a clear brief:

audience
pages
KPIs
slicers
tables
navigation
required branding
source semantic model
examples of questions the report must answer

Then let it generate the first PBIR structure.

This is useful when the report shape is known but the build work is repetitive.

Example prompt:

Create a Power BI report for an executive sales pipeline review.

Use the Sales semantic model.

Page 1: Executive Overview
- KPI cards: Revenue Won, Revenue in Pipeline, Win Rate, Open Opportunities
- Trend: Revenue Won by Month
- Bar chart: Pipeline by Region
- Slicers: Region, Sales Owner, Close Month

Page 2: Opportunity Detail
- Table: Opportunity, Account, Owner, Stage, Risk, Expected Close Date, Revenue
- Add slicers for Stage and Risk
- Use a clean executive layout with strong navigation between pages

The point is not to make the prompt poetic. The point is to make it operational.

2. Report modernization backlog

Most organizations have a long tail of reports that people still use but nobody wants to redesign manually.

This is a perfect pilot category.

Pick five reports that are useful but ugly. Save them as PBIP. Ask the agent to improve one report at a time.

Good prompts here are direct:

Modernize this report for a monthly operations review.

Keep the same business meaning, but improve page structure, spacing, alignment, navigation, and visual hierarchy.

Create a cleaner landing page with the most important KPIs at the top.
Use a consistent theme across all pages.
Reduce clutter and make the page easier to scan in under 30 seconds.

This is where the powerbi-report-design skill should shine.

3. Brand and theme standardization

If a company has many reports across teams, style drift becomes real.

The agent can help apply a reference design, logo, color palette, or layout style more consistently.

This is not only about making reports pretty. Consistent design reduces cognitive load. Users know where to look. Filters behave more predictably. Navigation feels familiar.

4. Semantic model plus report creation

The Power BI report authoring skill can work with the Modeling MCP server and the semantic model authoring skill.

That means the bigger workflow can become:

inspect or create the semantic model
define measures and relationships
create report pages over that model
reload in Desktop
capture screenshots
refine the report
publish when ready

This is where the long-term value is.

A report without a good semantic model is just a nice-looking surface over weak logic. Pairing report authoring with semantic model authoring is the right direction.

5. Screenshot-driven report QA

The screenshot loop can save a lot of back-and-forth.

A normal report iteration might look like this:

change PBIR files
open or reload Power BI Desktop
check the page visually
fix spacing or formatting
repeat

If the agent can reload, screenshot, inspect, and adjust, it can take over a chunk of that mechanical loop.

That does not remove the BI developer. It gives the developer a faster loop.

6. Fabric publishing preparation

The powerbi-report-management skill is aimed at managing Power BI report workspace items in Microsoft Fabric through the Fabric REST API.

That includes creating, updating, downloading, and managing report definitions.

For teams already using PBIP, Git, deployment pipelines, and Fabric workspaces, this could become part of a more automated report release workflow.

My first pilot: a practical playbook

If I were testing this inside a real Power BI team, I would not start with the biggest executive dashboard in the company.

I would start with one report that is valuable, visible, and safe to iterate on.

Step 1: choose the right report

Pick a report with these traits:

already has a working semantic model
has 2 to 4 pages
needs layout or usability improvement
has clear business questions
does not require complex custom visuals for the first test
can be saved as PBIP

Good pilot examples:

sales pipeline report
inventory risk report
operations review report
finance month-end variance report
support tickets and SLA report
project portfolio status report

Avoid the monster report with 19 pages, 47 bookmarks, custom visuals, hidden pages, and years of business politics. That can come later.

Step 2: save the report as PBIP

The agent skills work with file-based Power BI report definitions. PBIP and PBIR are the important pieces here.

That means the report should live in a project folder where the report definition can be edited, inspected, and committed.

A simple structure might look like this:

sales-pipeline-report/
  Sales Pipeline.pbip
  Sales Pipeline.Report/
  Sales Pipeline.SemanticModel/
  briefs/
    report-brief.md

Create a Git branch for the experiment:

git checkout -b ai-report-skills-sales-pipeline

Now every change the agent makes has a place to live.

Step 3: write a real report brief

The quality of the output will depend heavily on the quality of the brief.

I would create a short report-brief.md file before asking the agent to touch anything.

Example:

# Sales Pipeline Report Brief

Audience: VP Sales, Sales Directors, Revenue Operations

Business goal:
Show pipeline health, revenue at risk, and opportunities that need attention this month.

Pages:
1. Executive Overview
2. Pipeline Detail
3. Risk Review

Required KPIs:
- Revenue Won
- Revenue in Pipeline
- Win Rate
- Open Opportunities
- At-Risk Revenue

Required slicers:
- Region
- Sales Owner
- Close Month
- Stage

Design direction:
Clean executive report. Strong KPI row. Simple navigation. Low clutter.
Use brand colors from theme.json.

Success criteria:
A VP should understand the pipeline status in 30 seconds.
A Sales Director should find at-risk opportunities without opening another report.

That kind of brief gives the agent something useful to work with.

Step 4: ask the agent to plan first

I would not start with “build the report.”

I would start with planning:

Use the Power BI report planning skill.
Read briefs/report-brief.md.
Inspect the semantic model metadata.
Propose the report page plan, required visuals, navigation structure, and any missing fields or measures.
Do not edit files yet.

This uses the planning skill for what it is good at: turning a request into a report specification.

Step 5: use design before authoring

Then I would ask for a design brief:

Use the Power BI report design skill.
Create a design brief for this report.
Prioritize executive scanning, clean page hierarchy, consistent navigation, readable KPI cards, and low visual clutter.

This is important because “create a report” and “create a good report experience” are not the same request.

The design skill gives the authoring skill a better target.

Step 6: let the authoring skill create or modify PBIR

Once the plan and design are clear:

Use the Power BI report authoring skill.
Implement the approved report plan in PBIR.
Create the pages, visuals, slicers, navigation, and theme updates described in the design brief.
Validate the report definition after the first implementation pass.

This is where the agent writes or updates the report files.

Step 7: reload Desktop and capture screenshots

Now the loop becomes visual:

Reload the report in Power BI Desktop.
Capture screenshots of each page.
Inspect the screenshots for layout, spacing, readability, navigation, and visual hierarchy.
Make one improvement pass based on the rendered output.

This is the part I would test hardest.

If the screenshot loop works well, this becomes much more than a prompt-to-JSON tool.

Step 8: publish only after the artifact is clean

When the report is in good shape, the management skill can help create or update report items in Fabric.

That publishing step should come after the PBIR files, semantic model binding, screenshots, and report behavior are ready.

A clean local loop first. Fabric publish second.

What I would measure in the pilot

I would measure this like an engineering workflow, not like a novelty demo.

For one report, track:

time to first useful draft
number of manual layout fixes needed
number of agent screenshot iterations
PBIR validation issues
semantic model issues discovered
how much of the report structure was reusable
whether the final output was easier to maintain than a manual build

The best result is not “AI built everything.”

The best result is this:

The team got to a useful, file-based, maintainable Power BI report faster, with more of the repetitive work handled by the agent.

That is a practical win.

Where I think this goes next

This is still preview, but the direction is obvious.

Power BI development is moving toward a more code-aware, agent-aware workflow:

PBIP makes reports file-based.
PBIR makes report definitions more editable.
TMDL makes semantic models more inspectable.
MCP gives agents access to real tools.
Skills give agents the right operating instructions.
Desktop screenshots give agents feedback from rendered output.
Fabric APIs give the workflow a deployment path.

That combination is much more interesting than isolated AI features.

It means a future Power BI workflow could look like this:

A business owner writes a report brief.
An agent proposes the page plan.
The agent creates the semantic model and report draft.
Desktop screenshots drive the first visual refinement pass.
The BI developer improves the model, measures, layout, and usability.
The report is published to Fabric.
The report definition remains in source control for future changes.

That is a strong direction for teams that already want better engineering discipline around Power BI.

My take

I am very excited about this direction.

Power BI teams spend too much time on repetitive report setup, redesign cleanup, visual alignment, theme drift, and the boring mechanics around first drafts.

Agent skills are a good fit for that work because the work is structured, file-based, visible, and iterative.

The big idea is not “AI replaces Power BI developers.”

The big idea is better:

AI agents can now participate in the same report-building loop that Power BI developers already use: model, files, Desktop, screenshots, Git, and Fabric.

That is where this becomes useful.

Start with one PBIP report. Install the Power BI Authoring plugin. Give the agent a real report brief. Let it plan, design, author, reload, screenshot, and improve.

If the loop works, you have something much more valuable than a demo.

You have the beginning of an AI-assisted Power BI development workflow.

Sources

Shai Karmani
Let’s connect on LinkedIn

Fabric IQ Is GA. This Is the Context Layer I’ve Been Waiting For.

Shai Karmani — Thu, 04 Jun 2026 23:03:52 +0000

Originally published at https://shai-kr.github.io/data-ninja-ai-lab/blog/2026-06-04-fabric-iq-ga-context-layer.html.

Fabric IQ becoming generally available is one of the Fabric milestones I was waiting for.

Not because the industry needed another AI announcement.

Because production AI agents have been missing something very basic: a shared, governed understanding of the business.

Most AI agent demos can answer a question if the prompt is clean, the data source is obvious, and the scope is small. That is useful for a demo. It is not enough for an enterprise workflow where the same customer, shipment, asset, incident, product, or KPI can mean different things across systems.

Microsoft is positioning Fabric IQ as the shared context layer for people, applications, and AI agents. The GA announcement includes Fabric IQ as the production context layer, with Graph and Operations Agents generally available and Ontology continuing in preview.

That nuance matters. The whole direction is production-facing, but not every individual piece has the same maturity label yet.

My take: this is the moment where Fabric starts looking less like a reporting platform with AI features and more like an operating layer for business context.

Why this is strategically important

AI agents do not fail only because the model is weak.

They fail because the business context is scattered.

One team defines active customer one way. Finance defines revenue another way. Operations tracks incidents in a different system. A report hides business logic in measures. A warehouse stores clean tables but not the real meaning behind the process. Then an agent is expected to reason across all of it.

That is where things get risky.

A production agent needs to know more than where the data lives. It needs to know what the business entities are, how they relate, which metrics are trusted, what rules apply, and which source owns the truth.

Fabric IQ is Microsoft’s answer to that problem.

The strategic shift is simple:

Stop asking every agent, report, app, and workflow to rediscover business meaning from scratch.

Define the context once. Govern it. Reuse it.

What Fabric IQ actually does

Fabric IQ sits on top of the Fabric data foundation and gives business meaning to data that would otherwise live as tables, streams, events, reports, and models.

Microsoft describes three connected layers.

1. Unified data in OneLake

OneLake gives Fabric IQ the common data foundation. Analytical data, operational data, shortcuts, lakehouses, warehouses, semantic models, and other Fabric items can participate in the same platform story.

2. Business intelligence through semantic models

Power BI semantic models already hold a lot of trusted business logic: measures, hierarchies, dimensions, relationships, and KPI definitions.

Fabric IQ does not throw that away. It uses semantic models as part of the context layer. You can generate or align ontology from semantic models so the business language used in reports can also ground agents and applications.

Many companies already spent years building trusted semantic models. The smart move is to reuse that logic, not rebuild it in prompts.

3. Operational intelligence through ontology and graph

This is the part that gets interesting.

Ontology defines business entities, properties, relationships, rules, and actions. Think Customer, Shipment, Store, Sensor, Order, Contract, Incident, and Asset.

Graph makes connected data explicit and queryable. Instead of asking an agent to guess how things relate through joins and table names, relationships can become first-class business context.

The part I like most: agents can stop guessing relationships

Graph in Fabric is now generally available. Relationship-first modeling is no longer just a nice preview idea sitting outside the core platform conversation.

For AI agents, relationships are not decoration.

They are the difference between:

“Show sales by customer”
“Which customers are affected by a supplier delay through the products they bought and the shipments currently in transit?”

The first question is normal BI.

The second question needs relationships, paths, dependencies, and business meaning. It needs to understand how entities connect across domains.

Traditional joins can answer some of this, but they usually hide the relationship logic in technical implementation. Graph and ontology make those relationships explicit enough for humans to review and for agents to use.

A mini tutorial: how I would start small

I would not start a Fabric IQ pilot by modeling the whole company.

That is how architecture diagrams become shelfware.

I would start with one narrow process where the relationships matter.

Example: retail inventory risk.

The business question could be:

Which stores are at risk because a high-revenue product has low inventory, recent demand is increasing, and the supplier is already delayed?

That is a good Fabric IQ candidate because it crosses entities and systems: Store, Product, Inventory, SaleEvent, Supplier, Shipment, and DelayReason.

Here is the smallest practical path I would use.

Step 1: start from a trusted semantic model or OneLake data

If a Power BI semantic model already has clean relationships and trusted measures, use it as the starting point and generate an ontology from it. If not, create the ontology directly from OneLake sources.

Do not bring every table. Pick the few entities needed for the first business question.

Step 2: rename technical objects into business language

This is not cosmetic.

An agent should not reason over dimproducts, factsales, and store_id as the primary business language.

Rename entity types into business terms such as Product, Store, SaleEvent, Supplier, and Shipment. Choose stable keys. Bind properties from source data. Define relationships like Store sells Product, Supplier ships Product, Shipment supplies Store.

Step 3: bind data and verify relationships

Data binding connects the ontology definitions to real OneLake data.

Before connecting an agent, I would check:

Are entity keys correct?
Are the important properties bound?
Are relationship directions understandable?
Are source systems documented?
Is there an owner for each business concept?

Step 4: connect a Fabric Data Agent

Create a Fabric Data Agent and add the ontology as a source.

Then test questions that force relationship reasoning, not just lookup behavior:

Which stores have low inventory for products with rising revenue in the last 14 days?

Which delayed shipments affect high-revenue products?

Which suppliers are connected to the most at-risk stores this week?

The goal is to prove that the agent is using governed business context instead of guessing from table names.

The governance question teams should ask first

Fabric IQ will be powerful for teams that treat it like infrastructure.

It will become confusing for teams that treat it like another AI feature.

Before I would let an ontology-backed agent near production, I would want clear answers to these questions:

Which business concepts are in scope?
Who owns each entity definition?
Which semantic model or source system is trusted?
Which relationships are reviewed by the domain owner?
Which agents can use this context?
What actions are allowed?
How do we test whether the agent used the right definition?
What changes when the ontology changes?

This is the same lesson as semantic models, but with higher stakes.

A bad measure can create a bad report. A bad ontology can create a bad agent decision.

My takeaway

Fabric IQ going GA is not just another Fabric announcement.

It is a signal that Microsoft is building the missing layer between data platforms and production AI agents: business context that can be modeled, governed, queried, reused, and connected to action.

That is why I was waiting for this milestone.

Semantic models gave BI teams a trusted language for reporting.

Fabric IQ pushes that idea further: a trusted context layer for agents, planning, graph reasoning, and applications.

The opportunity is huge, but the implementation discipline matters.

Start with one business process, one ontology, one trusted semantic model or OneLake source, one narrow agent scenario, and one owner who can say whether the answer makes sense.

That is how Fabric IQ becomes useful infrastructure instead of another impressive demo.

Sources

Shai Karmani

Let’s connect on LinkedIn

Fabric Business Events Just Became an Architecture Pattern

Shai Karmani — Mon, 01 Jun 2026 23:28:15 +0000

Originally published at https://shai-kr.github.io/data-ninja-ai-lab/blog/2026-06-01-fabric-business-events-architecture-guide.html.

A Business Event is a meaningful business signal that says something important happened and that another process, person, dashboard, model, or workflow may need to react.

That sounds simple, but the distinction matters.

A raw technical event might say a row changed, a sensor value moved, or a query returned a result. A Business Event should describe a business moment: a shipment was delayed, a high-value order is ready, a payment failed, or a demand forecast moved outside tolerance.

That is why the latest Fabric Business Events update is more than another alerting feature.

It moves Business Events closer to a real architecture pattern for turning operational signals into governed, reusable events that analytics, automation, AI, and business workflows can all consume.

The update matters because it expands the pattern in four directions:

Eventstream can publish Business Events from operational streams.
Activator can publish Business Events when a condition is detected.
Eventhouse and Real-Time Dashboards can analyze Business Events as persistent, queryable history.
Business Events now have clearer capacity consumption behavior.

The short version: this is no longer just alerting. It is event modeling for business operations.

What is actually new

The June 2026 update makes Business Events more practical across publishers, consumers, history, and cost ownership.

1. Eventstream can publish Business Events

Eventstream can act as the signal-processing layer.

Instead of sending raw telemetry, CDC rows, or low-level operational messages to every downstream process, teams can filter, enrich, correlate, and publish a named business event.

That matters because downstream consumers should not need to know every detail of the source system.

A raw event might say:

order_status_changed

A business event should say something closer to:

HighValueOrderReadyForFulfillment

The second one carries intent. It tells the organization what happened and why someone should care.

2. Activator can publish Business Events

Activator is no longer only a consumer that reacts to events.

With the preview capability, Activator can detect a condition and publish a Business Event into Real-Time Hub.

That condition can come from places like:

a Power BI report
a Real-Time Dashboard
a KQL query
a Fabric Warehouse SQL query

This is important because many business signals are not born as clean source-system events. They are detected from data.

A downtime indicator, fraud pattern, SLA breach, or inventory threshold may only become meaningful after a query or rule evaluates current state. Activator can turn that detection into a governed event other teams can discover and consume.

3. Eventhouse gives Business Events memory

Business Events can now be analyzed in Eventhouse and surfaced through Real-Time Dashboards.

That changes the operating model.

If events only trigger actions, teams can react in the moment but struggle to learn from the pattern. If events are also stored in Eventhouse, teams can ask better questions later:

How often did this event happen?
Which customers, products, regions, or systems were affected?
Did the event rate change after a deployment?
Which events usually happen together?
Should this event feed a model, a dashboard, or an automation?

Microsoft says each Business Event maps to a dedicated KQL table in Eventhouse, with no extra pipelines or manual configuration required. That is the part that makes the feature more interesting for analytics teams.

4. Capacity ownership is now part of the design

Business Events now follow a consumption model aligned with Azure and Fabric events.

The update describes two operation types:

Event operations per event, covering publish, filtering, and delivery.
Event listener per hour, charged while a consumer is actively listening.

The split matters.

Publish operations are charged to the Event Schema Set item. Filtering and delivery are charged to the consumer capacity, such as Activator or Eventhouse. Listener time is also charged to the consumer capacity.

That means event design is also a cost design. If every noisy technical signal becomes a Business Event, the architecture gets expensive and hard to reason about.

The architecture pattern I would use

I would not start with the alert.

I would start with the event contract.

A Business Event should describe a meaningful change in business state, not every technical thing that happened along the way.

Here is the practical pattern.

Step 1: decide if this is really a Business Event

Not every event deserves the label.

A Business Event should pass three tests.

Test 1: Does the business care when it happens?

Good examples:

PaymentFailed
ShipmentDelayed
HighValueOrderDetected
RefundIssued
DemandForecastDeviationDetected

Weak examples:

DiskReadError
MemoryUsagePercent
CurrentTemperature
UnhandledExceptionLogged

Those weak examples may still matter. They may belong in telemetry, monitoring, or observability. But they are not automatically Business Events.

Test 2: Should more than one consumer care?

If the signal only feeds one internal process, a direct integration may be enough.

If the same event could feed operations, analytics, automation, support, finance, or AI workflows, a Business Event starts to make sense.

This is where the decoupling matters. The publisher emits one governed event. Multiple consumers can subscribe without changing the original publisher.

Test 3: Can you name it without describing the implementation?

A good Business Event name should sound like a business fact, not a pipeline step.

Better:

CustomerCreditLimitExceeded

Weaker:

SqlQueryReturnedRows

The first one is a business state. The second one is an implementation detail.

Step 2: define the event contract before the flow

This is where teams often skip a step.

They build the stream, wire the alert, and only then realize every consumer needs a slightly different payload.

That creates a familiar mess: field remapping, version drift, unclear ownership, and consumers guessing what the event means.

The Business Events documentation points to Schema Registry as the shared source for event schemas. That should be treated as the contract layer.

For each Business Event, define:

event name
business meaning
owner
source system or publisher
schema version
required fields
optional fields
event time and processing time
correlation identifiers
consumer expectations
retention and analysis needs

A useful minimum payload might look like this:

{
  "eventName": "ShipmentDelayed",
  "eventVersion": "1.0",
  "eventTime": "2026-06-01T18:04:09Z",
  "shipmentId": "SHP-104920",
  "customerId": "CUST-8841",
  "delayReason": "CarrierCapacity",
  "estimatedDelayMinutes": 180,
  "sourceSystem": "FulfillmentPlatform",
  "correlationId": "c9a4f4b2-8f3a-4f0c-9de1-9ab2d7c81240"
}

That payload is small, but it gives consumers enough context to act, analyze, and trace.

Step 3: choose the right publisher

The new update makes this decision more interesting.

Use Eventstream when the signal starts as operational stream data.

Examples:

CDC rows from an operational database
IoT or device events
Kafka or Event Hubs messages
incoming application events
high-volume signals that need filtering or enrichment

Use Activator when the signal is detected from a condition.

Examples:

a Power BI report threshold
a KQL query result
a warehouse query condition
a real-time dashboard rule
a business condition that only exists after evaluation

Use Notebook or User Data Functions when the event requires custom logic.

Examples:

model scoring
enrichment
validation
business rule evaluation
more complex event generation logic

The key is to avoid treating all publishers the same. A publisher is not just a connection point. It defines where the event becomes meaningful.

Step 4: separate action from history

This is the part I like most in the update.

Business Events can trigger action through consumers like Activator, Power Automate, notebooks, Spark jobs, Dataflows Gen2, or custom logic. But they can also land in Eventhouse for historical analysis.

That separation is healthy.

Action answers: what should happen now?

History answers: what keeps happening, where, and why?

If you only build action, you get automation without learning.

If you only build history, you get dashboards without response.

The better design does both.

Step 5: put capacity in the design review

Capacity should not be a surprise after go-live.

For every Business Event, ask:

How many events per hour do we expect?
Which consumers will listen continuously?
Which capacity pays for publishing?
Which capacity pays for filtering, delivery, and listening?
Do we need every event, or only state changes that matter?
Is this event too noisy for a business-level contract?

This is especially important when teams convert raw streams into Business Events. The point is not to rename telemetry. The point is to publish meaningful moments.

A practical checklist before you ship

Before I would put a Fabric Business Event into production, I would check this:

The event has a clear business name.
The event has an owner.
The schema is defined before consumers are built.
The schema includes event time, source, and correlation ID.
Eventstream, Activator, Notebook, or UDF was chosen intentionally.
At least one consumer has a real business action.
Eventhouse history is useful, not just stored by default.
Real-Time Dashboard visuals answer operational questions.
Capacity ownership is documented.
No raw telemetry stream is being disguised as a business event.

That last point is the trap.

Business Events are valuable when they create shared business language. They become expensive noise when every technical signal gets promoted without a contract.

Where this fits in Fabric architecture

Fabric is slowly closing the gap between analytics and operational response.

Power BI reports can surface conditions.

Activator can detect and act.

Eventstream can shape operational signals.

Real-Time Hub can organize event discovery.

Eventhouse can preserve and query event history.

Real-Time Dashboards can show live operational state.

The architecture question is no longer only “can Fabric send an alert?”

The better question is: which business events deserve to become reusable contracts across the data platform?

That is where this feature becomes useful.

Not because alerts got prettier.

Because Fabric is giving teams a way to model business moments as governed, discoverable, queryable events.

Used carefully, that can reduce the distance between data, action, and learning.

Used casually, it becomes another stream of noise.

The difference is the contract.

Sources

Shai Karmani

Let’s connect on LinkedIn

Build Power BI Columns That Adapt to Each User

Shai Karmani — Thu, 28 May 2026 22:35:27 +0000

Originally published at https://shai-kr.github.io/data-ninja-ai-lab/blog/2026-05-28-user-aware-calculated-columns-power-bi.html.

Power BI calculated columns are getting a new design option that is easy to underestimate.

The setting is called Expression Context.

The option is User Context.

The result is a calculated column that can be evaluated at query time, under the security context of the user who is running the report.

That opens a useful set of patterns for semantic model authors:

values that change by user culture
row-level calculations that do not need to be stored as physical columns
sensitive values that can stay visible to admins and blank for restricted users
Direct Lake and Import models that need cleaner control over calculated column materialization

The feature is still preview territory, so I would not treat it as a casual modeling shortcut. But it is already worth understanding because it changes how we think about calculated columns in Power BI.

Source: SQLBI: Introducing user-aware calculated columns in Power BI

What changes with User Context

A standard calculated column is evaluated when the table is processed.

In Import mode, the result is stored in the semantic model. Once it is processed, the value is the same for every user who queries the model.

A user-aware calculated column changes that behavior.

When Expression Context is set to User Context, the expression is evaluated at query time. It runs under the active user security context, and it can use user-aware DAX functions such as:

USERCULTURE()
USERPRINCIPALNAME()
USEROBJECTID()
USERNAME()
CUSTOMDATA()

That means the column can still behave like a column in the model, but the value can depend on who is asking the question.

I would think about it as a semantic model design tool, not only as a localization feature.

Pattern 1: build reports that speak the user language

The cleanest first use case is localization.

A Date table can expose month names or day names that change based on the user's culture. For example:

Month =
FORMAT (
    DATE ( 2020, 'Date'[Month Number], 1 ),
    "mmmm",
    USERCULTURE()
)

If the user culture is English, the report can show January.

If the user culture is French, the same column can show janvier.

The model does not need separate month-name columns for every language. The expression can return the correct value at query time.

This is where the feature becomes practical. Many organizations serve the same report to users in different regions. The metadata translation story already exists for names of tables, columns, and measures. User-aware calculated columns add another piece: values inside the model can adapt too.

The slicer detail that matters

Localization creates a subtle modeling problem.

If a slicer stores the selected value as translated text, that selection may not survive when the same report is viewed in another culture.

For example, a slicer selection of Sunday does not match dimanche.

The better design is to let the user see the translated label but keep the selection anchored to a stable key, such as Day of Week Number.

That is where Sort by Column and Group By Columns matter.

SQLBI shows the TMDL version clearly:

The principle is simple:

display the user-aware text column
sort it by a numeric column
group it by the stable numeric identifier
avoid storing report selections as translated strings

That is the difference between a nice demo and a report that behaves correctly across languages.

Pattern 2: create virtual columns for row-level calculations

The second pattern is less obvious and probably more important for model design.

A user-aware calculated column is not materialized in Import mode. It exists in the model, but its values are not stored as a physical column in memory.

That can be useful for simple row-level expressions.

A common example is line amount:

Line Amount = Sales[Quantity] * Sales[Net Price]

As a standard calculated column, that expression creates another stored column. If the table is large and the value has high cardinality, the column can add memory and processing cost.

As a User Context column, the same expression can behave more like a virtual column. It remains available to visuals, filters, slicers, and measures, but it does not need to be stored in the model.

This is useful when the expression is simple enough for the engine to compute efficiently during query execution.

Good candidates:

arithmetic on columns from the same table
simple classifications with stable input columns
labels or helper columns that are useful to report authors
logic that benefits from being a field, not only a measure

Poor candidates:

complex row-by-row DAX
expressions that call expensive table functions
logic that triggers formula engine callbacks at scale
anything that has not been tested with realistic data volume

The practical takeaway: User Context can reduce stored model bloat, but it moves work to query time. That tradeoff needs measurement.

Pattern 3: keep one report layout while hiding sensitive values

The third pattern is security-aware modeling.

Object-level security can hide a column completely. That is sometimes the right answer, but it can break report visuals that reference the hidden column.

User-aware calculated columns give another option for some scenarios: keep the column available in the report, but return blank values for restricted users.

SQLBI demonstrates this with income bracket data.

The supporting table stores the sensitive value. RLS blocks that table for restricted users. A user-aware calculated column uses LOOKUPVALUE() to bring the value into the visible table.

The key design choice is that the sensitive lookup table stays disconnected from the main customer table.

That matters because the RLS filter should block the lookup result. It should not propagate through relationships and remove the customer rows or sales rows from the report.

For an admin user, the report can show the income bracket values:

For a restricted user, the same report still renders, but the sensitive values become blank:

This is not a replacement for every object-level security scenario. Restricted users can still see that the column exists. But for reports where the layout must keep working while sensitive values are redacted, it is a useful pattern to test.

How I would evaluate this in a real model

I would not start by asking, "Can this replace my calculated columns?"

I would start with these questions:

1. Does the value need to change by user?

If the expression depends on culture, identity, role, or security context, User Context may be the right design.

If every user should see the same value, be more careful. The only benefit may be avoiding materialization, and that creates a query-time cost tradeoff.

2. Is this a value users need as a field?

Measures are great for aggregations.

Columns are useful when report authors need a field for slicers, filters, grouping, or visual axes.

User-aware calculated columns can fill a gap where the logic needs to live as a field, but the model author does not want to store another physical column.

3. Can the expression run cheaply at query time?

Simple arithmetic is a better candidate than complex DAX.

A virtual column that saves memory but slows every report page is not a win.

4. Have you tested role behavior?

For security-aware patterns, test with View as role before trusting the design.

Check that restricted users see blanks where expected, and that the rest of the report still returns the correct rows.

5. Are selections stable across languages?

If the value is localized, do not let the visible label become the identity of the selection.

Use stable keys for grouping and sorting.

Where this fits with the May 2026 Power BI update

The May 2026 Power BI update includes several modeling and reporting changes around Copilot, visual calculations, custom totals, report summaries, and locale behavior.

One line in the Microsoft update is especially relevant here: default format string locale affects visual display, while USERCULTURE() and metadata translations still use the viewer's browser locale.

That distinction matters.

Power BI is giving model authors more control over where logic lives:

visual layer logic with visual calculations
semantic model logic with DAX, TMDL, and PBIP
AI readiness metadata with Prep data for AI
user-aware values with Expression Context and User Context

The direction is clear: the semantic model is becoming more programmable, more reviewable, and more sensitive to the context of the person consuming the report.

Source: Microsoft Learn: May 2026 Power BI Update

A practical checklist before using it

Before I would ship a user-aware calculated column, I would check this:

Is the feature supported in the target Power BI Desktop and service environment?
Is the table storage mode compatible with the intended behavior?
Does the expression use user-aware DAX functions intentionally?
Is the expression simple enough to evaluate at query time?
Are translated labels grouped by stable keys?
Are RLS and View as role tests clean?
Are report visuals still valid for restricted users?
Is the behavior documented in the model repository or TMDL?

If the answer is yes, User Context becomes a powerful tool.

Not because it makes calculated columns more clever.

Because it lets the semantic model respond to the user, while keeping the logic in one place.

That is a useful direction for serious Power BI models.

Sources

Written by Shai Karmani

Copy Job CDC with SQL estate is now GA in Microsoft Fabric

Shai Karmani — Tue, 26 May 2026 23:00:53 +0000

Originally published at https://shai-kr.github.io/data-ninja-ai-lab/blog/2026-05-26-copy-job-cdc-sql-estate-ga.html.

Copy Job CDC with SQL estate is now generally available in Microsoft Fabric Data Factory.

That sounds like a data movement update.

It is more useful than that.

For many BI and data engineering teams, the hard part is not copying data once. The hard part is keeping analytical data aligned with operational systems after the business changes.

A customer address is updated.

An order status changes.

A subscription is canceled.

Inventory moves.

A record is deleted in the source system, but the reporting layer still needs to explain what happened.

Those are not edge cases. They are normal business behavior. If the analytical platform cannot handle them clearly, trust starts to leak out of the reporting layer.

That is why CDC matters.

What became generally available

Microsoft announced that Change Data Capture support for SQL estate in Copy Job is now generally available.

The generally available SQL estate sources include:

SQL Server
Azure SQL Database
Azure SQL Managed Instance
SAP Datasphere

The generally available destinations include:

SQL Server
Azure SQL Database
Azure SQL Managed Instance
Fabric SQL
Snowflake

Microsoft is also moving more CDC connectors through preview, including Fabric Lakehouse table, Google BigQuery, Snowflake, Oracle, SQL database in Fabric, and Fabric Data Warehouse scenarios.

The important part is not the connector list by itself. The important part is that CDC is becoming a normal Data Factory pattern inside Fabric, not a side script that each team has to invent again.

Source: Microsoft Fabric Updates Blog: Simplify your data movement with Copy job: CDC with SQL estate

What you can actually do with it

1. Stop rebuilding full loads when only changes matter

Full refresh is simple until it is not.

It works when the tables are small, the source systems can handle the load, and nobody cares about latency. That changes quickly with operational SQL systems.

CDC lets the pipeline focus on what changed. That can reduce load on the source system, reduce movement volume, and make analytical updates closer to operational reality.

This is especially useful for tables such as orders, customers, products, subscriptions, transactions, inventory, service tickets, and account status history.

2. Bring SQL estate replication closer to Fabric Data Factory

A lot of organizations already have replication logic around their SQL estate. Some of it is mature. Some of it is a set of custom jobs nobody wants to touch.

Copy Job CDC gives Fabric teams a cleaner option for the right workloads.

Instead of maintaining another custom replication layer, a team can move more of the pattern into Fabric Data Factory, where the data movement is visible as part of the platform.

That does not mean every existing pipeline should be replaced tomorrow. It does mean new Fabric architecture decisions should consider CDC as a first-class option.

3. Preserve history with SCD Type 2

For reporting, the latest value is often not enough.

If a customer changed region last month, some reports need the current region. Other reports need the region that was true when the order happened.

That is where slowly changing dimension Type 2 patterns matter.

Microsoft also highlighted extended SCD Type 2 support for Fabric Warehouse and Synapse SQL Pool. With native SCD Type 2 in Copy Job, teams can preserve historical versions of records with effective dating and soft delete handling.

That is not just a data warehouse modeling detail. It is the difference between a report that shows the current answer and a report that can explain the historical answer.

4. Treat deletes as audit events, not disappearing rows

Deletes are dangerous in analytics.

If a source record disappears and the destination simply removes it, the reporting layer may lose the ability to explain prior results.

Soft delete handling is useful because the destination can mark a record as inactive instead of physically deleting it. That keeps the history visible for audit, reconciliation, and operational reporting.

For finance, subscriptions, customer lifecycle, compliance, and operational analytics, that distinction matters.

The architecture conversation gets better

The real value is not that Fabric can copy data from A to B.

Teams have had ways to copy data for years.

The value is that Fabric is making change capture, history, deletes, latency, and ownership easier to discuss as platform concerns.

That changes the conversation from:

How do we move the data?

To:

How do we trust the data after it changes?

That is a better architecture question.

Where I would use this first

I would look for workloads with these signals:

Operational SQL source systems
Tables that change frequently
Reports that need fresher data than a nightly full load
Business processes where historical state matters
Deletes that need traceability
Custom replication jobs that are becoming hard to maintain
Fabric adoption where Data Factory is already part of the platform

Good candidates are customer dimension sync, order status tracking, subscription lifecycle reporting, inventory movement, financial transaction replication, and support or case management analytics.

Where I would still be careful

GA does not remove design responsibility.

Before moving a production workload, I would still define:

Source ownership
Expected latency
Initial load strategy
Change tracking assumptions
Delete behavior
SCD Type 2 rules
Failure handling
Reconciliation checks
Security and access model
Monitoring and support ownership

CDC makes the movement pattern easier. It does not automatically make the architecture clean.

My takeaway

Copy Job CDC with SQL estate becoming GA is a practical Fabric milestone.

It gives BI and data engineering teams a stronger native option for moving operational SQL changes into analytical systems, while preserving history and making deletes more traceable.

The best use of this feature is not to treat it as another ETL checkbox.

Use it to make change history, auditability, and trust explicit in the Fabric architecture.

That is where the feature starts to matter.

Shai Karmani

Let’s connect on LinkedIn

Fabric AI Functions Turn GenAI Into a Data Pipeline Step

Shai Karmani — Tue, 26 May 2026 00:33:12 +0000

Originally published at https://shai-kr.github.io/data-ninja-ai-lab/blog/2026-05-24-fabric-ai-functions-data-workflows.html

Most enterprise GenAI demos start in the wrong place.

They start with a chat window.

The more useful place is usually earlier: inside the data workflow, before the dashboard, before the semantic model, before the analyst has to clean the same messy text for the tenth time.

That is why Fabric AI Functions are worth paying attention to.

They let data teams use GenAI directly inside pandas and Spark workflows in Microsoft Fabric. Not as a separate app. Not as a one-off script sitting outside the platform. As a transformation step inside the work data teams already do.

That changes the shape of the use cases.

Instead of asking “how do we add a chatbot?”, the better question becomes:

Where is language, document mess, or unstructured content slowing down our data pipeline?

What you can actually do with it

Fabric AI Functions expose common GenAI operations as DataFrame-friendly functions.

You can use them to:

classify support tickets, survey responses, incidents, or customer feedback
summarize notes, long text fields, operational logs, and service records
extract fields from documents or semi-structured text
translate records as part of a data preparation flow
fix grammar or normalize messy text before reporting
create embeddings for search, RAG, and semantic retrieval
compare similarity between text values
generate structured responses from instructions
enrich rows in pandas or Spark without moving the workflow outside Fabric

That sounds simple, but it is a useful shift.

For years, a lot of GenAI work around data platforms has looked like this:

Export data from the platform.
Send it to a separate script or service.
Call an AI model.
Stitch the result back into the data estate.
Hope the process is governed enough to survive production.

Fabric AI Functions make a cleaner pattern possible.

The AI step can live closer to the lakehouse, notebook, Spark job, data science workflow, Power BI preparation layer, and downstream semantic model.

That is a much better starting point for teams that want AI to improve real data work, not just demo well.

The big changes that make this interesting

There are a few parts that matter more than the feature list.

1. GenAI becomes part of the pipeline

The most important change is architectural.

AI enrichment can become a normal transformation step.

A notebook can read raw records, apply an AI function, store the output as another column or table, and send that enriched dataset into the next layer of the platform.

That means AI output can be reviewed, versioned, refreshed, tested, governed, and consumed like other data assets.

That is very different from treating GenAI as a sidecar experiment.

2. Multimodal input makes the use cases much better

Text classification is useful, but many business workflows are not clean text.

They are PDFs.

Screenshots.

Images.

CSV files.

JSON files.

Markdown notes.

Operational documents that never quite made it into a table.

Microsoft documents AI Functions support for image files such as JPG, PNG, GIF, and WebP, documents such as PDF, and common text formats such as MD, TXT, CSV, JSON, and XML.

That opens better Fabric workflows.

A team can bring files into the lakehouse, use AI to extract or summarize what matters, and store the result in structured tables for review and reporting.

That is the kind of AI use case that can save real operational time.

3. Embeddings can be created where the content already lives

ai.embed is one of the more important functions because it connects Fabric directly to search and RAG preparation.

A team can take product documentation, policy files, support resolutions, internal wiki pages, field notes, or knowledge base articles and create embeddings as part of the data workflow.

That creates a cleaner path from raw business content to retrieval-ready datasets.

The useful part is not just the embedding itself. It is that the data team can decide what content is approved, what should be excluded, how often embeddings refresh, and what downstream applications are allowed to use.

4. The model/provider configuration is becoming more serious

The documentation now covers configuration details around providers and models, including the default model behavior.

That matters because production teams eventually need answers to basic governance questions:

Which model is being used?
Who approved it?
Which data can be sent to it?
Which capacity pays for it?
Which workloads are allowed to use it?
What happens when the output is wrong?

This is where Fabric AI Functions become more than a notebook convenience. They become part of the data platform operating model.

5. The best output is not “AI magic”. It is a reviewable data asset.

The mistake is to take AI output and treat it as automatically trusted.

The better pattern is to produce reviewable enrichment.

Keep the original value.

Add the AI-generated label, summary, extracted field, or embedding.

Add review flags where needed.

Store the result in a table with ownership and downstream rules.

Then decide what is safe enough for reporting, automation, search, or user-facing apps.

That is how this becomes useful without becoming sloppy.

Three practical things I would build first

1. Support ticket enrichment

Most support datasets contain useful signal, but the text is messy.

A Fabric notebook can add AI-generated columns for:

topic classification
urgency
sentiment
short summary
product area
likely ownership team

The key is not to pretend the model is perfect. The key is to create a reviewable enrichment layer that helps analysts and operations teams move faster.

A good output table might include the original text, AI-generated labels, confidence or review flags where available, and a human-reviewed status column.

That gives Power BI a better dataset without hiding the uncertainty.

2. Document extraction into structured tables

A lot of business data is trapped in semi-structured documents.

Invoices, forms, reports, agreements, field notes, inspection PDFs, and vendor files often contain fields that teams later retype manually.

With AI Functions, the useful pattern is:

Store the files in the lakehouse.
List file paths as input.
Use extraction or generation instructions to pull out the fields.
Store the result as a structured table.
Review exceptions before the data becomes trusted.

That does not replace proper document processing for every scenario. It does make small and medium internal automation projects much easier to test inside Fabric.

3. Embeddings for search and RAG preparation

A team can take approved internal content and create embeddings as part of the Fabric workflow.

That content might include:

product documentation
policy files
support resolutions
internal wiki pages
knowledge base articles
implementation notes

The output can become a governed retrieval layer instead of a random pile of files passed into an AI app.

That matters because RAG quality starts before the chat interface. It starts with content selection, metadata, refresh rules, ownership, and preparation.

Where I would be careful

Positive does not mean careless.

AI Functions make enrichment easier, but the usual production questions still matter:

Which data is allowed to be sent to the model?
Is the Fabric tenant setting for Copilot and Azure OpenAI enabled intentionally?
Does the workload require cross-geo processing approval?
Which Fabric capacity will pay for the work?
Which model/provider is configured?
How will output quality be reviewed?
Which outputs are allowed to flow into reports or user-facing apps?
How will failures, blanks, and hallucinated values be handled?

Microsoft notes that Fabric AI Functions require a paid Fabric capacity, F2 or higher, or any P capacity. The documentation also states that AI Functions are supported in Fabric Runtime 1.3 and later, and that the default model is gpt-4.1-mini unless a different model is configured.

Those details matter. They turn this from a cool notebook feature into a platform decision.

My take

Fabric AI Functions are useful because they move GenAI into the unglamorous part of AI work.

The pipeline.

The notebook.

The enrichment step.

The document cleanup.

The semantic preparation layer.

That is where a lot of business value actually sits.

Not every AI feature needs to become a chat window. Some of the most valuable AI work will happen quietly inside pipelines, quality checks, enrichment jobs, and retrieval preparation steps.

The practical opportunity is simple:

Take the data you already manage in Fabric. Add AI where language, documents, and meaning slow the team down. Store the result as a governed data asset. Review it before it reaches users.

That is a much better direction than treating AI as a separate island next to the data platform.

When did this become available?

The official Microsoft Learn page for Fabric AI Functions currently has a documentation date of November 13, 2025 and an updated timestamp of May 7, 2026.

The GitHub history for the Fabric documentation shows the AI Functions overview page existed by February 28, 2025. A later documentation commit on November 24, 2025 is titled “Update AI Functions documentation for GA release with enhancements.” Recent documentation updates in February, March, and May 2026 added more coverage around multimodal input, schema extraction, configuration, providers, and file workflows.

So the short version is:

The documentation trail starts in early 2025.
The GA documentation update appears in November 2025.
The more interesting expansion for practical teams is the 2026 work around multimodal inputs, broader model/provider configuration, schema extraction, and file workflows.

Sources

Shai Karmani

Let’s connect on LinkedIn

DEV Community: Shai Karmani

Fabric Warehouse Brings AI Enrichment Into T-SQL. Here’s the Practical Guide.

What Microsoft added

A simple example

The function guide

1. Use AI_ANALYZE_SENTIMENT for directional signals

2. Use AI_CLASSIFY when you control the categories

3. Use AI_EXTRACT to turn text into structured fields

4. Use AI_SUMMARIZE for readability, not evidence

5. Use AI_TRANSLATE when language blocks analysis

6. Use AI_FIX_GRAMMAR carefully

7. Use AI_GENERATE_RESPONSE with the most discipline

The production pattern I would use

Where this fits in a Fabric architecture

What I would avoid

Final take

Fabric Lakehouse Health Checks Make Optimization Practical. Here’s the Runbook.

The problem this solves

What the stored procedure gives you

The runbook I would use

1. Start with the critical tables

2. Run the health check before maintenance

3. Classify the result

4. Decide, then act

A practical pipeline pattern

Step 1: Table list

Step 2: Health check activity

Step 3: Decision rule

Step 4: Spark maintenance

Step 5: Post-check

What I would measure

Where this fits in a Fabric operating model

What I would avoid

Trap 1: Optimizing everything because you can

Trap 2: Treating the anomaly description as the whole diagnosis

Trap 3: Ignoring the upstream write pattern

Trap 4: Not logging the decision

A simple first-week rollout

Day 1: Identify targets

Day 2: Run manual checks

Day 3: Create the log table

Day 4: Add a conservative pipeline

Day 5: Add one maintenance action

The practical takeaway

Source

Fabric Data Factory Makes Multi-Cloud Integration Practical. Here’s the Architecture Checklist.

The opportunity

The architecture pattern I would use

1. Landing zone

2. Identity path

3. Transform boundary

4. Cost model

5. Failure contract

6. Business output

A low-risk rollout sequence

What I would avoid

The practical takeaway

Source

Fabric Warehouse Can Clean Messy Text Now. Here’s the Data Quality Playbook.

What changed

The pattern I would use

Step 1: Profile the messy field first

Step 2: Normalize before you match

Step 3: Compose clean keys and audit labels clearly

Step 4: Use similarity scoring for likely duplicates

Step 5: Create a review queue for uncertain matches

Step 6: Write the trusted result with audit fields

A simple checklist before production

Where this is useful immediately

Customer and supplier deduplication

Product catalog cleanup

Reference data validation

Migration cleanup

Semantic model trust

The bigger point

Sources

Fabric Skills Turn AI Prompts Into Platform Standards

What Microsoft released

Why this is not just prompt engineering

The real value: shared instructions for repeatable work

1. Use `AI_ANALYZE_SENTIMENT` for directional signals

2. Use `AI_CLASSIFY` when you control the categories

3. Use `AI_EXTRACT` to turn text into structured fields

4. Use `AI_SUMMARIZE` for readability, not evidence

5. Use `AI_TRANSLATE` when language blocks analysis

6. Use `AI_FIX_GRAMMAR` carefully

7. Use `AI_GENERATE_RESPONSE` with the most discipline