DEV Community: Sai Prudhvi Neelakantam

Databricks Genie: Can It Build My Deck and Deploy My App?

Sai Prudhvi Neelakantam — Fri, 13 Nov 2026 10:12:47 +0000

I used AI tools to help draft and edit this post. I checked the facts and the wording, and the original lives at geekindata.com. The diagrams are on that page.

Genie Code finishes platform-shaped jobs when context is already in the room

Key takeaways

Context is the product: native lakehouse and deploy awareness beats thin MCP babysitting.
Optimize for artifacts people present and run (editable decks, hosted apps), not chat answers alone.
Encode company PPTX templates as versioned skills; brand fidelity is not a first-prompt miracle.
Capture compute shape in the deploy runbook (serverless or classic, app identity, Unity Catalog grants).
Treat vendor task-success percentages as a signal; reproduce on your tables, template, and host.
Human review stays on the critical path before executive or external audiences.

Can Databricks Genie make a PowerPoint from my data and respect a company template? Can it develop and deploy a web app that lands on a Genie space?

Those are the questions I keep getting after demos. I have been stress-testing Genie Code on two jobs: turn governed data into an editable deck, and ship a small web app on the platform. That raises the bar for what an enterprise data platform assistant can own end to end.

Named model: context → artifact → review

Context: Genie Code already sees what the principal can see: tables, governance, and platform constraints. You inherit entitlements instead of pasting extracts into a consumer chatbot.
Artifact: The assistant continues past chat into decks, apps, and deploys. A correct SQL answer is not a board pack. A code sketch is not a hosted app.
Review: Editable decks still need metric-definition checks. Deployed apps still need auth, secrets, cost controls, and CI. Draft speed is the win. Unreviewed publish is the failure mode.

Figure 1. Named model for Genie Code work. Native context feeds artifact generation. Review gates what reaches an audience or a shared host.

Context and constraint

Most buzz around Genie Code focuses on data questions or ETL scaffolding. I wanted the day-to-day path: can it produce artifacts people actually present and run? After those tests, it feels closer to Claude Code or Cursor than to shallow autocomplete, with enterprise data security already in the room. Native Databricks context on tables, governance, and platform constraints means less token waste restating warehouse topology.

Genie Code matters when it continues past chat into artifacts. If it reasons about deploy constraints, you spend less time as the human MCP adapter. Figure 2 is the ownership split I use when someone asks whether a generic coding agent with thin MCP is "the same thing."

Figure 2. Same class of coding help, different babysitting load. Native context finishes platform-shaped jobs. Generic agents stay flexible across stacks but re-teach the warehouse and deploy reality every session.

Design

1. Deck generation as a skill-backed workflow

I connected Genie Code with an open-source PPTX skill (the same family used with Claude Code) to build a PowerPoint from Databricks data. Instead of chat-only insights, it generated a full editable multi-slide deck with charts. That is the difference between a screenshot of an answer and a boardroom-ready artifact for monthly or quarterly reviews. Figure 3 is the path: governed metrics into a skill, then an editable deck, then an editorial pass.

2. Templates as deliberate customization, not magic

If you customize the skill, it can work from a company template: unpack the structure, then pick and replace slides for the related data. Exact brand fidelity is a skill and asset problem, not a vibe. Budget time to encode the template rules. Do not expect a first prompt to invent your corporate master.

3. App build and deploy with platform-native constraints

App development and deploy inside Databricks is possible from IDEs like Cursor or Claude Code with Databricks MCP, but I still have to point those agents at the right path. Genie Code already has platform context. In my run a one-line prompt was enough for it to create an empty app, plan the work, and open the app page while compute started. After one nudge to continue, it wrote app.py, app.yaml and requirements.txt, deployed through the workspace and apps APIs, and the app reached Running on a Databricks Apps host.

Figure 3. Two artifact paths. Top: governed metrics through a PPTX skill into an editable deck, then editorial review. Bottom: app bound to a Genie space, compute-shape check, host deploy, then CI and identity before broad reach.

Figure 4. A demo app on its details page right after deploy. The compute shape (Medium) and the deployment record (source path, then every step from stop to start) sit on one page, so that is where I copy them from into the deploy runbook.

Running is not the same as working. Genie Code gave the app's service principal SELECT on the table, then stopped and asked before granting USE CATALOG and USE SCHEMA. Until someone approves those, the chart shows "Failed to load data" (Figure 5). That is the default I want from an agent, and it means the grants belong in the runbook, not in a follow-up prompt.

Figure 5. The agent's own summary after deploy. Table access is granted, but the catalog and schema grants wait for my OK, so the app runs and the chart cannot load data yet.

The app does not run as me. It gets its own service principal, and signed-in users go through a separate OAuth client with their own scopes (Figure 6). Decide which identity reads the data before you share the link.

Figure 6. Two identities on one app. The service principal acts for the app; the OAuth client acts for the signed-in user. Pick one on purpose.

The deck run hit the compute-shape wall instead. Serverless compute was not enabled in that workspace, so Genie Code put the python-pptx code in a notebook and ran it once on the existing cluster (Figure 7). Capture both facts, the grants and the compute you actually have, in the runbook so the next person does not rediscover them under demo pressure.

Figure 7. The deck build falls back to a notebook on a classic cluster because serverless is off. The agent explains the pivot instead of failing silently.

The chart in that deck is native, not a picture. In a rerun against a three-segment demo table, the pptx Genie Code built carries the chart as chart XML with an embedded workbook, so the values sit behind the chart and can be edited in place (Figure 8).

Figure 8. The generated deck opened in LibreOffice Impress, cropped below the slide title. The chart's Data Table shows the embedded values (A 42, B 28, C 55), which is what makes it editable rather than a screenshot.

4. Benchmarks as a signal, not a religion

Databricks has reported Genie Code at about 77.1% success on real-world tasks versus about 32.1% for leading coding agents using MCP. Treat that as evidence that native context changes outcomes, not as a guarantee for your Monday ticket. Reproduce the tasks that matter: deck from governed metrics, app bound to a Genie space, deploy on the compute you actually use.

5. Human review stays on the critical path

Editable decks still need a human to check claims against metric definitions. Deployed apps still need auth, secrets, and cost controls. Genie Code shortens the path to a draft. It does not retire ownership.

Common mistakes

Chat answers mistaken for board artifacts.
A correct SQL answer is not a deck. Design the skill path and template rules explicitly. Figure 3 is the minimum shape.

Template hope without template work.
One brand prompt yields a generic theme. Exact masters need encoding and test renders.

Generic agent plus thin MCP mistaken for native context.
You get code, then babysit deploy topology and CLI availability. Figure 2 is that babysitting made visible.

Serverless assumed to be there.
In my workspace it was off, so the deck build fell back to a notebook on a classic cluster. Identity wiring and egress differ too. Capture compute shape in the deploy runbook.

Benchmark theater.
You cite 77.1% and skip smoke tests on your tables, template, and app host.

Unreviewed publish.
An agent can deploy. Gate promotion, app identity, and catalog grants before broad reach. Figure 1 puts review on the critical path on purpose.

Use as-is or adapt

Use as-is

Genie Code allowed for draft decks and prototype apps; production publish follows existing review and CI gates.
Company PPTX templates encoded as versioned skills or assets with a smoke render checklist.
Auto-generated charts must resolve to governed tables or semantic metrics, not pasted CSV extracts.
Deploy runbooks document compute shape, app identity, and the Unity Catalog grants the app's service principal needs; serverless availability is checked, not assumed.
Human review mandatory before external or executive audiences see generated decks.
Compare native assistant vs MCP-from-IDE on a fixed internal task set periodically; do not rely only on vendor percentages.

Adapt before you use

Native assistants reduce context assembly on platform-shaped work. You accept vendor coupling. External coding agents stay flexible across stacks but re-teach warehouse and deploy reality every session.

Skill-based decks are powerful and still fragile around complex masters and semantic charts. Plan an editorial pass. Removing the last human check is how wrong KPIs reach the all-hands pack.

Assistant deploy speeds prototypes. Production still wants CI, reviews, and environment separation. Clear the first fog; do not bypass the release train.

Try it yourself

Both demos are in my public tech-demos repo: a minimal Databricks App you deploy with the CLI and a script that builds a deck with native, editable charts.

This also leaves a question for Copilot in Microsoft Fabric: agent mode with context of Fabric items and data soon? Track that peer question when you compare platform assistants, not only Databricks Genie.

Primary references: Genie Code, Anthropic agent skills docs, Databricks docs for Genie and Databricks Apps, and Databricks materials on Genie Code task success with vs without native platform context.

Context. Artifact. Review. Three steps. Native beats naive when the job is platform-shaped.

Azure Custom Vision: How Do You Train and Run Image Classification in an Azure Function?

Sai Prudhvi Neelakantam — Fri, 13 Nov 2026 10:12:11 +0000

I used AI tools to help draft and edit this post. I checked the facts and the wording, and the original lives at geekindata.com. The diagrams are on that page.

Keep the UI boring; make prediction operable

Key takeaways

Keep the UI boring: upload, predict, show label and confidence.
Validate type, size, and decodability on the server before inference.
Prediction credentials stay on the backend; the browser talks only to the app API.
Treat low confidence as a first-class UI outcome, not a silent top-label win.
Log correlation id, iteration id, latency, and top-N labels on every call.
Publish Custom Vision iterations like app releases: smoke set plus rollback.

Custom Vision can get you from labeled images to a prediction endpoint quickly. The product work starts after the happy path: bad uploads, low-confidence answers, iteration that breaks the published endpoint, and no breadcrumbs when the label is wrong.

Keep the UI boring: upload, predict, show label and confidence. Put the complexity in data quality, validation, request limits, logging, and a controlled publish path for new iterations.

Named model: upload → predict → explain → iterate

Upload: server-side type, size, and decode checks before you spend an inference call.
Predict: app API owns timeouts, retries, and credentials (keys or managed identity).
Explain: UI always shows label plus confidence; low score is a first-class outcome.
Iterate: train, smoke a fixed image set, publish the iteration, keep a rollback path.
Contract: Custom Vision owns the score; the web app owns hygiene, keys, and breadcrumbs.

Figure 1. Upload, predict, explain, iterate. The UI stays simple on purpose. Operability lives in validation, the API boundary, honest confidence, and a publish path you can roll back.

Context and constraint

I have seen too many vision demos that look polished until the first bad upload, the first low-confidence answer, or the first model iteration that breaks the published endpoint. A good demo becomes a product when input validation, request limits, logging, and a path to retrain stay in place from day one.

Custom Vision (or the equivalent Azure image classification path you choose under Microsoft Foundry / Azure AI Services) should own prediction. The web app should own upload hygiene, auth if you need it, presentation of labels and confidence, and the operational breadcrumbs that let you debug wrong answers later. If the UI tries to be clever, people stop trusting the score. If the backend skips basics, you cannot operate the cleverness you shipped.

Figure 2 is the Learn surface that makes the user contract visible: labels and confidence, not a second story invented in the app shell.

Figure 2. Labels and confidence are the user contract. The training UI is where iteration happens; the web app should not invent a second story. Source: Microsoft Learn: Custom Vision overview.

Figure 3 is the project home where iterations live. Treat a published endpoint as a release artifact, not as a casual overwrite after lunch.

Figure 3. Iteration lives in the project surface. Treat published endpoints as release artifacts, not as a casual overwrite. Source: Microsoft Learn: Custom Vision overview.

1. Start with the user flow, not the model card

The journey should be obvious: choose an image, submit, see a result. Minimal layout keeps focus on the core interaction. Extra toggles, advanced panels, and decorative charts can wait until the happy path is boringly reliable. If the flow is complicated, the AI feature feels harder than it is. Figure 1 is that boring path drawn as four jobs.

2. Validate before you spend an inference call

Check file type, size, and basic image decodability on the server. Reject clearly. Do not stream arbitrary blobs to the prediction endpoint and hope. Validation protects cost, protects the service from garbage, and gives users a faster failure than a timeout wrapped as something went wrong.

3. Wrap prediction in boring, predictable backend behavior

Timeouts, retries for transient faults, and mapped error responses belong in the API layer. Return plain-language failures when inference fails. Show label and confidence in plain language when it succeeds. Low confidence is a first-class outcome: say so in the UI instead of pretending the top label is gospel.

4. Keep secrets and keys out of the browser

Prediction keys and training keys belong on the server or in managed identity patterns, not in frontend bundles. A static page that calls Custom Vision directly is a short path to key leakage and uncontrolled quota burn. The browser uploads to your app; your app calls Azure. Figure 4 is the ownership split that keeps the key out of view-source.

Figure 4. Browser talks to your app. Your app talks to Custom Vision. Keys and managed identity stay on the server. A browser-side key is how quota burn gets a silent owner.

5. Log enough to debug bad results

Store correlation ids, model name or iteration id, latency, top labels, and whether the user accepted or corrected the result if you collect that. You do not need the raw image forever in every environment, but you need a path to reproduce class of failures. Without iteration ids in logs, root cause becomes folklore after the next publish.

6. Treat Custom Vision iteration as a release

New tags, new training images, and a newly published iteration change product behavior. Promote iterations the way you promote app builds: note what changed, smoke-test a fixed image set, and keep a rollback to the previous published iteration. Narrow classifiers drift when the world changes. Plan for that drift instead of discovering it in a stakeholder meeting. Figure 5 is the release shape.

Figure 5. Train, smoke, publish, live. The dashed return is rollback to the previous published iteration. Log the iteration id on every prediction so a bad publish is diagnosable.

Common mistakes

Demo happy path only.
PNG of a clear object on a white background. Real users upload HEIC phone photos, screenshots, and five-megabyte scans. The app chokes or the model shrugs.

Confidence ignored.
Always show the top label in bold. Users trust 0.41 the same as 0.97. That is how wrong operational decisions get a nice UI.

Keys in the browser.
Someone views source, copies the prediction key, and your bill explains itself next month. Figure 4 is the failure made visible.

No iteration discipline.
Retrain in the portal, publish, and hope. Yesterday's smoke images now misclassify and nobody can say which publish did it. Figure 5 is the discipline that prevents that.

Logging without model identity.
You know the request failed or looked weird. You do not know which iteration served it. Support becomes archaeology.

Unbounded uploads.
No size limit, no rate limit, no auth on an internal tool that somehow became external. Inference cost and abuse arrive together.

Use as-is or adapt

Use as-is

Server-side validation for type, size, and decodability before prediction.
Prediction credentials only on the backend; browser talks to the app API.
UI always shows label plus confidence; low-confidence threshold is product-defined.
Logs include correlation id, iteration or model id, latency, and top-N labels.
Published iterations are treated as releases with a smoke set and rollback.
Rate limits and auth posture documented for internal versus public exposure.
Retraining ownership and data retention rules written before the first production upload.

Adapt before you use

Custom Vision is fast to stand up for a narrow task. It is not a substitute for a carefully engineered vision stack when you need detection, segmentation, or strict latency at the edge. Choose it when labeled examples and a classification answer are enough for the product decision.

A thin web app is easy to explain and easy to replace. A thicker product shell (auth, history, human review queue) takes longer but is what internal tools often need after week two. Do not pretend the upload page is the whole system if reviewers must correct labels for retraining.

Hosting the model via Azure versus exporting for edge runtimes changes your update story and your offline story. Pick based on where images are allowed to travel, not based on which quickstart you pasted first.

Primary references: Microsoft Learn on Custom Vision, image classification, and Azure AI / AI Services authentication patterns for keeping keys off the browser. Prefer current Microsoft Foundry naming when your deploy target has moved under that umbrella.

Upload. Predict. Explain. Iterate. Four jobs. One score the user can actually trust.

Pose Estimation: MediaPipe or YOLOPose for Real-Time Pose Classification? (a real-time vision loop with its own latency budget)

Pose Estimation: MediaPipe or YOLOPose for Real-Time Pose Classification?

Sai Prudhvi Neelakantam — Fri, 13 Nov 2026 10:04:10 +0000

I used AI tools to help draft and edit this post. I checked the facts and the wording, and the original lives at geekindata.com. The diagrams are on that page.

Pose estimation is a systems problem, not a model bake-off

Key takeaways

Treat pose as a continuous capture → infer → render loop under one latency budget.
Measure p50 and p95 end-to-end on the demo machine, not only model milliseconds.
Separate inference work from overlays so drawing cannot stall the next frame.
Name a frame-drop policy under load (prefer drop oldest for live UX).
Pick MediaPipe-style vs YOLOPose-style for systems shape, not a bake-off slide.
Write capture conditions (light, distance, people count) into the product spec.

Point a camera at a person, draw a skeleton, call it live. The clip demo survives. The live stream often does not: lighting swings, frames drop, backgrounds get busy, and people feel 80 ms of lag even when they cannot name the number.

Keypoint accuracy without a calm loop is a lab number. What users experience is time from frame grab to a stable overlay. Capture, inference, and render are three jobs. Each can fail while the other two look fine.

Named model: capture → infer → render

Capture: camera or stream with a timestamp at frame grab.
Preprocess / infer: resize once, run keypoints on their own worker.
Render: overlays and confidence rules that never block the next grab.
Contract: one shared latency budget (capture timestamp to overlay), plus an explicit backpressure and drop policy when the queue fills.

Figure 1. Capture, preprocess, infer, render. Solid arrows move the frame forward. The return path is the next frame. The band under the stages is the latency budget users actually feel.

Context and constraint

I treat pose demos as product exercises. Frames have to enter the pipeline. Inference has to finish on time. Results have to present without lying about confidence. If any of those jobs lags, the experience feels unreliable even when the offline keypoint table looks strong.

You are not only detecting joints. You are shipping a stream processor that must stay responsive when conditions change. Optimize the loop. Model milliseconds on a warm GPU stay a lab number until you measure them on the same camera and hardware the app will use.

MediaPipe-style landmark runtimes and YOLOPose-style detect-then-pose stacks answer different ownership questions. Figure 2 is the shape difference, not a leaderboard. Lean local landmarks keep fewer stages in the loop. Detect-then-pose buys multi-person association and costs more stages to budget and observe. Pick the shape that matches your latency and ownership budget before you argue mAP.

Figure 2. Systems tradeoff, not a bake-off. Left is a lean landmark path. Right adds person detect before pose. More stages means more places for queues and association errors to hide.

The same discipline shows up in other vision products. Figure 3 is an Azure AI Vision image-analysis surface from Microsoft Learn. Boxes and tags are a different output. The lesson still transfers: keep the frame path explicit. Pose work swaps boxes for landmarks and still fails when presentation flickers or the path is opaque.

Figure 3. Vision pipelines succeed when the frame path stays explicit: capture, analyze, present. Pose work has the same loop, just with landmarks instead of boxes. Source: Microsoft Learn: image analysis overview.

Design

1. Budget the frame path first

Set target resolution, target FPS, and maximum end-to-end latency before you reach for a heavier backbone. Resize only as much as the model needs. Skip redundant color conversions and double copies between CPU and GPU. If capture already gives a usable size, do not upscale for aesthetics and then downscale for inference. Figure 1 is the budget surface: every stage eats from the same band from t0 grab to t3 overlay.

2. Separate inference from visualization

Keep the inference worker off the render path. Skeleton drawing, trail smoothing, and fancy overlays must not block the next frame grab. When visualization shares a thread with inference, a pretty effect becomes a latency bug. Clear overlays beat decorative ones when confidence flickers.

3. Measure end-to-end, in the real environment

Instrument capture timestamp, preprocessed frame ready, inference done, and frame presented. Track p50 and p95, not only averages. Measure on the laptop or edge hardware you will demo on, with the same camera and similar ambient load. A desktop GPU number does not forgive a thermal-throttled demo machine.

4. Name backpressure before the queue grows

Live systems need an explicit rule for what you discard under load. Figure 4 contrasts the live choice (drop oldest, keep presenting fresh frames) with the accidental failure (drop newest, let the queue go stale while capture still looks busy). Bound queue depth. Log drops. Do not invent the policy in a hallway demo.

Figure 4. Backpressure needs a named drop policy. Prefer drop oldest for live UX. Dropping newest is how overlays lag while the camera path still looks busy.

5. Design for unstable confidence

Pose models wobble on occlusions, odd angles, and fast motion. Decide how the UI behaves when confidence drops: hold the last good pose briefly, fade the skeleton, or mark low-confidence joints. Silent jitter reads as broken AI. Explicit uncertainty reads as an honest system.

6. Keep the pipeline lean enough to reason about

Fewer stages mean fewer places for backlog to hide. Prefer a boring path: capture, light preprocess, infer, overlay, display. Add tracking, multi-person association, or recording only when you can name the product need. Each extra stage needs its own backpressure story, or you will drop frames in the wrong place and not know why. That is why Figure 2 matters before you add a second detector "for free."

7. Treat demo conditions as requirements

Lighting, distance to camera, clothing contrast, and number of people in frame belong in the spec. Write them down. A pipeline tuned on a clean studio take fails in a conference hallway. If the product must work in hallways, collect failure clips there and tune against those clips, not marketing stills.

Common mistakes

Model-only optimization.
You shave 3 ms off inference and ignore 40 ms of preprocess and copy overhead. The user feels nothing. The slide looks better.

Ideal-case demos.
Perfect light, one person, facing the camera, no occlusion. Stakeholders applaud. The first real user stands side-on under fluorescent lights and the skeleton falls apart.

Shared thread for infer and draw.
Overlay code stalls capture. Apparent FPS collapses. You blame the model.

No backpressure policy.
Queues grow, latency climbs, then you drop newest instead of oldest (or the reverse) without a deliberate choice. Figure 4 is that failure made visible.

UI that hides uncertainty.
Low-confidence joints still draw at full opacity. People trust wrong elbows. Later that trust debt shows up as "computer vision does not work here."

Environment mismatch.
Train and eval on clean datasets. Ship on webcams where auto-exposure fights the face. Domain shift is the default, not a surprise.

Bake-off without a product shape.
You compare MediaPipe and YOLOPose on a static accuracy table and never decide lean single-person versus detect-then-pose multi-person. The slide winner loses on the hardware you actually demo.

Use as-is or adapt

Use as-is

End-to-end latency budget (capture to overlay) with a p95 target on target hardware.
Explicit pipeline stages and ownership: capture, preprocess, infer, render.
Backpressure and frame-drop policy under load (name drop oldest vs drop newest).
Confidence visualization rules for low-quality joints and lost tracks.
Supported capture conditions (lighting, distance, people count) as product requirements.
A demo checklist that matches those conditions, not a studio-only path.
Separation of inference work from UI effects so overlays cannot stall the loop.
Runtime choice framed as systems shape (lean landmarks vs detect-then-pose), not a leaderboard.

Adapt before you use

Lower resolution and simpler models buy latency and stability. You give up fine-grained joint precision. For coaching, ergonomics, or interactive demos, stability usually wins. For offline analysis where you can wait, accuracy can win. Name the product before you argue mAP.

Local inference reduces round trips and privacy surface. Cloud inference can give heavier models and easier updates. Real-time UX rarely forgives a flaky network hop. If you must go to the cloud, budget for jitter and design a degraded local mode, or accept that real-time becomes near-real-time with retries.

Multi-person tracking multiplies association errors and CPU cost. Ship single-person well before you promise a crowd. A crisp single skeleton beats three flickering ones in every stakeholder review I have run.

Primary references: docs for the pose runtime you actually ship (for example MediaPipe Pose or an equivalent keypoint stack), OpenCV capture and timing guidance, and the model card for keypoint definitions and known failure cases. Prefer vendor docs for your deploy target, not a blog benchmark from a different GPU. For the Learn surface in Figure 3, see Microsoft Learn: image analysis overview.

Capture. Infer. Render. Three stages. One latency budget. Honesty when confidence drops.

Azure Custom Vision: How Do You Train and Run Image Classification in an Azure Function? (the same honesty about low confidence, behind an Azure Function)

Fabric Airflow: Is It Possible to Use Apache Airflow with Fabric?

Sai Prudhvi Neelakantam — Fri, 13 Nov 2026 10:03:35 +0000

I used AI tools to help draft and edit this post. I checked the facts and the wording, and the original lives at geekindata.com. The diagrams are on that page.

Keep Fabric executing; put Airflow on schedule and graph

Key takeaways

Fabric executes the work; Airflow owns schedule, graph, retries, and alerts.
Success means Fabric terminal state, not trigger acceptance.
Prefer many thin tasks named for the pipeline step.
Put quality gates in the DAG where they are visible.
Do not rebuild Fabric transforms inside Airflow operators.
Reach for Airflow when cross-job dependencies need an honest graph.

Yes, when you treat them as two planes. Fabric should do the work. Airflow should own the schedule, the dependency graph, retries, and the on-call story of the run.

Airflow earns its place when a Fabric environment looks fine in the portal and still falls apart because three jobs have to finish before a fourth can start. Native schedules handled single jobs. They did not explain a graph. Collapsing coordination into notebooks made every change feel riskier than the transform itself.

Named model: two planes

Execution plane (Fabric): notebooks, pipelines, lakehouse work, warehouse ops, semantic refresh.
Orchestration plane (Airflow): schedule, dependencies, retries, alerts, run history humans can read.
Contract: each Airflow task waits for Fabric terminal state, not trigger acceptance.
Shape: many thin tasks named for the pipeline step; quality gates are visible DAG steps.

Figure 1. Airflow coordinates, Fabric executes. The solid arrow is wait for terminal state. The dashed return is status, run id, and alert text. Not fire and forget.

Context and constraint

Fabric notebooks and pipelines are strong at doing work. They are weaker as the calendar and dependency graph for everything around them. The moment the path is land, validate, transform, publish, you need an orchestration plane that is not pretending to be another transform engine.

Airflow fits when you want an honest DAG. Keep tasks thin. Call Fabric. Do not rebuild Fabric inside operators because it was convenient on a Friday. If you start reimplementing transforms in Airflow Python, you have created a second source of truth. Convenience is how orchestration layers go thick and then opaque.

Fabric Airflow jobs make that split concrete in the product surface. Figure 2 is the Learn tile that starts a DAG whose job is to run a Fabric item, not to host the Spark logic. The orchestrator stays thin on purpose.

Figure 2. Start from Run Fabric Artifact when the DAG should trigger notebooks or pipelines without stuffing transform logic into the orchestrator. Source: Microsoft Learn: run a Fabric item using Apache Airflow.

Design

1. One DAG, thin tasks, honest names

Map the flow so a new hire can read it without a walkthrough: land, validate, transform, publish, notify. Each task should do one clear thing. If a task grows into everything after bronze, you lose clean retries and you lose the ability to say which step actually failed. Prefer many small operators that call Fabric over one giant operator that hides a private workflow. Figure 3 shows the shape difference.

Figure 3. Thin tasks keep retries and blame local. One fat bronze-to-gold operator turns a gold failure into a full rerun.

2. Fabric executes; Airflow coordinates

Trigger Fabric jobs, pipelines, or notebooks from Airflow. Do not copy the Spark logic into Airflow Python for speed. The execution plane stays in Fabric where capacity, identities, and lakehouse bindings already live. Airflow should pass parameters, wait for completion, and react to status. When status is ambiguous, fix the Fabric job contract. Do not paper over it with longer sleeps. Figure 1 is the ownership split; Figure 2 is the product entry point that keeps transforms out of the DAG.

3. Explicit success and failure contracts

Every task needs a definition of done that is checkable: row counts, file presence, quality gate pass, pipeline status, semantic refresh state. "It ran" is not a contract. Partial success without a named gate is how gold tables get refreshed from incomplete silver. Put the gate in the DAG where it is visible, not only in a notebook comment. Figure 4 is the contract that matters most between the planes.

Figure 4. HTTP 200 from the trigger is not done. Wait for Fabric terminal state, then mark the Airflow task success or fail.

4. Retries that match the failure type

Transient platform blips deserve automatic retry with backoff. Bad data and broken schemas do not. Treat validation failures as terminal for that run unless you have a deliberate quarantine path. Blind retries on quality failures burn capacity and hide the real incident under a green looking retry storm.

5. Observability that names the step

Operators, run ids, and alert text should point at the failing Fabric artifact and the DAG task in the same message. A ping that says "pipeline failed" without the task id and the Fabric run url forces people to reconstruct context under pressure. Design the alert as part of the DAG, not as an afterthought in a shared mailbox rule.

Figure 5 is the monitoring surface that makes that contract readable: task id, operator, status, and duration for a Fabric item run. When the DAG waits for terminal state, this view and the Fabric run history should agree.

Figure 5. Tasks view for a Fabric item run: task id, operator, status, and duration in one place. Source: Microsoft Learn: run a Fabric item using Apache Airflow (Monitor your DAG).

6. Environments as parameters, not forks

Dev, Test, and Prod should be the same DAG shape with different Fabric targets, identities, and schedules. Copy pasted DAGs per environment drift on the first hotfix. Parameterize workspace, lakehouse, and connection ids the same way you would parameterize a Fabric deploy. Pretty names change. Ids should be the contract.

Common mistakes

Orchestrator does the transform.
Airflow notebooks or Python operators slowly recreate what Fabric already owns. Now you have two places to patch logic and no single owner. Figure 2 exists so you do not take that path by default.

One fat task for the whole medallion path.
Failure at gold forces a full rerun from bronze. Time and cost explode. Partial recovery becomes folklore. Figure 3 is the blast radius made visible.

Hidden dependencies outside the DAG.
A semantic model refresh that someone runs after lunch is still a dependency. If it is not in the graph, on-call will learn about it from a broken report, not from Airflow.

Success means HTTP 200 from a trigger.
The trigger accepted the job. The job later failed. Your DAG already moved on. Wait for terminal state, or you will publish lies. Figure 4 is the weak contract versus the strong one.

Secrets and identities living in two places.
Airflow connections and Fabric workspace identities disagree. Runs work on one laptop schedule and fail in the shared environment. Centralize how service principals and key vault references are bound.

Alerts without a step name.
A mailbox ping that cannot name the DAG task and the Fabric run leaves people reconstructing context under pressure. Figure 5 is the minimum surface the alert should point back to.

Use as-is or adapt

Use as-is

Fabric owns data plane work; Airflow owns schedule, dependency, retry, and alert contracts.
Tasks are single-purpose and named for the pipeline step, not for the tool call.
Terminal state of the Fabric job is the task success condition, not trigger acceptance.
Quality gates are explicit DAG tasks with defined fail behavior.
Environment targets are parameterized; no forked DAGs per stage.
Alerts include DAG task id, Fabric run url, and the artifact that failed.
No transform logic duplicated into Airflow operators without a written exception.

Adapt before you use

Airflow adds another runtime to operate: workers, schedulers, package versions, and network paths into Fabric APIs. That cost is real. For a single nightly notebook with no dependents, Fabric native scheduling may be enough. Reach for Airflow when cross-job dependencies, shared SLAs, or multi-system coordination show up.

You will also negotiate ownership. Platform teams often want Fabric-native orchestration. Data engineers often want DAG readability and ecosystem operators. Pick based on the dependency graph you must explain, not on which demo looked smoother in a workshop.

API throttling matters when many DAGs poke the same Fabric environment. Set concurrency limits before you scale hourly jobs.

A good next read once the orchestration contract is honest: lakehouse topology choices. Start with One lakehouse with schemas vs many lakehouses when workspace and schema boundaries are the next decision.

Primary references: Microsoft Learn on Fabric Data Factory and pipelines, notebooks in Fabric, run a Fabric item using Apache Airflow, and Apache Airflow docs on DAGs and tasks.

Workload. Schedule. Contract. Three concerns. Two tools. One readable DAG.

Fabric or Databricks? How I Decide

Sai Prudhvi Neelakantam — Fri, 13 Nov 2026 08:56:20 +0000

I used AI tools to help draft and edit this post. I checked the facts and the wording, and the original lives at geekindata.com. The diagrams are on that page.

Not a feature war. Five questions I ask before choosing Fabric, Databricks or both, where each choice bites a year later, and when both is honest.

Key takeaways

Both platforms can run a medallion lakehouse. Feature lists rarely decide; people, consumers, governance and the billing model do.
If Power BI is the main consumer and the team thinks in SQL and reports, Fabric starts ahead. If the team ships code with tests and CI, Databricks does.
A Fabric capacity is shared. A heavy background job can push report users into delays and rejections.
Databricks scales to zero, but someone has to own compute policies, or the bill spreads across many small decisions.
"Both" works when each table has one owner and data flows one way. A mirrored catalog does not carry Unity Catalog grants.

Clients rarely ask which platform has more features. They ask which one they will regret less. The honest answer depends less on the platforms than on the people who will run the thing in its second year.

I deliver on both. The medallion patterns I use, SCD2 history in bronze, current-row silver, dimensional gold, look the same on each. What changes is everything around them: who writes the code, how it deploys, who governs access, and how the cost behaves.

Named model: five questions, in this order

People: who builds and runs it in year two?
Consumers: what reads the gold layer, and is that Power BI?
Footprint: where does the data already land, and what is already governed?
Cost: a shared capacity you reserve, or compute you pay for per use?
Edge: is there streaming or machine learning that sets the bar?

Figure 1. Each question tilts; none decides alone. Two strong tilts in opposite directions is the signal for both.

Context and constraint

This is a framework, not a scorecard. I do not weight the questions or add them up; I look for the one or two that are not negotiable for this client, and let those lead. Product status below is as of October 2026, with Learn links where it matters.

Design

1. People: who runs it in year two

Fabric lowers the entry bar: pipelines, Dataflows, notebooks and Power BI in one workspace and one place to look. That is a real advantage for a team of BI developers and analysts. The bite comes later: low-code workspaces grow fast and deploy badly unless someone brings Git discipline early.

Databricks assumes engineers: code in a repo, bundles, tests, a pipeline. Hand that to a team without engineers and it does not get simpler, it gets abandoned, and notebooks get edited in the workspace.

So my first question is not about data at all: who on the client side can review a pull request?

2. Consumers: is Power BI the point?

In most of my projects, Power BI is where the business meets the data, whichever platform sits underneath. If it is the main consumer, Fabric removes a hop: Direct Lake reads Delta tables in OneLake without an import refresh, and the semantic model lives next to the data.

Databricks data can still reach Direct Lake. A mirrored Azure Databricks catalog in Fabric is generally available: Fabric mirrors the catalog structure, reads the tables through shortcuts without copying them, and you can build Direct Lake reports on it. Changes can take seconds to minutes to appear. Plan for two prerequisites: the EXTERNAL USE SCHEMA privilege on the Unity Catalog side, and network access from Fabric to the storage account behind the catalog (tutorial). Publishing a catalog to OneLake from the Databricks side is a second route, still in Public Preview (Fabric with Azure Databricks).

If the gold layer also feeds applications, APIs, other teams' notebooks and models, Power BI is one consumer among many and stops being the deciding factor.

3. Footprint: what is already governed

Where the data lands today matters more than where it could land. A client with an existing Unity Catalog setup, groups and grants already in place, should not rebuild that governance in another product to save a hop for reports. A client whose sources already land in Microsoft services, with no lakehouse yet, has nothing to migrate away from.

The bite is governance in two places. Unity Catalog grants do not travel with a mirrored catalog: the same tutorial tells you to set access in Fabric's own permission model, and tables with row filters or column masks are not mirrored at all (limitations). On the Fabric side, OneLake security is generally available, but its roles do not restrict workspace Admins, Members and Contributors (OneLake row and column security). Workspace roles are a security decision, not a convenience.

Figure 2. The OneLake role grants Read on one schema, and it is set on the lakehouse rather than the workspace.

Figure 3. Members are added to the OneLake role separately from workspace access, and workspace Admins, Members and Contributors are not restricted by it.

4. Cost: reserved and shared, or per use

No numbers here, because they age badly and depend on your agreement. The shapes are what matter.

A Fabric capacity is reserved compute shared by every workload on it. Smoothing spreads background jobs over the next 24 hours, which makes steady loads pleasant. Overuse it, though, and Fabric first delays and then rejects interactive requests, and report users are the interactive requests (throttling). One oversized nightly job becomes a morning of slow reports. Separate capacities, or on-demand billing for Spark, which takes Spark off the capacity, are the usual fixes.

Databricks bills compute as it runs, and idle compute can cost nothing. That suits spiky loads. The bite is ownership: without compute policies, auto-termination and tags, the bill is the sum of many small choices nobody reviews.

5. Edge: streaming and machine learning

Both platforms stream and both do machine learning. The question is how deep. Code-first ML with its own lifecycle, feature pipelines and model serving leans Databricks. Event data that business users explore and alert on leans toward Fabric's Real-Time Intelligence. If neither is on the roadmap, do not let a hypothetical one pick the platform.

6. When the honest answer is both

Two strong tilts in opposite directions, for example an engineering team with an existing Unity Catalog setup and a business that lives in Power BI, usually mean both. The pattern I trust: Databricks owns engineering and the tables; Fabric mirrors the gold catalog and owns semantic models and reports.

It works if three rules hold. Each table has one owner and one writer. Data flows one way per layer. And access is designed twice on purpose: Unity Catalog for the engineering side, Fabric permissions and OneLake security for consumers, reviewed together.

The reverse direction exists too. Azure Databricks can read Fabric Lakehouse and Warehouse tables through OneLake catalog federation, read only, with several Fabric tenant settings to enable. Writing from Databricks into OneLake through an external location is in Beta, and Learn warns that any non-viewer in that Fabric workspace can read, write and delete the data. I would not build a two-way flow on that yet.

Figure 4. One owner per table, one direction per layer, two permission models designed on purpose.

Figure 5. Pick the bites you can staff for.

Common mistakes

Choosing by feature list. Both have the feature; the question is who will run it.
One capacity for engineering and reports. The nightly job slows the morning.
Databricks with no engineers. It ends up as edited notebooks and an unowned bill.
Assuming mirrored means governed. Unity Catalog grants stay behind; masked tables do not mirror.
Workspace Contributor as a convenience. It bypasses OneLake security roles.
"Both" without table ownership. Two writers, two truths.
Letting a future ML project decide today. Decide on what is on the roadmap.

Use as-is or adapt

Use as-is

The five questions, people first.
One owner and one writer per table when you run both.
Separate heavy engineering from report capacity, or take Spark off it.
Designing access on each side explicitly, never assuming it carries over.

Adapt before you use

The order of consumers and footprint, when what already exists is large.
How you split capacity, by workload or by environment.
Whether reports read a mirrored catalog or a copied gold layer, given sync latency and the row filter and column mask limits.
Whether you need the reverse flow at all.

Try it yourself

The Fabric and Databricks demo objects for this post are in my public tech-demos repo, with a bundle and a compute policy you can deploy to rebuild the Databricks side.

Pick the platform your team can run, then design the seams on purpose.

DEV Community: Sai Prudhvi Neelakantam

Databricks Genie: Can It Build My Deck and Deploy My App?

Genie Code finishes platform-shaped jobs when context is already in the room

Context and constraint

Design

1. Deck generation as a skill-backed workflow

2. Templates as deliberate customization, not magic

3. App build and deploy with platform-native constraints

4. Benchmarks as a signal, not a religion

5. Human review stays on the critical path

Common mistakes

Use as-is or adapt

Try it yourself

Related

Azure Custom Vision: How Do You Train and Run Image Classification in an Azure Function?

Keep the UI boring; make prediction operable

Context and constraint

1. Start with the user flow, not the model card

2. Validate before you spend an inference call

3. Wrap prediction in boring, predictable backend behavior

4. Keep secrets and keys out of the browser

5. Log enough to debug bad results

6. Treat Custom Vision iteration as a release

Common mistakes

Use as-is or adapt

Related

Pose Estimation: MediaPipe or YOLOPose for Real-Time Pose Classification?

Pose estimation is a systems problem, not a model bake-off

Context and constraint

Design

1. Budget the frame path first

2. Separate inference from visualization

3. Measure end-to-end, in the real environment

4. Name backpressure before the queue grows

5. Design for unstable confidence

6. Keep the pipeline lean enough to reason about

7. Treat demo conditions as requirements

Common mistakes

Use as-is or adapt

Related

Fabric Airflow: Is It Possible to Use Apache Airflow with Fabric?

Keep Fabric executing; put Airflow on schedule and graph

Context and constraint

Design

1. One DAG, thin tasks, honest names

2. Fabric executes; Airflow coordinates

3. Explicit success and failure contracts

4. Retries that match the failure type

5. Observability that names the step

6. Environments as parameters, not forks

Common mistakes

Use as-is or adapt

Related

Fabric or Databricks? How I Decide

Key takeaways

Context and constraint

Design

1. People: who runs it in year two

2. Consumers: is Power BI the point?

3. Footprint: what is already governed

4. Cost: reserved and shared, or per use

5. Edge: streaming and machine learning

6. When the honest answer is both

Common mistakes

Use as-is or adapt

Try it yourself