DEV Community: Paulet Wairagu

QN : Get started with lakehouses in Microsoft Fabric

Paulet Wairagu — Thu, 04 Jun 2026 17:08:23 +0000

A lakehouse is a unified platform that combines:
- The flexible and scalable storage of a data lake
- The ability to query and analyze data of a data ware*house*
A lakehouse uses Apache Spark and SQL compute engines to process and analyze data at scale.
Traditional Warehouses handle structured data but struggle on semi-structured and unstructured data from app logs , IoT devices etc hence data silos and complex integration efforts
Data Lakes offer flexibility and scalability but lack structure and performance for b/s analytics
Data Warehouses have strong analytical capabilities but struggle with different data formats and costly to scale.
Lakehouse design:
- tables : delta lake table that provide structured, queryable data
  - Support SQL queries through the SQL analytics endpoint
  - Enforce schemas and support ACID transactions
  - Can be accessed in Power BI for reporting
  - Benefit from automatic optimization and maintenance
- files : stores raw or semi-structured data files in their native format
  - Support any file format (CSV, JSON, Parquet, images, documents)
  - Provide flexibility for data exploration and processing
  - Can be staged before transformation into tables
  - Don't enforce schema or support direct SQL queries
Delta Lake is a open source storage layer taht brings reliability to data lakes.
Data is stored in delta format in OneLake storage
Delta Lake advanatges
- ACID Transactions : consistency with frequent reads
- Schema enforcement : validates the data against the table schema
- Time Travel : maintains transaction logs
- Updates and Deletes :
Delta table has parquet data files + transaction logs
This design support batch + straeming workloads
Lakehouse access :
- workspace roles for collaborators who need access to all items in the workspace
- Item-level sharing to grant read-only access for specific needs, such as analytics or Power BI report development
- SQL analytics endpoint supports row-level and column-level security, so you can restrict what specific users see when they query through SQL
- schema-level permissions to control access by business domain
Well-organized lakehouse data becomes the foundation that intelligent experiences across Microsoft Fabric depend on.
investment you make in organizing, naming, and structuring lakehouse data pays dividends beyond your immediate analytics needs. Good data engineering practices in the lakehouse create a reusable foundation for intelligent experiences across the platform.

QN:Introduction to end-to-end analytics using Microsoft Fabric

Paulet Wairagu — Thu, 04 Jun 2026 16:29:34 +0000

Quick Short notes series

Microsoft Fabric is an end-to-end analytics platform that provides a single, integrated environment where data professionals and the business collaborate on data projects. Built on a unified data lake called OneLake, Fabric brings together the tools you need across that entire lifecycle.
Fabric is a unified software-as-a-service (SaaS) platform where all data is stored in a single open format in OneLake. All analytics engines in the platform can access OneLake, ensuring scalability, cost-effectiveness, and accessibility from anywhere with an internet connection.
OneLake is Fabric's centralized data storage architecture that enables collaboration by eliminating the need to move or copy data between systems
OneLake is built on Azure Data Lake Storage Gen2 (ADLS Gen2) and supports various formats, including Delta, Parquet, CSV, and JSON
All compute engines in Fabric automatically store their data in OneLake, making it directly accessible without the need for movement or duplication.
For tabular data, the analytical engines in Fabric write data in delta-parquet format and all engines interact with the format seamlessly.
Shortcuts are references to files or storage locations within OneLake or external data sources, such as Azure Data Lake Storage, Amazon S3, or Dataverse. Shortcuts allow you to access existing data without copying it, ensuring data consistency and enabling Fabric to stay in sync with the source.
workspaces serve as logical containers that help you organize and manage your data, reports, and other assets.
workspace has its own set of permissions, ensuring that only authorized users can view or modify its contents.
Workspaces allow you to manage compute resources and integrate with Git for version control. You can optimize performance and cost by configuring compute settings, while Git integration helps track changes, collaborate on code, and maintain a history of your work.
Fabric administration is centralized in the Admin portal.
In the admin portal you can manage groups and permissions, configure data sources and gateways, and monitor usage and performance. You can also access the Fabric admin APIs and SDKs in the admin portal, which can automate common tasks and integrate Fabric with other systems.
OneLake catalog helps you analyze, monitor, and maintain data governance. It provides guidance on sensitivity labels, item metadata, and data refresh status, offering insights into the governance status and actions for improvement.
Fabric increases collaboration between data professionals by removing data silos and the need for multiple systems.
In Workspace settings, you can configure:
- License type to use Fabric features.
- OneDrive access for the workspace.
- Azure Data Lake Gen2 Storage connection.
- Git integration for version control.
- Spark workload settings for performance optimization

THINKING GAME DOCUMENTARY: MY REVIEW

Paulet Wairagu — Wed, 26 Nov 2025 18:47:57 +0000

So I finally watched AlphaGo, that documentary about the Google DeepMind AI that took on the world champion in Go, and honestly… I didn’t expect to enjoy it this much. I thought it would be one of those “tech bros hype themselves” films, but wueh, it’s actually deep.

First things first,the game itself. Go is like chess on steroids. Watching those pros talk about it felt like watching athletes explaining how they breathe. The amount of strategy, intuition, and reading the board… it made me respect the game a lot more. And the way the documentary broke it down for normal people? Lovely. Even those of us who have never touched a Go board can follow the tension.

Then there's Lee Sedol. Man, that guy carried the emotional weight of the whole thing. You feel the pressure on him — not just to win, but to defend human creativity. The guy literally said he felt like he was playing on behalf of everyone. That scene where he loses a game and walks out looking completely defeated? Your heart just sinks. Been there, that feeling of “I did everything and still lost.”

And then AlphaGo. The AI itself almost feels like a character. Quiet, calculating, no hype, just vibes and probabilities. The wild part is when it makes those “impossible” moves that even the Go masters can’t understand. Move 37 especially — the commentators looked like they’d seen witchcraft. Even Sedol was like, “No human plays like that.” That’s the moment I realized AI isn’t just copying; sometimes it’s genuinely creating.

But my favourite part is how the doc doesn’t frame it as “humans vs robots.” It shows how the match changed how humans think. After the loss, the pros started studying AlphaGo games and discovering new strategies. Like the AI unlocked creativity instead of killing it. That hit me because we’re in that same AI era now — people thinking AI is coming to take all jobs, yet here we are, learning new ways of thinking from it.

Cinematography was also clean — the slow, quiet shots, the close-ups, the music. It’s not rushed or over-dramatic. Just calm, like meditation.

If you like strategy, tech, psychology, or you just want to see a human fight for dignity against a machine, this is a solid watch. It’s not a hype documentary; it’s a thoughtful one. Emotion, tension, and a bit of “eish, surely, how is an algorithm beating a whole world champion?”

Highly recommend.

The 5 Most Common Data Quality Issues (and How Analysts Can Fix Them)

Paulet Wairagu — Mon, 24 Nov 2025 10:50:56 +0000

Data analysts spend more time cleaning data than analyzing it. In fact, in most real-world projects, 60–80% of your time goes into preparing data for meaningful insights.
Poor data quality leads to incorrect conclusions, broken dashboards, and bad decisions which is why understanding common issues and knowing how to fix them is a core skill for every analyst.

Here are the five most common data quality problems and practical steps to solve each one.

1. Missing or Null Values

Missing data can distort metrics, create gaps in reports, or lead to inaccurate ML models.

Causes:
• Manual data entry errors
• Incomplete integrations
• System migration issues

How to fix it:

Identify missingness patterns using COUNT(*) in SQL or df.isna().sum() in Python.
Drop rows only when missing data is irrelevant.
Impute using averages, medians, or domain logic.
Use Power Query’s “Replace Errors” or “Fill Down” functions for structured fixes.

2. Inconsistent Formatting

You’ve probably seen this: “Kenya”, “kenya”, “K E N Y A”, or mismatched date formats in the same column.

Why it happens:
• Different data sources
• Manual inputs
• Lack of data validation rules

How to fix it:

Apply standard casing (upper/lower/title).
Convert all dates to a unified ISO format (YYYY-MM-DD).
Use Excel Power Query’s “Transform → Format” options.
In SQL, standardize with functions like UPPER(), TRIM(), or TO_DATE().

3. Duplicate Records
Duplicates inflate counts, break KPIs, and cause incorrect aggregations.

Why it happens:
• Multiple data entry points
• Poor primary key definition
• System sync issues

How to fix it:

Identify duplicates using ROW_NUMBER() windows in SQL.
Use Power Query’s “Remove Duplicates”.
Implement unique IDs early in the pipeline.
In Python, use df.drop_duplicates().

4. Outliers and Incorrect Values

Some values are valid extreme cases; others are simply errors (like a customer aged 600).

Why it happens:
• Typographical errors
• Faulty sensors or scraping issues
• Incorrect units (meters vs. feet)

How to fix it:

Visualize distributions using box plots or histograms.
Apply domain thresholds or rule-based logic.
Use interquartile ranges or z-scores for statistical outlier detection.
Create automated validations in Power BI or SQL.

5. Mixed Granularity

Data at different levels combined into one column or table — e.g., weekly and monthly data in the same dataset.

Why it happens:
• Data integration from multiple systems
• Poorly designed source tables

How to fix it:

Split datasets by granularity before analysis.
Create dimensional tables for dates, products, etc.
Aggregate or disaggregate consistently before joining.
Use a proper star schema when possible.