DEV Community: Only Machua

Manage your stacks

Only Machua — Tue, 07 Jul 2026 23:05:41 +0000

Only Machua

Jul 7

PowerBi is smart but also lazy

#data #dataengineering #microsoft #performance

2 min read

PowerBi is smart but also lazy

Only Machua — Tue, 07 Jul 2026 23:04:18 +0000

Introduction

For the past few days I have been doing data cleaning using PowerBi 34 columns to be precise.Data cleaning and transformation to me always seemed like a light task compared to other data related tasks.To my surprise it was the most obvious simple-complicated task after I experienced two stack overflow errors during cleaning.PowerBi was unresponsive which means closing it and all the cleaned data was gone and back to square one cleaning again.I felt like a data janitor literally. With that it made me dive into this "stack overflow error" to avoid this kind of problems again.

What is stack overflow first?

Stack overflow:Its when a program has exceeded memory usage on the call stack when too many functions are called without returning the required output which causes the stack to fill up and overflow.

Stack overflow error in PowerBi

A stack overflow error in powerBi occurs when there is an infinity loop. In this case PowerBi is trapped looking for instance a calculation e.g Calculation B is looking at Calculation C and C is also looking for B to get an output. This will force PowerBi to crash to Protect itself since its in a infinite loop that will eventually lead to the overflow.

What causes this?

With this case it happened during Data transformation in Power Query.

In a query a step might be self referencing itself
step 1=step 1
This causes PowerBi to evaluate the dependency between the two until it overflows since it keeps referencing to the same thing leading to an endless loop
- Complex nested Transformations in excess: In my PowerBi,I had close to 300 chain of transformations and this lead to the creation of a deep expression tree. This was caused by:
- A lot of nested transformation functions i.e Table.Transform columns()
- Repeated Replace value functions i.e Table.Replace value()
- Deep nested if...and...elsestatements which leads to formation of a huge expression that is difficult for Power Query to evaluate.

How do we fix it?

Loading the entire data you are working on directly to your local memory(RAM) of your computer using the Table.Buffer() function.[This is very important since Power Query operates in a "lazy-smart" way through "lazy evaluation" which means it does not process data until you click Apply. If you were in a deep expression tree like me Power Query crashes and you lose all the transformed data.Quite frustrations there]
During data transformation you can decide to break it into smaller executable queries.

The `Table.Buffer()` function;

After finding out about this function and what it does, I had to dig deeper and see what else it can do and found some interesting areas it can be used:

Before performing a complex Join i.e Merging a large table into another large table, the Table.Buffer(Table_Name) function will help in speeding up the merging process.
In web scraping or pulling from a stubborn API the function will help Power Query hit the web sources at once and locks the data.Its very important since you will avoid spamming the API with requests every time.

Linux Fundamentals for Data Engineering

Only Machua — Wed, 24 Jun 2026 12:23:16 +0000

Introduction:

Two weeks into Data Engineering and I dived into the basics of Linux as an operating system and also the fundamentals of it in data engineering. While researching on how linux is used in data engineering, it got more interesting as it's also used in crucial areas such as pipelines, handling and manipulating files, not forgetting how it's used widely in the cloud.

Table of contents

Commands used in navigationFile Operations
Directory Operations
File Viewing and Editing
File Permissions
Search and Pattern Matching
System Information
Data Processing Commands

**Commands used in Navigation

**
To change from one file directory to the other we use the command

cd

followed by your directory

Go back to home directory

cd ~

Go up one level

cd ..

_Tip: If you ever get lost on which file you're working on pwd shows your current working directory which saves you a lot of time.

Listing directories

List files to see the files in your folder
ls

Listing with permissions and timestamps
ls -l

Sort by time (newest first)
ls -lt

Sort by time (oldest first)
ls -ltr

Show hidden files
ls -a

## File Operations

Creating and Copying Files

Create empty file

touch linux.csv

Copy file

cp linux.csv Desktop

Copy with preservation of metadata.

cp -p linux.csv Desktop

Copying directory recursively

cp -R /Desktop /Linux_project

Files Removal

Remove only the file

rm linux.csv

Removing a file directory and its contents

rm -r Linux_project

Force removing a file

rm -f linux.csv

Viewing and Editing

Viewing File Contents

View entire file

cat linux.csv

View first 10 lines

head linux.csv

View first n lines

head -n 20 linux.csv

View last 10 lines

tail linux.csv

View last n lines

tail -n 20 linux.csv

Follow log file in real-time (Important for monitoring data pipelines)

tail -f pipeline.log

File Permissions

Understanding Permissions

Permission structure:

r (read) = 4
w (write) = 2
x (execute) = 1

View file permissions

ls -l

Output: -rw-r--r-- 1 data datagrp 1024 June 6 10:00 linux.csv

Change permissions

chmod 644 linux.csv

# Owner: rw-, Group: r--, Others: r--

chmod 755 script.sh

` # Owner: rwx, Group: r-x, Others: r-x File Editing

Basic vim commands:

i - enter insert mode
esc- exit insert mode
:w - save
:q - quit
:wq - save and quit
:q! - quit without saving

Opening a file in vi editor

vi linux.csv

*## Search and Pattern Matching *

Finding Files

Finding files by name
find /data -name "*.csv"
Find and execute command
find /data -name "*.tmp" -exec rm {} \;
Find files modified in last 24 hours
find /data -mtime -1
Using grep
Recursive search in directory
grep -r "FAILED" /logs/
Search for pattern in file
grep "ERROR" pipeline.log

Count occurrences
grep -c "SUCCESS" pipeline.log
Case insensitive search
grep -i "error" pipeline.log

Data Processing Commands

Sorting data
sort linux.csv > sorted_linux.csv
Count lines in file
wc -l linux.csv
Removing duplicates
sort linux.csv | uniq > unique_linux.csv

Data Transformation

Extracting specific columns (using cut)
cut -d',' -f1,2 linux.csv > subset.csv
Replacing text
sed 's/old/new/g' linux.csv > modified.csv
Filter rows
awk -F',' '$3 > 2000' linux.csv > filtered.csv

Pipelines Operations

Checking file counts
ls -1 /data/input | wc -l
Monitoring disk usage
du -h /data/warehouse
Verifying file integrity
md5sum linux.csv > checksum.txt
Archiving and File compression
Create tar archive
tar -czf archive.tar.gz /data/files/
Extract tar archive

tar -xzf archive.tar.gz
Compress files

gzip large_file.csv

DEV Community: Only Machua

Manage your stacks

PowerBi is smart but also lazy

PowerBi is smart but also lazy

Introduction

What is stack overflow first?

Stack overflow error in PowerBi

What causes this?

How do we fix it?

The Table.Buffer() function;

Linux Fundamentals for Data Engineering

*Introduction: *

**Commands used in Navigation

Listing directories

Files Removal

Viewing and Editing

Viewing File Contents

*File Permissions *

Understanding Permissions

Permission structure:

Basic vim commands:

Opening a file in vi editor

Finding Files

Data Processing Commands

Data Transformation

Pipelines Operations

The `Table.Buffer()` function;

Introduction:

File Permissions