Data Engineer's Guide to Linux: Why It Is Your Secret Weapon

Grace N Muchiri — Mon, 06 Apr 2026 06:43:33 +0000

If you are stepping into the world of data engineering, you will quickly realize that if SQL and Python are the languages spoken, then Linux is the room they are spoken in. Most modern infrastructure, from cloud servers to Docker containers run on Linux. This read is here to walk you through why Linux matters and help you manage the flow of data by navigating the terminal.

Data engineering involves processing large amounts of data. While Windows and Mac are great for local development, they carry bloat. (GUI updates & background updates which enable smooth rendering of desktop icons). Since Linux is headless, it means that 100% of RAM & CPU goes towards processing queries & scripts, not rendering desktop icons. Other advantages of working with Linux include:-

It gives you granular control over memory and CPU which is important when working with terabytes of data
Automation. It has crontab which is the Linux scheduler. Here you can script repetitive tasks like moving files or checking logs using Bash
It is a cloud standard. AWS, Azure and Google Cloud are built on Linux
It has Security Features

Linux as an operating system has four mandates

Hardware management
File system management
Process management
Security

In this article we will briefly look at commands used in each of the four mandates

Hardware Management

The commands for this mandate help you explore the advantage of "granular control over memory and CPU."

The commands for this mandate help you explore the advantage of “granular control over memory and CPU”. Below are a list of some of the commands used and what they achieve

lscpu - This displays info about the architecture of your CPU
free -h - This shows how much space is available in your CPU. The -h flag stands for human readable (GB instead of bytes)
lsblk – Lists all blocked devices attached to the system
df -h – shows how much disk space is used Vs. what is available on mounted devices

Process Management

Everytime you run a script or python query, Linux interpret that as a process. Managing these processes ensures that one “stuck script” doesn’t lead to crashing of the whole server.

ps aux – Shows a snapshot of every running process on the system
top or htop – this is the primary command used to understand why your pipeline is slow. It gives you a colourful interactive list of what is running. Its is more like task manager in your windows
kill<PID> - Stops a hanging process or one that is consuming too many resources using its process ID (PID)
& - adding this to your script ensures it is running in the background so you can keep usng your terminal ( python3 script.py &)
Jobs – lists the processes you have running in the background of your current session

Security

Linux is a multi-user system. Security and permission features ensure that user A cannot accidentally delete or alter files, folders, or even pipelines for user B.

chmod – This command changes file permissions to show who can read, write, and execute
chown – this command changes the ownership of the files or folder
passwd – allows you to make changes to your password
whoami – this is a quick way to check which user account you are currently running on
history – shows a list of recently executed commands

File System Management

File management involves mainly managing workflow. In this section, we will learn how to manage files: opening them, reading them, moving them from one folder to another, and even giving permissions to the files. We will use screenshots as a practical illustration.

Connecting to the cloud server

In this case, we use ssh root@yourIpAdress

Finding out which user account is running

Here, you use the command whoami

Adding a user successfully

Since Linux is a multi-user this is how to add users

Switching user to the newly created user

Creating a new folder in the homepage of the new user account

Creating subfolders

Creating files in the subfolders

Changing file permission to make it executable

Creating a backup file in a different folder

Moving files from one folder to another

From the images above, you have observed some of the most common file navigation commands used to create directories, move between directories, explore directories, view changes, and change permissions, etc. Some of the basic commands include:-

ssh - used to connect to a cloud server
cd - change directory, move to a directory within the current working directory
pwd - print current working directory
mkdir - make a directory/folder
su - switch user
whoami - gives you the current user
ls - list the files and folders in the current directory
ls -l - list the files and folders in the current directory and include the hidden files
cat - output the contents of the file in text
vi - use vim to edit a file
nano - use nano to edit a file Usually written as nano file.py to mean open file.py for editing
sudo useradd -m user_name- used to add users
sudo passwd - used to change the administrator's password
passwd - when entered, it prompts you to give the old password, then asks you to type the new password
touch - create a file
man - This is a user manual to search for any command
wget - used to download files online
head - prints the first 10 rows of a document without opening the whole doc. Useful for large files
mv - used to move files from one folder to another
cp - used to copy files from one folder to another. It is also used to rename files
grep text - grep is used to search for text in a file. grep text will return instances of the word "text" in a document

While there are more commands used in Linux, the read provides a beginner-friendly guide into understanding what Linux is all about and basic navigation.

If you enjoyed the read, please like, share, and subscribe for more articles as we document my journey from a beginner to an expert in data engineering.

DEV Community: Grace N Muchiri