Linux Fundamentals for Data Engineering

Introduction

From a hobby to one of the best operating systems, Linux is an important tool in the world of Data Engineering.Linux is a free, open-source operating system kernel first developed by Linus Torvalds in 1991. Built on the principles of Unix, it has since grown into the backbone of modern computing — powering everything from smartphones and personal laptops to enterprise servers, supercomputers, and cloud infrastructure.

LinuX Basic Commands

• ls — The most frequently used command in Linux to list directories
• pwd — Print working directory command in Linux
• cd — Linux command to navigate through directories
• mkdir — Command used to create directories in Linux
• mv — Move or rename files in Linux
• cp — Similar usage as mv but for copying files in Linux
• rm — Delete files or directories
• touch — Create blank/empty files
• ln — Create symbolic links (shortcuts) to other files
• cat — Display file contents on the terminal
• clear — Clear the terminal display
• echo — Print any text that follows the command
• less — Linux command to display paged outputs in the terminal
• man — Access manual pages for all Linux commands
• uname — Linux command to get basic information about the OS
• whoami — Get the active username
• tar — Command to extract and compress files in Linux
• grep — Search for a string within an output
• head— Return the specified number of lines from the top
• tail — Return the specified number of lines from the bottom
• diff — Find the difference between two files
• cmp — Allows you to check if two files are identical
• comm — Combines the functionality of diff and cmp
• sort — Linux command to sort the content of a file while outputting
• export — Export environment variables in Linux
• zip — Zip files in Linux
• unzip — Unzip files in Linux
• ssh — Secure Shell command in Linux
• service — Linux command to start and stop services
• ps — Display active processes
• kill and killall — Kill active processes by process ID or name
• df— Display disk filesystem information
• mount — Mount file systems in Linux
• chmod — Command to change file permissions
• chown — Command for granting ownership of files or folders
• ifconfig — Display network interfaces and IP addresses
• traceroute — Trace all the network hops to reach the destination
• wget — Direct download files from the internet
• ufw— Firewall command
• iptables — Base firewall for all other firewall utilities to interface with
• apt, pacman, yum, rpm — Package managers depending on the distro
• sudo — Command to escalate privileges in Linux
• cal — View a command-line calendar
• alias — Create custom shortcuts for your regularly used commands
• dd — Majorly used for creating bootable USB sticks
• whereis — Locate the binary, source, and manual pages for a command
• whatis — Find what a command is used for
• top — View active processes live with their system usage
• useradd and usermod — Add new user or change existing users data
• passwd — Create or update passwords for existing users

Practical Examples of Linux Commands

The ls command

The most frequently used command in Linux to list directories.
ls command options;

1. -l - Long listing format
2. -a - Include hidden files
3. -h - Human-readable sizes
4. -t - Sort by modification time
5. -r - Reverse order while sorting
6. -R - List subdirectories recursively
7. -S - Sort by file size
8. -1 - List one file per line
9. -d - List directories themselves, not their contents
10. -F - Append indicator (one of */=@|) to entries

Editors in Linux

We use text editors in Linux because "everything is a file" in the Linux philosophy, meaning system settings, hardware configurations, scripts, and logs are all stored as plain text files. Unlike word processors (like Microsoft Word) that embed hidden formatting code, text editors handle pure, raw text, which is required to safely modify system operations without corrupting data.
They include;

1. Vim(vi): A highly powerful, modal editor available by default on almost every Linux distribution. It utilizes specific modes for typing versus navigating.
2. Nano :A simple, beginner-friendly editor that displays helpful keyboard shortcuts at the bottom of the screen.

File Permissions in Linux

Reading the Permission String
This shows the permissions allowed to each user in a file.
Example: (-rw-r--r--)

• First character(-): Shows that this is a file, if first character is (d), then it is a directory.
• Characters 2-4(rw-): The owner has read and write access, but cannot execute it.
• Characters 5-7(r--): Members of the group have read-only access it.
• Characters 8-10(r--): All other users on the system have read-only access

Numeric Permission Values

• 0(---) :No permission

• 1(--x) :Execute only

• 2(-w-) :Write only

• 3(-wx) :Write + Execute

• 4(r--) :Read only

• 5(r-x) :Read + Execute

• 6(rw-) :Read + Write

• 7(rwx) :Read + Write + Execute

Chmod and Chown Commands

The chmod command modifies read, write, and execute permissions for files or directories, while the chown command modifies the user and/or group ownership of those files. Together, they form the core of the Linux file security and access control model.

Chmod examples

chmod 777 file :Everyone can read, write, execute
chmod 755 file :Owner full permission; others read + execute
chmod 644 file :Owner read/write; others read only
chmod 600 file :Private file (owner only)

Chown examples

chmod u+x file :Add execute for user
chmod g-w file :Remove write from group
chmod o+r file :Add read for others
chmod a+x file :Allow execute for everyone

Ownership commands

ls -l :View ownership and permissions
sudo chown user file :Change owner
sudo chown user:group file :Change owner and group
sudo adduser username :Create new user
sudo usermod -aG devops username :Add user to group

Conclusion

Learning and understanding the Linux fundamentals is necessary for all data engineer. It was the perfect start for my data engineering journey.