Linux Fundamentals for Data Engineering

*Introduction: *

Two weeks into Data Engineering and I dived into the basics of Linux as an operating system and also the fundamentals of it in data engineering. While researching on how linux is used in data engineering, it got more interesting as it's also used in crucial areas such as pipelines, handling and manipulating files, not forgetting how it's used widely in the cloud.

Table of contents

1. Commands used in navigationFile Operations
2. Directory Operations
3. File Viewing and Editing
4. File Permissions
5. Search and Pattern Matching
6. System Information
7. Data Processing Commands

**Commands used in Navigation

**
To change from one file directory to the other we use the command

cd

followed by your directory

Go back to home directory

cd ~

Go up one level

cd ..

_Tip: If you ever get lost on which file you're working on pwd shows your current working directory which saves you a lot of time.

Listing directories

List files to see the files in your folder
ls

Listing with permissions and timestamps
ls -l

Sort by time (newest first)
ls -lt

Sort by time (oldest first)
ls -ltr

Show hidden files
ls -a

## File Operations

Creating and Copying Files

Create empty file

touch linux.csv

 
Copy file

cp linux.csv Desktop

Copy with preservation of metadata.

cp -p linux.csv Desktop

Copying directory recursively

cp -R /Desktop /Linux_project

Files Removal

Remove only the file

rm linux.csv

Removing a file directory and its contents

rm -r Linux_project

Force removing a file

rm -f linux.csv

Viewing and Editing

Viewing File Contents

View entire file

cat linux.csv

View first 10 lines

head linux.csv

View first n lines

head -n 20 linux.csv

View last 10 lines

tail linux.csv

View last n lines

tail -n 20 linux.csv

Follow log file in real-time (Important for monitoring data pipelines)

tail -f pipeline.log

*File Permissions *

Understanding Permissions

Permission structure:

r (read) = 4
w (write) = 2
x (execute) = 1

View file permissions

ls -l

Output: -rw-r--r-- 1 data datagrp 1024 June 6 10:00 linux.csv

Change permissions

chmod 644 linux.csv

# Owner: rw-, Group: r--, Others: r--

chmod 755 script.sh

` # Owner: rwx, Group: r-x, Others: r-x File Editing

Basic vim commands:

i - enter insert mode
esc- exit insert mode
:w - save
:q - quit
:wq - save and quit
:q! - quit without saving

Opening a file in vi editor

vi linux.csv

*## Search and Pattern Matching *

Finding Files

Finding files by name
find /data -name "*.csv"
Find and execute command
find /data -name "*.tmp" -exec rm {} \;
Find files modified in last 24 hours
find /data -mtime -1
Using grep
Recursive search in directory
grep -r "FAILED" /logs/
Search for pattern in file
grep "ERROR" pipeline.log

Count occurrences
grep -c "SUCCESS" pipeline.log
Case insensitive search
grep -i "error" pipeline.log

Data Processing Commands

Sorting data
sort linux.csv > sorted_linux.csv
Count lines in file
wc -l linux.csv
Removing duplicates
sort linux.csv | uniq > unique_linux.csv

Data Transformation

Extracting specific columns (using cut)
cut -d',' -f1,2 linux.csv > subset.csv
Replacing text
sed 's/old/new/g' linux.csv > modified.csv
Filter rows
awk -F',' '$3 > 2000' linux.csv > filtered.csv

Pipelines Operations

Checking file counts
ls -1 /data/input | wc -l
Monitoring disk usage
du -h /data/warehouse
Verifying file integrity
md5sum linux.csv > checksum.txt
Archiving and File compression
Create tar archive
tar -czf archive.tar.gz /data/files/
Extract tar archive

tar -xzf archive.tar.gz
Compress files

gzip large_file.csv