Introduction
Data engineering is the process of collecting, moving, cleaning, storing, and preparing data such that it is easily accessed by data analysts, data scientists, AI systems, and businesses.
Basically data engineering is building systems that make data available, usable and reliable.
When learning data engineering, there is much focus tools such python, SQL and cloud platforms. While these technologies are of importance, there is one data system called linux which is the background in this field of data engineering.
In this article, I am going to share how linux commands is playing a vital role in data engineering and how it is used in a linux environment.
Why Linux is important in Data Engineering
Linux is widely used in real-world data engineering because it provides a stable, efficient, and flexible environment for handling large amounts of data.
Most companies stores data in a database, processes large datasets, schedules automated workflows, and deploys applications to the cloud.
Linux is used by data engineers to connect to remote servers,install and configure databases,transfer files between systems,monitor server performance, manage permissions and troubleshoot production issues in a linux environmemt. Therefore, without a basic understanding of Linux, working with most of the projects become challenging.
Basic Linux commands
ssh (secure shell) ->This commands allows one to access a remote server which has a host name and authenticate with a password.
pwd -> This command is used to show current working directory
ls -> Lists all files and directories
cd -> This command is used to change a directory
mkdir -> Create a new directory
ls -a -> Displays all files including hidden files
ls -l -> Displays long-list of files
ls -lh -> Displays file size in human- readable format
mv -> Moving files and directories
cp-> Copying files and directories
scp -> secure copying of files from a local machine to a remote server.
rm -> Removing files
touch -> Create an empty file
echo -> Print text or variables
exit -> Clears the terminal
Installing postgres
sudo apt install postgresql postgresql contrib -y
Starting postgres ->sudo systemctl start postgresql
Checking the status of postgres -> sudo systemctl status postgresql
Checking the psql version -> psql --version
Connecting from Linux server to Postgress
sudo -u postgres psql
Conclusion
Linux is a fundamental skill in data engineering that helps data engineers access, manage databases, transfer data, automating tasks. Through the practice that I have been doing, I have learned how to access a remote server perform various basic linux commands. I also learnt how to install postgres in linux environment and be able create database, schema, and perform basic SQL querries in linux environment.Finally, linux skills is essential for building, managing, and maintaining reliable data engineering solutions.
Top comments (0)