Introduction
Linux is an open-source operating system. It manages the hardware and software on a computer. Linux also acts as a bridge between a computer's physical hardware and the software applications running on it. For example the cpu and memory.
I worked on a Data Engineering assignment that tested my skills in Linux, PostgreSQL and data management.
In this article I will walk you through the tasks I completed and provide examples of the commands and techniques I used.
Why Linux is important in Data Engineering
Data Engineering mainly focuses on the process of collecting, moving, cleaning and storing of data for business use so that it can be analyzed effectively.
One of the main reasons as to why Linux is said to be the backbone of data engineering is because many big data technologies such as Kafka and Airflow are commonly deployed on Linux servers, and therefore this makes it easier for data engineers to automate repetitive tasks using shell scripting and manage database efficiently from the terminal.
Linux is also commonly used for Cloud Computing.
Cloud computing basically means using computing resources over the internet instead of buying and managing your own physical servers.
Most cloud platforms such as Microsoft Azure, AWS, and Google cloud use Linux based servers because they are secure,stable and cost-effective.
Setting up the Environment
The first step was to log in to a server using SSH. I used the following command:
ssh @root<ip_address> -p 22
Once logged in I created a Linux user with my first name and last name initials. For example my user name is JaneN. I created a directory with the name:
adduser janen
Practicing Linux Commands
As part of the assignment I practiced Linux commands. Here are some examples:
Listing files and folders:
lslists all files and folders in the current directory.Listing hidden files:
ls -aincludes all dotted files.formatted files:
ls -lshows the long formatted files.Human-readable format:
ls -lhlists all files and folders in a human-readable format.Removing files and directories:
rm -iconfirms the deletion of a file before deletingrm -rremoves directories and everything in them recursively andrmremoves a file.Moving files:
cpcreates a duplicate copy of a file while leaving the original copy unchanged andmvmoves the file without leaving any copy behind.Securely transferring files:
scpis used to transfer files and directories between local and remote servers.
Some other useful commands I practiced include:
mkdir. Making a directorytouch. Used when creating filescd ... Used to move one step behind in a directoryclear. Used to clear the terminalhead. Shows the first results in a certain filetail. Shows the last results in a certain fileman ls. Displays the manual documentation for the ls commandsscp -r. Used to securely transfer directories between local and remote servers but in a recursive mannerwhoami. Shows the current logged-in userfind -iname. Searches for files and directories case-insensitively.less. Used to view contents in a file one screen at a time unlike the cat commandsort. Sorts data alphabeticallycat. Displays the contents of a file directly in the terminalpwd. Prints the current working directory.cd. Changing directory
Below attached is a sample of how I used the above mentioned commands;
Checking PostgreSQL Installation
Next I checked if PostgreSQL was installed on my machine by running the following command:
psql --version
This command displays the version of PostgreSQL installed on your system.
Testing External Connections
I then tested if the server allows connections on port 5432 and if users can successfully connect remotely to PostgreSQL. I used Dbeaver to test the connection.
Database Setup
For my database setup I created a database using my Linux user credentials:
CREATE DATABASE JaneN;
Then I created a schema called staging:
create schema staging
Importing Sample Data
I generated sample data from Mockaroo and called it cars.csv. I then imported my csv data to my postgresql database schema using DBeaver.
Challenges faced and Lessons learnt
During this assignment below are some of the challenges and lessons learnt;
I came across an SSH connection issue because I didn't have the correct password to the server.
Remembering some of the commands taught earlier in class.I had to revisit the recorded sessions so that I can be able to complete some of the tasks.
I learnt about file management skills such as creating, moving, deleting and searching of files and directories using Linux commands.
I learnt that Linux commands improves productivity and the importance of having terminal skills because they helped me complete the tasks faster.
Conclusion
While working on this task, I gained hands-on experience in server management using SSH, database creation and configuration, and data import processes using a tool called DBeaver.
Additionally, practicing Linux commands helped me understand how powerful the terminal is for managing and creating of files, navigating directories, and performing systems operations efficiently. I also learned that small command-line mistakes such as incorrect syntax or a forgotten command, can slow down progress but consistent practice helps build confidence and accuracy over time.









Top comments (2)
Goodstuff. The only improvement I would added is the structure of the storyline, we began with linux, then we went to linux commands, then we spoke about why DE use linux and lastly conclusion. Let's attempt this, the introduction, the reasons why we use linux as DE's. We can have the definitions of the commands used like ssh - secure shell protocol for connecting to servers, explained in the article as you use it instead of at the bottom. Good work highlighting what you learnt at the conclusion. Anyway, I hope to read more articles from you.
Noted, thanks