DEV Community

Jane Ndung'u
Jane Ndung'u

Posted on Edited on

Linux Fundamentals for Data Engineering

Linux image

Introduction

Linux is an open-source operating system. It manages the hardware and software on a computer. Linux also acts as a bridge between a computer's physical hardware and the software applications running on it. For example the cpu and memory.

I worked on a Data Engineering assignment that tested my skills in Linux, PostgreSQL and data management.

In this article I will walk you through the tasks I completed and provide examples of the commands and techniques I used.

Why Linux is important in Data Engineering

Data Engineering mainly focuses on the process of collecting, moving, cleaning and storing of data for business use so that it can be analyzed effectively.

One of the main reasons as to why Linux is said to be the backbone of data engineering is because many big data technologies such as Kafka and Airflow are commonly deployed on Linux servers, and therefore this makes it easier for data engineers to automate repetitive tasks using shell scripting and manage database efficiently from the terminal.

Linux is also commonly used for Cloud Computing.

Cloud computing basically means using computing resources over the internet instead of buying and managing your own physical servers.

Most cloud platforms such as Microsoft Azure, AWS, and Google cloud use Linux based servers because they are secure,stable and cost-effective.

Setting up the Environment

The first step was to log in to a server using SSH. I used the following command:


ssh @root<ip_address> -p 22

Enter fullscreen mode Exit fullscreen mode

Server login

Once logged in I created a Linux user with my first name and last name initials. For example my user name is JaneN. I created a directory with the name:


adduser janen

Enter fullscreen mode Exit fullscreen mode

Practicing Linux Commands

As part of the assignment I practiced Linux commands. Here are some examples:

  • Listing files and folders: ls lists all files and folders in the current directory.

  • Listing hidden files: ls -a includes all dotted files.

  • formatted files: ls -l shows the long formatted files.

  • Human-readable format: ls -lh lists all files and folders in a human-readable format.

  • Removing files and directories: rm -i confirms the deletion of a file before deleting rm -r removes directories and everything in them recursively and rm removes a file.

  • Moving files: cp creates a duplicate copy of a file while leaving the original copy unchanged and mv moves the file without leaving any copy behind.

  • Securely transferring files: scp is used to transfer files and directories between local and remote servers.

secure transfer of files

Some other useful commands I practiced include:

  • mkdir. Making a directory

  • touch. Used when creating files

  • cd ... Used to move one step behind in a directory

  • clear. Used to clear the terminal

  • head. Shows the first results in a certain file

  • tail. Shows the last results in a certain file

  • man ls. Displays the manual documentation for the ls commands

  • scp -r. Used to securely transfer directories between local and remote servers but in a recursive manner

  • whoami. Shows the current logged-in user

  • find -iname. Searches for files and directories case-insensitively.

  • less. Used to view contents in a file one screen at a time unlike the cat command

  • sort. Sorts data alphabetically

  • cat. Displays the contents of a file directly in the terminal

  • pwd. Prints the current working directory.

  • cd. Changing directory

Below attached is a sample of how I used the above mentioned commands;

Linux commands

Checking PostgreSQL Installation

Next I checked if PostgreSQL was installed on my machine by running the following command:


psql --version

Enter fullscreen mode Exit fullscreen mode

This command displays the version of PostgreSQL installed on your system.

Testing External Connections

I then tested if the server allows connections on port 5432 and if users can successfully connect remotely to PostgreSQL. I used Dbeaver to test the connection.

postgres connection to Dbeaver

Database Setup

For my database setup I created a database using my Linux user credentials:


CREATE DATABASE JaneN;

Enter fullscreen mode Exit fullscreen mode

database creation

Then I created a schema called staging:


create schema staging

Enter fullscreen mode Exit fullscreen mode

Schema creation

Importing Sample Data

I generated sample data from Mockaroo and called it cars.csv. I then imported my csv data to my postgresql database schema using DBeaver.

Importing the csv file

Results of my csv file after importing

Challenges faced and Lessons learnt

During this assignment below are some of the challenges and lessons learnt;

  • I came across an SSH connection issue because I didn't have the correct password to the server.

  • Remembering some of the commands taught earlier in class.I had to revisit the recorded sessions so that I can be able to complete some of the tasks.

  • I learnt about file management skills such as creating, moving, deleting and searching of files and directories using Linux commands.

  • I learnt that Linux commands improves productivity and the importance of having terminal skills because they helped me complete the tasks faster.

Conclusion

While working on this task, I gained hands-on experience in server management using SSH, database creation and configuration, and data import processes using a tool called DBeaver.

Additionally, practicing Linux commands helped me understand how powerful the terminal is for managing and creating of files, navigating directories, and performing systems operations efficiently. I also learned that small command-line mistakes such as incorrect syntax or a forgotten command, can slow down progress but consistent practice helps build confidence and accuracy over time.

Top comments (2)

Collapse
 
leslie_angu_ profile image
leslie angu •

Goodstuff. The only improvement I would added is the structure of the storyline, we began with linux, then we went to linux commands, then we spoke about why DE use linux and lastly conclusion. Let's attempt this, the introduction, the reasons why we use linux as DE's. We can have the definitions of the commands used like ssh - secure shell protocol for connecting to servers, explained in the article as you use it instead of at the bottom. Good work highlighting what you learnt at the conclusion. Anyway, I hope to read more articles from you.

Collapse
 
jane_ndungu profile image
Jane Ndung'u •

Noted, thanks