DEV Community

Cover image for Mastering Hadoop FS Shell: copyFromLocal and get Commands
LabEx profile image Labby
Labby for LabEx

Posted on

Mastering Hadoop FS Shell: copyFromLocal and get Commands

#labex #hadoop #coding #programming

Introduction

MindMap

In a galaxy far far away, there exists a ongoing space war between different factions. The galaxy is also bustling with space traders trying to make profits amidst the chaos. Our story focuses on one such space trader who needs to transfer valuable data files using Hadoop's HDFS file system. The trader's goal is to successfully copy files from their local system to Hadoop using the copyFromLocal command and retrieve files from Hadoop to their local system using the get command.

Copy File to Hadoop

In this step, the space trader needs to copy a file named data.txt from their local system to Hadoop HDFS.

Open the terminal and follow the steps below to get started.

  1. Switch to the hadoop user for proper permissions: 
   su - hadoop
Enter fullscreen mode Exit fullscreen mode
  1. Create a directory named space_data in Hadoop: 
   hdfs dfs -mkdir /space_data
Enter fullscreen mode Exit fullscreen mode
  1. Copy the local file data.txt to Hadoop: 
   hdfs dfs -copyFromLocal /home/hadoop/data.txt /space_data/
Enter fullscreen mode Exit fullscreen mode
  1. Verify that the file was successfully copied: 
   hdfs dfs -ls /space_data
Enter fullscreen mode Exit fullscreen mode

Retrieve File from Hadoop

Now, the space trader needs to retrieve the file data.txt from Hadoop and save it to their local system.

  1. Change to the directory where you want to save the retrieved file: 
   mkdir /home/hadoop/space_data && cd /home/hadoop/space_data
Enter fullscreen mode Exit fullscreen mode
  1. Retrieve the file data.txt from Hadoop to the local system: 
   hdfs dfs -get /space_data/data.txt
Enter fullscreen mode Exit fullscreen mode
  1. Verify that the file was successfully retrieved: 
   ls -l data.txt
Enter fullscreen mode Exit fullscreen mode

Summary

In this lab, we simulated a space trading scenario where a trader needed to transfer data files between their local system and Hadoop HDFS. By practicing the copyFromLocal and get commands in Hadoop FS Shell, users can learn how to efficiently move files to and from Hadoop's distributed file system. This hands-on experience helps in understanding the basic file management operations in Hadoop and enhances one's skills in working with big data ecosystems.

🚀 Practice Now: Hadoop FS Shell copyFromLocal/get

Want to Learn More?

Top comments (0)

Read next

mikeyoung44 profile image

AI Breakthroughs: Language Models Can Now Control Computer Interfaces Like Humans

Mike Young -

dpc profile image

Daily JavaScript Challenge #JS-35: Calculate the Product of Digits

DPC -

sunilkumrdash profile image

We made an AI SWE that solved 48.60% of issues on the SWE bench, 100% open-source.

Sunil Kumar Dash -

mdarifulhaque profile image

2097. Valid Arrangement of Pairs

MD ARIFUL HAQUE -