Scribd
Upload
42 views

Practical 2 Hadoop Distributed File System (HDFS)

Uploaded by

black hello
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
42 views

Practical 2 Hadoop Distributed File System (HDFS)

Uploaded by

black hello
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as DOCX, PDF, TXT or read online on Scribd
You are on page 1/ 4

1 of 4

BMCS2013 DATA ENGINEERING

PRACTICAL 2 Hadoop Distributed File System (HDFS)

1. Launch the Ubuntu-22.04-de distro:


In PowerShell (run as administrator), launch the distro for this course:
PS C:\Users\TARUMT> wsl ~
hduser@PC25:~$

~~~ As the user hduser ~~~


2. Start HDFS and YARN
2.1. Start the HDFS service
hduser@PC25:~$ start-dfs.sh
Starting namenodes on [localhost]
Starting datanodes
Starting secondary namenodes [PC25]

To check the services currently running, use the jps command:


hduser@PC25:~$ jps
1392 Jps
1114 SecondaryNameNode
876 DataNode
685 NameNode

2.2. Start the YARN service


hduser@PC25:~$ start-yarn.sh
Starting resourcemanager
Starting nodemanagers

hduser@PC25:~$ jps
2112 Jps
1696 NodeManager
1542 ResourceManager
1114 SecondaryNameNode
876 DataNode
685 NameNode

FYI only, the following actions have already been completed in the distro:
# Create the directories named user and tmp in the distributed file system:
# The /user directory is where all Hadoop users’ home directories will be created later on.
hduser@PC25:~$ hdfs dfs -mkdir /user
hduser@PC25:~$ hdfs dfs -mkdir /tmp

# Give full permissions for all users to the tmp directory:


hduser@PC25:~$ hdfs dfs -chmod -R 777 /tmp
2 of 4

3. Create User Directories in HDFS


3.1. Create a HDFSuser directory for student:
hduser@PC25:~$ hdfs dfs -mkdir /user/student

3.2. Change ownership for the newly created directory:


hduser@PC25:~$ hdfs dfs -chown student:hduser /user/student

Note (FYI only):

HDFS file permissions are similar to Linux file permissions.


E.g., to change the permission of the file shakespeare.txt to 664:
$ hdfs dfs -chmod 664 shakespeare.txt
where 664 is an octal representation of the flags to set for the permission triple.
The above statement changes the permissions to -rw-rw-r--:
● 6 is 110, which means read and write, but not execute.
● 7 is 111, which means complete permissions.
● 4 is 100, which means read-only.

~~~ As the user student ~~~

4. Switch user to student


hduser@PC25:~$ su - student
student@PC25:~$

5. HDFS Basic File System Operations


5.1. See the available commands in the dfs shell
student@PC25:~$ hdfs dfs -help

5.2. Download the shakespeare.txt file from Google Drive into your local file
system (Ubuntu 22.04)
student@PC25:~$ wget --no-check-certificate
'https://docs.google.com/uc?
export=download&id=122PnuKaSaA_OyYOKnxQOdlMc5awdyf5v' -O
shakespeare.txt

💡 Remember to confirm that the above action is successful.

5.3. Copy the downloaded file shakespeare.txt from the local file system to HDFS
student@PC25:~$ hdfs dfs -put shakespeare.txt shakespeare.txt

💡 Remember to confirm that the above action is successful.


3 of 4
4 of 4

5.4. Read the contents of the file in HDFS using the cat command, and then pipe the
output to less in order to view the contents of the remote file.
student@PC25:~$ hdfs dfs -cat shakespeare.txt | less

Note: use the arrow keys to navigate the file. Type q to quit.

5.5. Copy the file from HDFS to the local file system and rename it as shakespeare-
dfs.txt.
student@PC25:~$ hdfs dfs -get shakespeare.txt ./shakespeare-
dfs.txt
💡 Remember to confirm that the above action is successful.

6. To end your practical sessions


6.1. Logout from the student account
student@PC25:~$ exit
hduser@PC25:~$ su - student

~~~ As the user hduser ~~~


6.2. Terminate the YARN service
hduser@PC25:~$ stop-yarn.sh

6.3. Terminate the HDFS service


hduser@PC25:~$ stop-dfs.sh

6.4. Logout from the hduser account


hduser@PC25:~$ exit
PS C:\Users\TARUMT>

6.5. Terminate the WSL instance


PS C:\Users\TARUMT> exit

Other HDFS Commands


Recall that the HDFS shell commands are similar to POSIX-like commands and invoked using:
$ hdfs dfs <args> <command>

Other HDFS commands include:


cat chown ls rm
chgrp cp mkdir stat
chmod du mv tail

You might also like

pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy