Home
Free eBook
Start Here
Contact
About
Quick Apache Hadoop Admin Command Reference
Examples
by KARTHIKEYAN SADHASIVAM on FEBRUARY 18, 2015
If you are working on Hadoop, youll realize
there are several shell commands available to manage your hadoop cluster.
This article provides a quick handy reference to all Hadoop administration commands.
If you are new to big data, read the introduction to Hadoop article to understand the
basics.
1. Hadoop Namenode Commands
Command
Description
hadoop namenode -format
hadoop namenode -upgrade
start-dfs.sh
stop-dfs.sh
start-mapred.sh
stop-mapred.sh
hadoop namenode -recover
Format HDFS filesystem from Namenode
Upgrade the NameNode
Start HDFS Daemons
Stop HDFS Daemons
Start MapReduce Daemons
Stop MapReduce Daemons
Recover namenode metadata after a cluster failure
-force
(may lose data)
2. Hadoop fsck Commands
Command
Description
hadoop fsck /
hadoop fsck / -files
hadoop fsck / -files -blocks
hadoop fsck / -files -blocks
-locations
hadoop fsck / -files -blocks
-locations -racks
hadoop fsck -delete
Filesystem check on HDFS
Display files during check
Display files and blocks during check
Display files, blocks and its location
during check
Display network topology for data-node
locations
Delete corrupted files
Move corrupted files to /lost+found
directory
hadoop fsck -move
3. Hadoop Job Commands
Command
hadoop job -submit <jobfile>
hadoop job -status <jobid>
hadoop job -list all
hadoop job -list-activetrackers
Description
Submit the job
Print job status completion percentage
List all jobs
List all available TaskTrackers
hadoop job -set-priority
<job-id> <priority>
Set priority for a job. Valid priorities:
VERY_HIGH, HIGH, NORMAL, LOW,
VERY_LOW
hadoop job -kill-task
<task-id>
Kill a task
hadoop job -history
Display job history including job details, failed and
killed jobs
4. Hadoop dfsadmin Commands
Command
Description
hadoop dfsadmin -report
hadoop dfsadmin
-metasave file.txt
hadoop dfsadmin
-setQuota 10 /quotatest
hadoop dfsadmin
-clrQuota /quotatest
Report filesystem info and statistics
hadoop dfsadmin
-refreshNodes
hadoop fs -count -q
/mydir
hadoop dfsadmin
-setSpaceQuota /mydir
100M
hadoop dfsadmin
-clrSpaceQuota /mydir
hadooop dfsadmin
-saveNameSpace
Save namenodes primary data structures to file.txt
Set Hadoop directory quota to only 10 files
Clear Hadoop directory quota
Read hosts and exclude files to update datanodes that
are allowed to connect to namenode. Mostly used to
commission or decommsion nodes
Check quota space on directory /mydir
Set quota to 100M on hdfs directory named /mydir
Clear quota on a HDFS directory
Backup Metadata (fsimage & edits). Put cluster in
safe mode before this command.
5. Hadoop Safe Mode (Maintenance Mode) Commands
The following dfsadmin commands helps the cluster to enter or leave safe mode, which
is also called as maintenance mode. In this mode, Namenode does not accept any
changes to the name space, it does not replicate or delete blocks.
Command
hadoop dfsadmin -safemode
enter
hadoop dfsadmin -safemode
leave
hadoop dfsadmin -safemode get
Description
Enter safe mode
Leave safe mode
Get the status of mode
Wait until HDFS finishes data block
hadoop dfsadmin -safemode wait
replication
6. Hadoop Configuration Files
File
Description
hadoop-env.sh
core-site.xml
hdfs-site.xml
mapred-site.xml
masters
slaves
Sets ENV variables for Hadoop
Parameters for entire Hadoop cluster
Parameters for HDFS and its clients
Parameters for MapReduce and its clients
Host machines for secondary Namenode
List of slave hosts
7. Hadoop mradmin Commands
Command
Description
hadoop mradmin -safemode get
hadoop mradmin -refreshQueues
hadoop mradmin -refreshNodes
Check Job tracker status
Reload mapreduce configuration
Reload active TaskTrackers
Force Jobtracker to reload service
ACL
Force jobtracker to reload user
group mappings
hadoop mradmin -refreshServiceAcl
hadoop mradmin
-refreshUserToGroupsMappings
8. Hadoop Balancer Commands
Command
Description
start-balancer.sh
hadoop dfsadmin -setBalancerBandwidth
<bandwidthinbytes>
Balance the cluster
Adjust bandwidth used by the
balancer
Limit balancing to only 20%
resources in the cluster
hadoop balancer -threshold 20
9. Hadoop Filesystem Commands
Command
Description
hadoop fs -mkdir mydir
hadoop fs -ls
Create a directory (mydir) in HDFS
List files and directories in HDFS
hadoop fs -cat myfile
hadoop fs -du
hadoop fs -expunge
hadoop fs -chgrp hadoop file1
hadoop fs -chown huser file1
hadoop fs -rm file1
hadoop fs -touchz file2
hadoop fs -stat file1
hadoop fs -test -e file1
hadoop fs -test -z file1
hadoop fs -test -d file1
View a file content
Check disk space usage in HDFS
Empty trash on HDFS
Change group membership of a file
Change file ownership
Delete a file in HDFS
Create an empty file
Check the status of a file
Check if file exists on HDFS
Check if file is empty on HDFS
Check if file1 is a directory on HDFS
10. Additional Hadoop Filesystem Commands
Command
Description
hadoop fs -copyFromLocal <source>
<destination>
Copy from local fileystem to
HDFS
e.g: Copies file1 from local FS
to data dir in HDFS
copy from hdfs to local
filesystem
e.g: Copies file1 from HDFS
data directory to /var/tmp on
local FS
Copy from remote location to
HDFS
Copy from HDFS to remote
directory
Copy data from one cluster to
another using the cluster URL
Move data file from the local
directory to HDFS
Set the replication factor for
file1 to 3
Merge files in mydir directory
and download it as one big file
hadoop fs -copyFromLocal file1 data
hadoop fs -copyToLocal <source>
<destination>
hadoop fs -copyToLocal data/file1 /var/tmp
hadoop fs -put <source> <destination>
hadoop fs -get <source> <destination>
hadoop distcp hdfs://192.168.0.8:8020/input
hdfs://192.168.0.8:8020/output
hadoop fs -mv file:///data/datafile
/user/hduser/data
hadoop fs -setrep -w 3 file1
hadoop fs -getmerge mydir bigfile
