HADOOP 2.7.
2 INSTALLING ON
UBUNTU 20
(SINGLE-NODE CLUSTER)
1- Install Java 1.8.
aramadan@ubuntu: ~$ cd ~
# Update the source list
aramadan@ubuntu: ~$ sudo apt-get update
aramadan@ubuntu: ~$ sudo apt-get upgrade
aramadan@ubuntu: ~$ sudo apt-get install openjdk-8-jdk
# Verify Java Installation
aramadan@ubuntu: ~$ java -version
openjdk version "1.8.0_292"
OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~20.04-b10)
OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)
2- Adding a dedicated Hadoop user.
aramadan@ubuntu: ~$ sudo addgroup hadoop
Adding group `hadoop' (GID 1003) ...
Done.
aramadan@ubuntu: ~$ sudo adduser --ingroup hadoop hduser
Adding user `hduser' ...
Adding new user `hduser' (1002) with group `hadoop' ...
Creating home directory `/home/hduser' ...
Copying files from `/etc/skel' ...
New password: hduser
Retype new password: hduser
passwd: password updated successfully
Changing the user information for hduser
Enter the new value, or press ENTER for the default
Full Name []:
Room Number []:
Work Phone []:
Home Phone []:
Other []:
3- Installing SSH.
ssh has two main components:
ssh : The command we use to connect to remote machines - the client.
sshd : The daemon that is running on the server and allows clients to
connect to the server.
aramadan@ubuntu: ~$ sudo apt-get install ssh
#Verify ssh installation
4- Create and Setup SSH Certificates
Hadoop requires SSH access to manage its nodes, i.e. remote machines
plus our local machine. For our single-node setup of Hadoop, we therefore
need to configure SSH access to localhost.
su hduser
ssh-keygen -t rsa -P ""
cat $HOME/.ssh/id_rsa.pub >> $HOME/.ssh/authorized_keys
5- Install Hadoop
hduser@ubuntu: ~$ wget "https://www.apache.org/dyn/mirrors/mirrors.cgi?
action=download&filename=hadoop/common/hadoop-2.7.2/hadoop-2.7.2.tar.gz"
#Take some time to finish the installation.
hduser@ubuntu: ~$ tar xvzf hadoop-2.7.2.tar.gz
hduser@ubuntu: ~$ cd Hadoop-2.7.2
hduser@ubuntu: ~$ sudo mv * /usr/local/hadoop
[sudo] password for hduser:
hduser is not in the sudoers file. This incident will be reported.
hduser@ubuntu: ~$ su aramadan #Type your primary username.
hduser@ubuntu: ~$ sudo adduser hduser sudo #add hduser in sudo group.
hduser@ubuntu: ~$ sudo su hduser
hduser@ubuntu:~/hadoop-2.7.2$ sudo mv * /usr/local/hadoop
mv: target '/usr/local/hadoop' is not a directory
hduser@ubuntu:~/hadoop-2.7.2$ sudo mkdir /usr/local/hadoop #Create Hadoop directory.
hduser@ubuntu:~/hadoop-2.7.2$ sudo mv * /usr/local/hadoop
#Verify files moving .
hduser@ubuntu:~/hadoop-2.7.2$ ls /usr/local/hadoop/
hduser@ubuntu:~/hadoop-2.7.2$ sudo chown -R hduser:hadoop /usr/local/hadoop
6- Setup configuration Files:
1. ~/.bashrc
2. /usr/local/hadoop/etc/hadoop/hadoop-env.sh
3. /usr/local/hadoop/etc/hadoop/core-site.xml
4. /usr/local/hadoop/etc/hadoop/mapred-site.xml.template
5. /usr/local/hadoop/etc/hadoop/hdfs-site.xml
1.~/.bashrc
hduser@ubuntu:~ $ update-alternatives --config java
There is only one alternative in link group java (providing /usr/bin/java): /usr/lib/jvm/java-8-
openjdk-amd64/jre/bin/java
Nothing to configure.
hduser@ubuntu:~ $ sudo gedit ~/.bashrc
# Add to the end of the file (Java & Hadoop Variables Environment)
#HADOOP VARIABLES START
export JAVA_HOME=usr/lib/jvm/java-8-openjdk-amd64
export HADOOP_INSTALL=/usr/local/hadoop
export PATH=$PATH:$HADOOP_INSTALL/bin
export PATH=$PATH:$HADOOP_INSTALL/sbin
export HADOOP_MAPRED_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_HOME=$HADOOP_INSTALL
export HADOOP_HDFS_HOME=$HADOOP_INSTALL
export YARN_HOME=$HADOOP_INSTALL
export HADOOP_COMMON_LIB_NATIVE_DIR=$HADOOP_INSTALL/lib/native
export HADOOP_OPTS="-Djava.library.path=$HADOOP_INSTALL/lib"
#HADOOP VARIABLES END
hduser@ubuntu:~ $ source ~/.bashrc
hduser@ubuntu:~ $ javac -version
hduser@ubuntu:~ $ which javac
hduser@ubuntu:~ $ readlink -f /usr/lib/jvm/java-8-openjdk-amd64/bin/javac
2./usr/local/hadoop/etc/hadoop/hadoop-env.sh
hduser@ubuntu:~ $ sudo gedit /usr/local/hadoop/etc/hadoop/hadoop-env.sh
#add the following line: export JAVA_HOME=/usr/lib/jvm/java-8-openjdk-amd64
3./usr/local/hadoop/etc/hadoop/core-site.xml
hduser@ubuntu:~ $ sudo mkdir -p /app/hadoop/tmp
hduser@ubuntu:~ $ sudo chown hduser:hadoop /app/hadoop/tmp
hduser@ubuntu:~ $ sudo gedit /usr/local/hadoop/etc/hadoop/core-site.xml
#Replace Configuration tag with the following block
<configuration>
<property>
<name>hadoop.tmp.dir</name>
<value>/app/hadoop/tmp</value>
<description>A base for other temporary directories.</description>
</property>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:54310</value>
<description>The name of the default file system. A URI whose
scheme and authority determine the FileSystem implementation. The
uri's scheme determines the config property (fs.SCHEME.impl) naming
the FileSystem implementation class. The uri's authority is used to
determine the host, port, etc. for a filesystem.</description>
</property>
</configuration>
4./usr/local/hadoop/etc/hadoop/mapred-site.xml
hduser@ubuntu:~ $ cp /usr/local/hadoop/etc/hadoop/mapred-site.xml.template
/usr/local/hadoop/etc/hadoop/mapred-site.xml
hduser@ubuntu:~ $ sudo gedit /usr/local/hadoop/etc/hadoop/mapred-site.xml
#Replace Configuration tag with the following block
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:54311</value>
<description>The host and port that the MapReduce job tracker runs
at. If "local", then jobs are run in-process as a single map
and reduce task.
</description>
</property>
</configuration>
5./usr/local/hadoop/etc/hadoop/hdfs-site.xml
hduser@ubuntu:~ $ sudo mkdir -p /usr/local/hadoop_store/hdfs/namenode
hduser@ubuntu:~ $ sudo mkdir -p /usr/local/hadoop_store/hdfs/datanode
hduser@ubuntu:~ $ sudo chown -R hduser:hadoop /usr/local/hadoop_store
hduser@ubuntu:~ $ sudo gedit /usr/local/hadoop/etc/hadoop/hdfs-site.xml
#Replace Configuration tag with the following block
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
<description>Default block replication.
The actual number of replications can be specified when the file is created.
The default is used if replication is not specified in create time.
</description>
</property>
<property>
<name>dfs.namenode.name.dir</name>
<value>file:/usr/local/hadoop_store/hdfs/namenode</value>
</property>
<property>
<name>dfs.datanode.data.dir</name>
<value>file:/usr/local/hadoop_store/hdfs/datanode</value>
</property>
</configuration>
Format the New Hadoop Filesystem & Run Hadoop
hduser@ubuntu:~ $ hadoop namenode -format
hduser@ubuntu:~ $ start-all.sh #Run Hadoop Services
hduser@ubuntu:~ $ jps
15888 Jps
15682 NodeManager
15218 DataNode
15415 SecondaryNameNode
15050 NameNode
15550 ResourceManager
