We will describe Hadoop setup on single node and multi node. The Hadoop environment setup and configuration will be described in details. First you need to download the following software (rpm).
- Java JDK RPM
- Apache Hadoop 0.20.204.0 RPM
A) Single node system Hadoop setup
1) Install JDK on a Red Hat or CentOS 5+ system.
$ ./jdk-6u26-linux-x64-rpm.bin.sh
Java is installed and set JAVA_HOME to /usr/java/default
2) Install Apache Hadoop 0.20.204.
$ rpm -i hadoop-0.20.204.0-1.i386.rpm
3) Setup Apache Hadoop configuration and start Hadoop processes.
$ /usr/sbin/hadoop-setup-single-node.sh
The setup wizard will guide you through a list of questions to setup Hadoop. Hadoop should be running after answering ‘Y’ to all questions.
Create a user account on HDFS for yourself.
$ /usr/sbin/hadoop-create-user.sh -u $USER
B) Multi-nodes Hadoop setup
1) Install both the JDK and Hadoop 0.20.204.0 RPMs on all nodes
2) Generate hadoop configuration on all nodes:
$ /usr/sbin/hadoop-setup-conf.sh \
--namenode-url=hdfs://${namenode}:9000/ \
--jobtracker-url=${jobtracker}:9001 \
--conf-dir=/etc/hadoop \
--hdfs-dir=/var/lib/hadoop/hdfs \
--namenode-dir=/var/lib/hadoop/hdfs/namenode \
--mapred-dir=/var/lib/hadoop/mapred \
--datanode-dir=/var/lib/hadoop/hdfs/data \
--log-dir=/var/log/hadoop \
--autoWhere ${namenode} and ${jobtracker} should be replaced with hostname of namenode and jobtracker.
3) Format namenode and setup default HDFS layout.
$ /usr/sbin/hadoop-setup-hdfs.sh
4) Start all data nodes.
$ /etc/init.d/hadoop-datanode start
5) Start job tracker node.
$ /etc/init.d/hadoop-jobtracker start
6) Start task tracker nodes.
$ /etc/init.d/hadoop-tasktracker start
7) Create a user account on HDFS for yourself.
$ /usr/sbin/hadoop-create-user.sh -u $USER
C) Setup Environment for Hadoop
$ vi ~/.bash_profile
In INSERT mode set path for JAVA_HOME
Export JAVA_HOME
Save file by clicking esc:wq
Run the .bash_profile
$ source ~/.bash_profile
Set JAVA_HOME path in Hadoop Environment file
$ /etc/hadoop/hadoop-env.sh
D)Configuration for Hadoop
Use the following:
conf/core-site.xml:
<configuration>
<property>
<name>fs.default.name</name>
<value>hdfs://localhost:9000</value>
</property>
</configuration>conf/hdfs-site.xml:
<configuration>
<property>
<name>dfs.replication</name>
<value>1</value>
</property>
</configuration>conf/mapred-site.xml:
<configuration>
<property>
<name>mapred.job.tracker</name>
<value>localhost:9001</value>
</property>
</configuration>E) Hadoop Commands
$ hadoop $ hadoop namenode –format (Format the namenode, If ask to answer press ‘Y’) $ hadoop namenode (Start the namenode) $ find / -name start-dfs.sh (find the file in directory) $ cd usr/sbin (Go to respective directory directly) $ start-dfs.sh $ start-mapred.sh $ hadoop fs –ls / (Shows the HDFS root folder) $ hadooop fs –put input/file01 /input/file01 (Copy local input/file01 to HDFS root /input/file01)
