bigdata-hadoop

hadoop

concept

4v:

history of Hadoop

function & goal

inclusion

• hdfs
• yarn
• map-reduce

installation (require jdk)

three supported modes

• Local (Standalone) Mode
• Pseudo-Distributed Mode
• Fully-Distributed Mode

step by step

assure /opt/software and /opt/module exists and current user has privillege

• cd /opt/software/
• tar -zxvf hadoop-2.7.2.tar.gz -C /opt/module/
• ls /opt/module/
• sudo vi /etc/profile

  #HADOOP_HOME
  export HADOOP_HOME=/opt/module/hadoop-2.7.2
  export PATH=$PATH:$HADOOP_HOME/bin
  export PATH=$PATH:$HADOOP_HOME/sbin

• source /etc/profile
• hadoop version

Pseudo-Distributed Mode

Fully-Distributed Mode

所以：在h152上执行start-dfs.sh，在h153上执行start-yarn.sh。

tests with sample

• grep sample

$ mkdir input
$ cp etc/hadoop/*.xml input
$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-3.2.1.jar grep input output 'dfs[a-z.]+'
$ cat output/*

执行过后，会生成output目录，且里面含有文件，如下图

• word count
  和grep类似

$ bin/hadoop jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.7.2.jar wordcount wcinput/ wcoutput

常用操作

• hdfs dfs -put /opt/student.json /user/spark/student.json

ui

http://localhost:50070 查看hdfs
http://localhost:8088 查看mapreduce
http://localhost:19888 查看任务历史