Hadoop配置文件
/etc/profile
配置Java和Hadoop环境
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
export CLASSPATH=.:$JAVA_HOME/jre/lib/rt.jar:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar
export HADOOP_HOME=/usr/local/hadoop-2.6.5/hadoop-2.6.5
export PATH=$JAVA_HOME/bin:${PATH}:$HADOOP_HOME/bin:$HADOOP_HOME/sbin
配置文件在/etc/hadoop
core-site.xml
fs.defualtFS:配置主节点namenode地址
hadoop.tmp.dir:在本地文件系统所在的NameNode的存储空间和持续化处理日志
<configuration> <property> <name>fs.defaultFS</name> <value>hdfs://192.168.94.140:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/usr/local/hadoop-2.6.5/data</value> </property> </configuration>
mapred-site.xml.template
mapreduce.framework.name:执行框架设置为 Hadoop YARN
<configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> </configuration>
yarn-site.xml
yarn.resourcemanager.hostname:namenode的yarn地址
yarn.nodemanager.aux-services:Shuffle service 需要加以设置的Map Reduce的应用程序服务
<property> <name>yarn.resourcemanager.hostname</name> <value>CentOS7One</value> </property> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> </configuration>
hadoop-env.sh
JAVA_HOME换成绝对路径
export JAVA_HOME=/usr/lib/jvm/java-1.8.0-openjdk
slaves
配置子节点，伪分布式，一台机器，配置子节点为自身
CentOS7One
Hadoop免密钥配置
方法1：
1.生成私钥和公钥
[root@CentOS7One ~]# ssh-keygen -t rsa Generating public/private rsa key pair. Enter file in which to save the key (/root/.ssh/id_rsa): Created directory '/root/.ssh'. Enter passphrase (empty for no passphrase): Enter same passphrase again: Your identification has been saved in /root/.ssh/id_rsa. Your public key has been saved in /root/.ssh/id_rsa.pub. The key fingerprint is: eb:68:01:dc:9a:71:ee:f9:5d:3b:db:6f:46:90:47:f9 root@CentOS7One The key's randomart image is: +--[ RSA 2048]----+ | .| | ..| | . . o.| | + o o E| | B S o | | o o . .| | . o. . . | | +o . ..o o| | ...o . oo.+.| +-----------------+
2.将CentOS7One的id_rsa_pub拷贝到CentOSTwo中
[root@CentOS7One .ssh]# [root@CentOS7One .ssh]# scp id_rsa.pub CentOS7Two:/root The authenticity of host 'centos7two (192.168.94.139)' can't be established. ECDSA key fingerprint is dd:e2:09:9d:e2:6e:86:c3:2a:62:52:3f:f6:3a:f2:37. Are you sure you want to continue connecting (yes/no)? yes Warning: Permanently added 'centos7two,192.168.94.139' (ECDSA) to the list of known hosts. root@centos7two's password: id_rsa.pub 100% 397 0.4KB/s 00:00 [root@CentOS7One .ssh]#
3.重复步骤1，在CentOS7Two主机上生成私钥和公钥
[root@CentOS7Two ~]# ssh-keygen -t rsa
4.在CentOS7Two中新建authorized_keys，权限为600，向其追加从CentOS7One拷贝的id_rsa.pub
[root@CentOS7Two .ssh]# touch authorized_keys [root@CentOS7Two .ssh]# chmod 600 authorized_keys [root@CentOS7Two .ssh]# cat /root/id_rsa.pub >> authorized_keys
authorized_keys是被允许免密连接到CentOS7Two的主机的公钥
[root@CentOS7Two .ssh]# cat authorized_keys ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABAQDGBMCc2AOqbMUT9uB8tDyrcNkjV/nCum/Ez5OMRZlVEITERtMLN8NUZP0kQjqiRU7kr5oIlA8FoE6sgSF7ciTu1IIDb2pg3roe0PpcaIwdSjP/b6ktnnkkcIXl74194AY/I+A9UFGErdaarTzAlMEougMC6G9IYTefqyMywsUZ5lH3PP72vqQNwZZX/LZtq6AK+yZ4C2jiErfF3i7hL1bTxVDBvGmLg37U8xNhei0Z5SDq9tCGP9EFabVuaw+mehPxGwFTbyuQj6X1xDmRD8lfjRWTK7M88dVImKdrf85KJAL5kyquIQi0tSAskkSlaroIDzNh1ebacKlOuWh6eWhd root@CentOS7One
方法2：
先在本机上生成私钥和公钥
[root@CentOS7Three ~]# ssh-keygen -t rsa
然后，运行ssh-copy-id命令，快捷的配置免密登录
[root@CentOS7Three ~]# ssh-copy-id CentOS7Seven
Hadoop运行
Hadoop格式化
对Hadoop格式化，对data目录下写初始化文件
hadoop namenode -format
启动dfs
启动目录在sbin/
start-dfs.sh
Jps检验是否启动成功
[root@CentOS7One current]# jps 5125 NameNode 5239 DataNode 5384 SecondaryNameNode 5487 Jps
启动yarn
start-yarn.sh
Jps检验是否成功
[root@CentOS7One current]# jps 5125 NameNode 5239 DataNode 5879 NodeManager 5384 SecondaryNameNode 5611 ResourceManager 5918 Jps
Hadoop命令
上传
hadoop fs -put hadoop-2.6.5.tar.gz hdfs://CentOS7One:9000/
hadoop fs -copyFromLocal /usr/local/hadoop-2.6.5/hadoop-2.6.5/share/hadoop/mapreduce/hadoop-mapreduce-examples-2.6.5.jar /
下载
hadoop fs -get hdfs://CentOS7One:9000/hadoop-2.6.5.tar.gz
创建文件
路径是hdfs://CenOS7One:9000/wordcount/input的简写
hadoop fs -mkdir /wordcount/input
查看文件列表
[root@CentOS7One mapreduce]# hadoop fs -ls /wordcount/output Found 2 items -rw-r--r-- 1 root supergroup 0 2019-02-12 10:34 /wordcount/output/_SUCCESS -rw-r--r-- 1 root supergroup 47 2019-02-12 10:34 /wordcount/output/part-r-00000
查看文件详情
[root@CentOS7One mapreduce]# hadoop fs -cat /wordcount/output/part-r-00000 fuckbaby 1 hello 5 jim 1 kitty 1 tom 1 world 1
删除文件
[root@CentOS7One ~]# hadoop fs -rm -r /qingshu.txt 19/03/01 10:59:34 INFO fs.TrashPolicyDefault: Namenode trash configuration: Deletion interval = 0 minutes, Emptier interval = 0 minutes. Deleted /qingshu.txt
查看文件系统空间使用情况
[root@CentOS7One data]# hadoop fs -df -h / Filesystem Size Used Available Use% hdfs://192.168.94.140:9000 17.5 G 210.6 M 14.1 G 1%
查看文件夹、文件具体占用多少空间
[root@CentOS7One data]# hadoop fs -du -s -h hdfs://CentOS7One:9000/* 190.4 M hdfs://CentOS7One:9000/hadoop-2.6.5.tar.gz 285.8 K hdfs://CentOS7One:9000/hadoop-mapreduce-examples-2.6.5.jar 18.1 M hdfs://CentOS7One:9000/mapreduce 0 hdfs://CentOS7One:9000/user 106 hdfs://CentOS7One:9000/wordcount