=================== =============================== ----------------hadoop集群搭建 ----------------------- ====================== =========================== 192.168.75.7 255.255.255.0 192.168.75.2 00:50:56:27:0C:F1 ----------------------虚拟机基础配置 ------------------- 1.编辑硬件设备，设共享目录 2-添加hosts头 --------------------hosts头----------------------- 127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4 ::1 localhost localhost.localdomain localhost6 localhost6.localdomain6 ---------------------------------------------------- ------------------------------System eth0 网络配置 ------root--------- ----------------------- VM tool 安装 -----------------root----------- 1-虚拟机/安装VMware tool、双击打开VMwareTool tar包/解压到文件系统tmp ./tmp/vmware-tools-distrib/vmware-install.pl reboot 1.网络桥接 设置物理地址 vi /etc/sysconfig/network 2.面板删连接配置，只剩System eth0 rm -rf /etc/udev/rules.d/70-persistent-net.rules cp /mnt/hgfs/setup/hosts /etc/hosts reboot vi /etc/udev/rules.d/70-persistent-net.rules vi /etc/sysconfig/network-scripts/ifcfg-eth0 物理地址大写 -------------------------ifcfg-eth0文件--------------------- DEVICE="eth0" BOOTPROTO=none IPV6INIT="yes" NM_CONTROLLED="yes" ONBOOT="yes" TYPE="Ethernet" IPADDR=192.168.1.120 PREFIX=24 GATEWAY=192.168.1.1 DNS1=192.168.1.1 DEFROUTE=yes IPV4_FAILURE_FATAL=yes IPV6_AUTOCONF=yes IPV6_DEFROUTE=yes IPV6_FAILURE_FATAL=no NAME="System eth0" HWADDR=00:50:56:2A:C2:8D IPV6_PEERDNS=yes IPV6_PEERROUTES=yes -------------------------------------------------------- service iptables stop chkconfig iptables off service network restart -------------------- jdk 安装 -----------------root------- cp /mnt/hgfs/setup/jdk-8u211-linux-x64.rpm /opt/ rpm -ivh /mnt/hgfs/setup/jdk-8u211-linux-x64.rpm which java ll /usr/java/jdk1.8.0_161/bin/java Java路径：/usr/java/jdk1.8.0_161 vi /etc/profile ----------------------profile 文件--------------- export JAVA_HOME=/usr/java/jdk1.8.0_161 export JRE_HOME=$JAVA/jre export PATH=$JAVA_HOME/bin:$PATH export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar ----------------------------------------------- ***********************卸载JDK*********************************** rpm -qa|grep jdk ---看到：jdk-1.6.0_22-fcs rpm -e --nodeps jdk-1.6.0_22-fcs *************************************************************** source /etc/profile java -version =============== Hadoop 安装 ------先root后分配权限给Hadoop ========= cp /mnt/hgfs/setup/hadoop-2.7.6.tar.gz /opt/ tar -zxvf /opt/hadoop-2.7.6.tar.gz -C /opt/ vi /etc/profile ----------------------profile 文件--------------- export HADOOP_DEV_HOME=/opt/hadoop-2.7.6 export PATH=$PATH:$HADOOP_DEV_HOME/bin export PATH=$PATH:$HADOOP_DEV_HOME/sbin export HADOOP_MAPARED_HOME=${HADOOP_DEV_HOME} export HADOOP_COMMON_HOME=${HADOOP_DEV_HOME} export HADOOP_HDFS_HOME=${HADOOP_DEV_HOME} export YARN_HOME=${HADOOP_DEV_HOME} export HADOOP_CONF_DIR=${HADOOP_DEV_HOME}/etc/hadoop ----------------------------------------------- source /etc/profile vi /opt/hadoop-2.7.6/etc/hadoop/hadoop-env.sh ----------------------hadoop-env.sh 文件--------------- export JAVA_HOME=/usr/java/jdk1.8.0_161 ------------------------------------------------------------ vi /opt/hadoop-2.7.6/etc/hadoop/core-site.xml -------- core-site.xml 文件-------要在/opt/hadoop-2.7.6/新建目录 /data/tmp--- <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://cMater:9000</value> </property> <property> <name>hadoop.tmp.dir</name> <value>/root/bigdata/hadoop-2.6.5/data/tmp</value> </property> </configuration> ----------------------------------------------------------------------------- vi /opt/hadoop-2.7.6/etc/hadoop/hdfs-site.xml ------ hdfs-site.xml 文件----要在/opt/hadoop-2.7.6/新建目录 /data/cMater /data/namenode /data/checkpoint ------- <configuration> <property> <name>dfs.datanode.data.dir</name> <value>/opt/hadoop-2.7.6/data/cMater</value> </property> <property> <name>dfs.namenode.name.dir</name> <value>/opt/hadoop-2.7.6/data/namenode</value> </property> <property> <name>dfs.namenode.checkpoint.dir</name> <value>/opt/hadoop-2.7.6/data/checkpoint</value> </property> <property> <name>dfs.replication</name> <value>2</value> </property> </configuration> ------------------------------------------------------------ mkdir -p /opt/hadoop-2.7.6/data/tmp mkdir -p /opt/hadoop-2.7.6/data/cMater mkdir -p /opt/hadoop-2.7.6/data/namenode mkdir -p /opt/hadoop-2.7.6/data/checkpoint vi /opt/hadoop-2.7.6/etc/hadoop/yarn-site.xml ---------------------- yarn-site.xml 文件--------------- <configuration> <property> <name>yarn.nodemanager.aux-services</name> <value>mapreduce_shuffle</value> </property> <property> <name>yarn.resourcemanager.hostname</name> <value>cMater</value> </property> </configuration> ------------------------------------------------------------ cp /opt/hadoop-2.7.6/etc/hadoop/mapred-site.xml.template /opt/hadoop-2.7.6/etc/hadoop/mapred-site.xml vi /opt/hadoop-2.7.6/etc/hadoop/mapred-site.xml ---------------------- mapred-site.xml 文件--------------- <configuration> <property> <name>mapreduce.framework.name</name> <value>yarn</value> </property> <property> <name>mapreduce.jobtracker.staging.root.dir</name> <value>/user</value> </property> </configuration> ------------------------------------------------------------ ========================用户 权限====================== useradd hadoop passwd hadoop chmod u+w /etc/sudoers vi /etc/sudoers ------------sudoers 文件配置---root ALL=(ALL) ALL下添加一行---- hadoop ALL=(ALL) ALL ---------------------------------------------------------------------- chmod u-w /etc/sudoers chown -R hadoop:hadoop /opt/hadoop-2.7.6 ============启动 Hadoop ========切换Hadoop用户============= su hadoop ******格式化 HDFS**************** hdfs namenode -format *******单点重复格式化处理******** rm -rf 进行清空 /data/tmp /data/cMater /data/namenode /data/checkpoint 内的缓存 ***********单点启动失败********* 检查core-site.xml、hdfs-site.xml 、yarn-site.xml、mapred-site.xml文件格式 source /etc/hosts 失败 --------------------单节点启动--------------------------------namenode---- //hadoop-daemon.sh start secondarynamenode hadoop-daemon.sh start namenode hadoop-daemon.sh start datanode yarn-daemon.sh start resourcemanager yarn-daemon.sh start nodemanager mr-jobhistory-daemon.sh start historyserver yarn-daemon.sh start proxyserver //hadoop-daemon.sh stop secondarynamenode hadoop-daemon.sh stop namenode hadoop-daemon.sh stop datanode yarn-daemon.sh stop resourcemanager yarn-daemon.sh stop nodemanager mr-jobhistory-daemon.sh stop historyserver yarn-daemon.sh stop proxyserver ---------------- ./opt/hadoop-2.7.6/sbin/hadoop-daemon.sh start namenode -----------datanode----Hadoop找不到类时检查nodemanager是否拼写正确-------- hadoop-daemon.sh start datanode yarn-daemon.sh start nodemanager hadoop-daemon.sh stop datanode yarn-daemon.sh stop nodemanager -------------------端口解析----------------- HDFS 50070 http服务的端口 yarn 8088 http服务的端口 proxyserver WebAppProxyServer history JobHistoryServer =========================================== --------------------HDFS 权限问题--------------------- useradd 20160216048 # root操作 passwd 20160216048 #root操作 sudo usermod -a -G hadoop 20160216048 # root操作 20160216048添加到Hadoop组 hadoop fs -put /opt/hadoop-2.7.6.tar.gz /root/ #hadoop操作,root无法上传 hadoop fs -chown -R 20160216048:hadoop /root/ #hadoop操作 hadoop dfs -chmod -R 755 /abc #hadoop操作 hadoop dfsadmin -safemode leave #解除hadoop的安全模式 hadoop dfs -chmod -R 777 /abc #hadoop操作,修改权限，让组外用户进行操作 __________________________权限比重_______________________ drwxr-xr-x -rw-r--r-- d:文件夹 r:4 w:2 x:1 注意：hadoop fs -rm -r /tmp/ 与hadoop fs -rm -r /tmp没有区别！ ========================== ========================== ------------------- HDFS 使用 ---------------------- ===================== ========================== ------------------杀掉当前正在执行的hadoop任务--------------- 列出当前hadoop正在执行的jobs: hadoop job -list 杀掉job: job_201212111628_11166：
[hadoop@192.168.10.11 bin]$ ./hadoop job -kill job_201212111628_11166 -----------------------Linux自定义命令---------------------- vi /root/.bashrc alias 你自定的命令=系统原来的命令 如：alias hf='hadoop fs' -----------------------------HDFS文件回传---------------------- hadoop fs -get /user/root/ds_out /mnt/hgfs/setup/data/ *****************************Linux补充 免密码登录*************** 方法一：主从主机间要相互做一次 ssh-keygen -t rsa ssh-copy-id -i ~/.ssh/id_rsa.pub cSlave01 方法二： 全部会话 ： ssh-keygen -t rsa -P '' -f ~/.ssh/id_rsa ssh-copy-id node11/node12/node13 ------------------------------禁用IPV6----------------------- ## 禁用IPv6 echo " " >> /etc/modprobe.d/dist.conf echo "alias net-pf-10 off" >> /etc/modprobe.d/dist.conf echo "alias ipv6 off" >> /etc/modprobe.d/dist.conf ## 查看是否追加成功 tail /etc/modprobe.d/dist.conf ---------------------设置文件打开数目和用户最大进程数--------------- ## 设置用户最大进程数 vim /etc/security/limits.conf ## 结尾添加以下内容 * soft nofile 32768 * hard nofile 1048576 * soft nproc 65536 * hard nproc unlimited * soft memlock unlimited * hard memlock unlimited ---------------------------时区设置------------------------- 统一时区为东八区 我们国家的时区为东八区（+0800）： #查看当前时区 date -R;cat /etc/sysconfig/clock 不是北京时间，要设置一下时区，方法如下，执行命令： # 设置东八区时区为当前时区 rm -rf /etc/localtime cp /usr/share/zoneinfo/Asia/Shanghai /etc/localtime # 手动同步下网络时间 ntpdate -u cn.pool.ntp.org vim /etc/sysconfig/clock ZONE="Asia/Shanghai" 在次查看 date -R;cat /etc/sysconfig/clock ----------------------------Linux操作系统系统语言采用------------------ -------------转英文--------------- # 查看操作系统系统语言 echo $LANG # 修改操作系统系统语言 vim /etc/sysconfig/i18n LANG="en_US.UTF-8" ----------------转中文------------ # 查看操作系统系统语言 echo $LANG # 修改操作系统系统语言 vim /etc/sysconfig/i18n LANG="zh_CN.UTF-8"