官方参考文档:
https://www.cloudera.com/documentation/enterprise/5-15-x/topics/configure_cm_repo.html
安装cdh前如果安装过其他版本的,记得删除各种目录残留文件,比如/run下面的
如果误删了log4j/properties文件,文件内容在这
cmf.root.logger=INFO,CONSOLE
cmf.log.dir=.
cmf.log.file=cmf-server.log
# Define the root logger to the system property "cmf.root.logger".
log4j.rootLogger=${cmf.root.logger}
# Logging Threshold
log4j.threshhold=ALL
# Disable most JDBC tracing by default.
log4j.logger.org.jdbcdslog=FATAL
# Disable overly loud Avro IPC logging
log4j.logger.org.apache.avro.ipc.NettyTransceiver=FATAL
# Disable overly loud Flume config validation logging
log4j.logger.org.apache.flume.conf.FlumeConfiguration=ERROR
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.target=System.err
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} %5p [%t:%c{2}@%L] %m%n
log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.LOGFILE.MaxFileSize=10MB
log4j.appender.LOGFILE.MaxBackupIndex=10
log4j.appender.LOGFILE.File=${cmf.log.dir}/${cmf.log.file}
log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.LOGFILE.layout.ConversionPattern=%d{ISO8601} %5p [%t:%c{2}@%L] %m%n
1.配置源
wget https://archive.cloudera.com/cm5/ubuntu/xenial/amd64/cm/archive.key sudo apt-key add archive.key cd /etc/apt/sources.list.d/ wget https://archive.cloudera.com/cm5/ubuntu/xenial/amd64/cm/cloudera.list curl -s https://archive.cloudera.com/cm5/ubuntu/trusty/amd64/cm/archive.key| sudo apt-key add - sudo apt-get update
2.安装Java
3.关防火墙
参考
https://wenku.baidu.com/view/5462a2132f3f5727a5e9856a561252d380eb20e5.html
sudo service ufw stop
4.安装CDH server
sudo apt-get install cloudera-manager-daemons cloudera-manager-server
或者通过下载的deb包来进行安装
sudo dpkg -i ./cloudera-manager-daemons_5.16.2-1.cm5162.p0.7~xenial-cm5_all.deb sudo dpkg -i ./cloudera-manager-server_5.16.2-1.cm5162.p0.7~xenial-cm5_all.deb sudo dpkg -i ./cloudera-manager-agent_5.16.2-1.cm5162.p0.7~xenial-cm5_amd64.deb
期间如果依赖不满足的话
sudo apt-get install -f
4.安装MySQL JDBC Driver
sudo apt-get install libmysql-java
5.按照官方教程配置数据库等
CREATE DATABASE scm DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE amon DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE rman DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE hue DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE metastore DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE sentry DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE nav DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE navms DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE DATABASE oozie DEFAULT CHARACTER SET utf8 DEFAULT COLLATE utf8_general_ci; CREATE USER 'scm'; CREATE USER 'amon'; CREATE USER 'rman'; CREATE USER 'hue'; CREATE USER 'hive'; CREATE USER 'sentry'; CREATE USER 'nav'; CREATE USER 'navms'; CREATE USER 'oozie'; GRANT ALL ON scm.* TO 'scm'@'%' IDENTIFIED BY 'scm'; GRANT ALL ON amon.* TO 'amon'@'%' IDENTIFIED BY 'amon'; GRANT ALL ON rman.* TO 'rman'@'%' IDENTIFIED BY 'rman'; GRANT ALL ON hue.* TO 'hue'@'%' IDENTIFIED BY 'hue'; GRANT ALL ON metastore.* TO 'hive'@'%' IDENTIFIED BY 'hive'; GRANT ALL ON sentry.* TO 'sentry'@'%' IDENTIFIED BY 'sentry'; GRANT ALL ON nav.* TO 'nav'@'%' IDENTIFIED BY 'nav'; GRANT ALL ON navms.* TO 'navms'@'%' IDENTIFIED BY 'navms'; GRANT ALL ON oozie.* TO 'oozie'@'%' IDENTIFIED BY 'oozie';
7.初始化数据库,每执行完要输入密码
sudo /usr/share/cmf/schema/scm_prepare_database.sh mysql scm scm
8.启动
systemctl start cloudera-scm-server systemctl start cloudera-scm-agent
1.如果启动的时候,/var/log/cloudera-scm-server/cloudera-scm-server.out出现
Caused by: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'entityManagerFactoryBean': FactoryBean threw exception on object creation; nested exception is javax.persistence.PersistenceException: org.hibernate.exception.GenericJDBCException: Could not open connection
可以参考
https://blog.csdn.net/qq_41623990/article/details/83008860
在/usr/share/cmf/schema执行
bash scm_prepare_database.sh mysql -uroot -p --scm-host localhost scm scm scm
2.如果启动的时候,/var/log/cloudera-scm-server/cloudera-scm-server.out出现,但是cloudera-scm-server.log和cmf-server-perf.log日志都没有
Caused by: org.springframework.beans.BeanInstantiationException: Could not instantiate bean class [com.cloudera.server.cmf.log.components.ServerLogFetcherImpl]: Constructor threw exception; nested exception is java.io.FileNotFoundException: Unable to locate the Cloudera Manager log file in the log4j settings Caused by: java.io.FileNotFoundException: Unable to locate the Cloudera Manager log file in the log4j settings
查看/etc/cloudera-scm-server/log4j.properties是否为空
为空的话,加入内容,启动cloudera-scm-server,然后cloudera-scm-server.log和cmf-server-perf.log日志都成功出现
# Copyright (c) 2012 Cloudera, Inc. All rights reserved.
#
# !!!!! IMPORTANT !!!!!
# The Cloudera Manager server finds its log file by querying log4j. It
# assumes that the first file appender in this file is the server log.
# See LogUtil.getServerLogFile() for more details.
#
# Define some default values that can be overridden by system properties
cmf.root.logger=INFO,CONSOLE
cmf.log.dir=.
cmf.log.file=cmf-server.log
cmf.perf.log.file=cmf-server-perf.log
# Define the root logger to the system property "cmf.root.logger".
log4j.rootLogger=${cmf.root.logger}
# Logging Threshold
log4j.threshhold=ALL
# Disable most JDBC tracing by default.
log4j.logger.org.jdbcdslog=FATAL
# Disable overly loud Avro IPC logging
log4j.logger.org.apache.avro.ipc.NettyTransceiver=FATAL
# Disable overly loud Flume config validation logging
log4j.logger.org.apache.flume.conf.FlumeConfiguration=ERROR
# Disable overly loud CXF logging
log4j.logger.org.apache.cxf.phase.PhaseInterceptorChain=ERROR
# Disable "Mapped URL path" messages from Spring
log4j.logger.org.springframework.web.servlet.mvc.annotation.DefaultAnnotationHandlerMapping=WARN
log4j.appender.CONSOLE=org.apache.log4j.ConsoleAppender
log4j.appender.CONSOLE.target=System.err
log4j.appender.CONSOLE.layout=org.apache.log4j.PatternLayout
log4j.appender.CONSOLE.layout.ConversionPattern=%d{ISO8601} %p %t:%c: %m%n
log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.LOGFILE.MaxFileSize=10MB
log4j.appender.LOGFILE.MaxBackupIndex=10
log4j.appender.LOGFILE.File=${cmf.log.dir}/${cmf.log.file}
log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.LOGFILE.layout.ConversionPattern=%d{ISO8601} %p %t:%c: %m%n
log4j.appender.LOGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.LOGFILE.MaxFileSize=10MB
log4j.appender.LOGFILE.MaxBackupIndex=10
log4j.appender.LOGFILE.File=${cmf.log.dir}/${cmf.log.file}
log4j.appender.LOGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.LOGFILE.layout.ConversionPattern=%d{ISO8601} %p %t:%c: %m%n
log4j.additivity.com.cloudera.server.cmf.debug.components.PerfLogger=false
log4j.logger.com.cloudera.server.cmf.debug.components.PerfLogger=INFO,PERFLOGFILE
log4j.appender.PERFLOGFILE=org.apache.log4j.RollingFileAppender
log4j.appender.PERFLOGFILE.MaxFileSize=10MB
log4j.appender.PERFLOGFILE.MaxBackupIndex=10
log4j.appender.PERFLOGFILE.File=${cmf.log.dir}/${cmf.perf.log.file}
log4j.appender.PERFLOGFILE.layout=org.apache.log4j.PatternLayout
log4j.appender.PERFLOGFILE.layout.ConversionPattern=%d{ISO8601} %p %t:%c: %m%n
之后访问 http://localhost:7180/cmf/login
向集群添加主机
填master的host
继续
继续
9.安装的时候提供 SSH 登录凭据使用root和密码
或者使用私钥,要保证root用户可以用私钥ssh过去
如果安装的时候报
sed: can't read /etc/cloudera-scm-agent/config.ini: No such file or directory
拷贝一个正确的文件,把文件的内容补上
如果遇到 安装失败。 无法接收 Agent 发出的检测信号。
将/etc/cloudera-scm-agent/config.ini中security的注释掉,并将设置use_tls=0,就能安装成功
安装parcel,等待即可
完成
继续安装组件
分配角色
配置集群数据库
继续
全部启动成功
9.安装Manager Service,否则会包host monitor连不上
需要在这一步输入report manager的数据库账号密码,如果连不上可能是防火墙没关,
mysql的host填localhost
10.查看parcels的下载情况,如果要离线安装的,将parcels下载后拷贝到/opt/cloudera/parcel-repo目录,注意修改权限一起重启server和agent
/opt/cloudera/parcel-repo$ ls -alh 总用量 2.4G drwxr-xr-x 2 cloudera-scm cloudera-scm 4.0K 12月 15 11:31 . drwxr-xr-x 6 cloudera-scm cloudera-scm 4.0K 12月 15 00:03 .. -rwxr-xr-x 1 cloudera-scm cloudera-scm 1.9G 12月 15 11:12 CDH-5.16.2-1.cdh5.16.2.p0.8-xenial.parcel -rwxr-xr-x 1 cloudera-scm cloudera-scm 41 12月 15 11:12 CDH-5.16.2-1.cdh5.16.2.p0.8-xenial.parcel.sha -rw-r----- 1 cloudera-scm cloudera-scm 75K 12月 15 11:31 CDH-5.16.2-1.cdh5.16.2.p0.8-xenial.parcel.torrent -rw-r----- 1 cloudera-scm cloudera-scm 84M 12月 15 02:55 KAFKA-4.1.0-1.4.1.0.p0.4-xenial.parcel -rw-r----- 1 cloudera-scm cloudera-scm 41 12月 15 02:55 KAFKA-4.1.0-1.4.1.0.p0.4-xenial.parcel.sha -rw-r----- 1 cloudera-scm cloudera-scm 3.5K 12月 15 02:55 KAFKA-4.1.0-1.4.1.0.p0.4-xenial.parcel.torrent -rw-r----- 1 cloudera-scm cloudera-scm 453M 12月 15 04:29 KUDU-1.4.0-1.cdh5.12.2.p0.8-xenial.parcel -rw-r----- 1 cloudera-scm cloudera-scm 41 12月 15 04:29 KUDU-1.4.0-1.cdh5.12.2.p0.8-xenial.parcel.sha -rw-r----- 1 cloudera-scm cloudera-scm 18K 12月 15 04:29 KUDU-1.4.0-1.cdh5.12.2.p0.8-xenial.parcel.torrent -rw-r--r-- 1 cloudera-scm cloudera-scm 66K 6月 18 21:21 manifest.json
安装kafka，分配并激活
然后点击添加服务，选择kafka进行安装
失败，查看stderr日志
发现
+ exec /opt/cloudera/parcels/KAFKA-4.1.0-1.4.1.0.p0.4/lib/kafka/bin/kafka-server-start.sh /run/cloudera-scm-agent/process/106-kafka-KAFKA_BROKER/kafka.properties 三月 08, 2020 2:29:36 下午 org.glassfish.jersey.internal.inject.Providers checkProviderRuntime 警告: A provider nl.techop.kafka.KafkaTopicsResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider nl.techop.kafka.KafkaTopicsResource will be ignored. 三月 08, 2020 2:29:36 下午 org.glassfish.jersey.internal.inject.Providers checkProviderRuntime 警告: A provider nl.techop.kafka.TopicMetricNameResource registered in SERVER runtime does not implement any provider interfaces applicable in the SERVER runtime. Due to constraint configuration problems the provider nl.techop.kafka.TopicMetricNameResource will be ignored. Redaction rules file doesn't exist, not redacting logs. file: redaction-rules.json, directory: /run/cloudera-scm-agent/process/106-kafka-KAFKA_BROKER
所以执行命令
/opt/cloudera/parcels/KAFKA-4.1.0-1.4.1.0.p0.4/lib/kafka/bin/kafka-server-start.sh /run/cloudera-scm-agent/process/106-kafka-KAFKA_BROKER/kafka.properties
发现是没有log4j.peoperties文件
/opt/cloudera/parcels/KAFKA-4.1.0-1.4.1.0.p0.4/lib/kafka/bin/kafka-server-start.sh /run/cloudera-scm-agent/process/106-kafka-KAFKA_BROKER/kafka.properties log4j:ERROR Could not read configuration file from URL [file:/opt/cloudera/parcels/KAFKA-4.1.0-1.4.1.0.p0.4/lib/kafka/bin/../config/log4j.properties]. java.io.FileNotFoundException: /opt/cloudera/parcels/KAFKA-4.1.0-1.4.1.0.p0.4/lib/kafka/bin/../config/log4j.properties (没有那个文件或目录) at java.io.FileInputStream.open0(Native Method) at java.io.FileInputStream.open(FileInputStream.java:195) at java.io.FileInputStream.<init>(FileInputStream.java:138) at java.io.FileInputStream.<init>(FileInputStream.java:93) at sun.net.www.protocol.file.FileURLConnection.connect(FileURLConnection.java:90) at sun.net.www.protocol.file.FileURLConnection.getInputStream(FileURLConnection.java:188) at org.apache.log4j.PropertyConfigurator.doConfigure(PropertyConfigurator.java:557) at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526) at org.apache.log4j.LogManager.<clinit>(LogManager.java:127) at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:66) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:358) at com.typesafe.scalalogging.Logger$.apply(Logger.scala:48) at kafka.utils.Log4jControllerRegistration$.<init>(Logging.scala:25) at kafka.utils.Log4jControllerRegistration$.<clinit>(Logging.scala) at kafka.utils.Logging$class.$init$(Logging.scala:47) # Change the two lines below to adjust the general broker logging level (output to server.log and stdout) at com.cloudera.kafka.wrap.Kafka$.<init>(Kafka.scala:30) at com.cloudera.kafka.wrap.Kafka$.<clinit>(Kafka.scala) at com.cloudera.kafka.wrap.Kafka.main(Kafka.scala) log4j:ERROR Ignoring configuration file [file:/opt/cloudera/parcels/KAFKA-4.1.0-1.4.1.0.p0.4/lib/kafka/bin/../config/log4j.properties].
添加该文件，再执行，发现
kafka.common.InconsistentBrokerIdException: Configured broker.id 72 doesn't match stored broker.id 94 in meta.properties. If you moved your data, make sure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).
去kafka的配置里面将broke.id从72修改成94
启动成功
