hadoop hbase hive spark对应版本

Hadoop集群安装配置教程_Hadoop3.1.3_Ubuntu

http://dblab.xmu.edu.cn/blog/2544-2/

林子雨编著《大数据技术原理与应用（第3版）》教材配套大数据软件安装和编程实践指南

http://dblab.xmu.edu.cn/post/13741/

hadoop hbase hive spark对应版本

https://blog.csdn.net/oppo62258801/article/details/103871069?utm_medium=distribute.pc_aggpage_search_result.none-task-blog-2~all~first_rank_v2~rank_v25-1-103871069.nonecase

hbase与phoenix整合(使用phoenix操作hbase数据)

http://blog.itpub.net/25854343/viewspace-2638600/

hbase时间不同步问题引起的bug

https://www.cnblogs.com/parent-absent-son/p/10096064.html

HBase启动后RegionServer自动挂原因及解决办法【ntp】

https://blog.csdn.net/csq031231/article/details/51011076

关闭Hbase出现stopping hbasecat:/tmp/hbase-root-master.pid:No such file or directory

https://blog.csdn.net/sinat_23225111/article/details/82695008

解决方式是在hbase-env.sh中修改pid文件的存放路径，配置项如下所示： # The directory where pid files are stored. /tmp by default. export HBASE_PID_DIR=/var/hadoop/pids

saveasnewapihadoopdatast 保存数据到hbase报空指针异常什么情况 python语言开发的

https://bbs.csdn.net/topics/392392966

conf = SparkConf().setMaster("local").setAppName("ReadHBase").set("spark.hadoop.validateOutputSpecs", False)

Linux重要命之sed命令详解

https://www.linuxprobe.com/detailed-description-of-sed.html

Linux sed命令完全攻略（超级详细）

http://c.biancheng.net/view/4028.html

Linux sed 命令

https://www.runoob.com/linux/linux-comm-sed.html

Sed命令中含有转义字符的解决方法

https://www.cnblogs.com/zwldyt/p/12996846.html

linux sed命令删除特殊字符(含斜线、冒号等转义字符)

https://blog.csdn.net/xhoufei2010/article/details/53521625/

CentOS 查看系统 CPU 个数、核心数、线程数

https://www.cnblogs.com/heqiuyong/p/11144652.html

Linux机器之间免密登录设置

https://blog.csdn.net/u013415591/article/details/81943189

Shell脚本实现SSH免密登录及批量配置管理

https://blog.csdn.net/qq_40907977/article/details/103258588?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.channel_param&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-1.channel_param

 https://www.cnblogs.com/30go/p/11458457.html

WARN: Establishing SSL connection without server's

https://blog.csdn.net/a458383896/article/details/86519220

大数据hive之hive连接mysql并启动，出现SSL警告，如何解决？

https://blog.csdn.net/baidu_34122324/article/details/84975839

分布式集群一键部署稳定版了解一下

https://blog.csdn.net/github_37835908/article/details/80150650?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-6.nonecase&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromBaidu-6.nonecase

mysql与hive2.1.1安装和配置

https://www.cnblogs.com/K-artorias/p/7141479.html

hive建表出错：Specified key was too long; max key length is 767 bytes

https://blog.csdn.net/qq_42826453/article/details/86182070

XML转义特殊字符

https://www.cnblogs.com/masonlu/p/9816418.html

&lt; < 小于号

&gt; > 大于号

&amp; & 和

&apos; ' 单引号

&quot; " 双引号

&nbsp; 空格

FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeExcepti

https://blog.csdn.net/qq_35078688/article/details/86137440

 

若不配置characterEncoding=UTF-8，则在hive中会出现中文乱码，若mysql开启了ssl验证，（高版本mysql默认开启）但是my.cnf里没有配置秘钥，则会出现以下通信错误。 

MySQL开启SSL认证,以及简单优化

https://www.cnblogs.com/so-cool/p/9239385.html

MySQL8中的SSL连接的关闭

https://jingyan.baidu.com/article/5552ef470a1522118ffbc9ef.html

FAILED: SemanticException org.apache.hadoop.hive.ql.metadata.HiveException: java.lang.RuntimeExcepti

https://blog.csdn.net/qq_35078688/article/details/86137440

hive的服务端没有打开 hive --service metastore & 然后Ctrl+C 再hive,进去

大数据hive之hive连接mysql并启动，出现SSL警告，如何解决？

https://blog.csdn.net/baidu_34122324/article/details/84975839

首先查看mysql　ssl是否开启

show global variables like '%ssl%';

启动Hive报错：mysql://localhost:3306/hive?createDatabaseIfNotExist=true, username = root. Terminating con

https://blog.csdn.net/weixin_43968936/article/details/102872636

hive安装配置过程中还需要注意的一些问题：

–>hive 2.0以上版本，安装配置完成之后需要先初始化元数据库

执行： schematool -dbType mysql -initSchema

–>比如一定要把这个mysql-connector-java-5.1.40-bin.jar包放在hive安装目录的lib下，不能是包含这个这个包的压缩包等。

–>Exception in thread “main” java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient 这个错误应该就是没有正确的将jar包放在Lib目录下。

–>启动hive过程中还遇到过，报这个警告，但是可以启动， Sat Nov 02 15:42:13 CST 2019 WARN: Establishing SSL connection without server’s identity verification is not recommended. According to MySQL 5.5.45+, 5.6.26+ and 5.7.6+ requirements SSL connection must be established by default if explicit option isn’t set. For compliance with existing applications not using SSL the verifyServerCertificate property is set to ‘false’. You need either to explicitly disable SSL by setting useSSL=false, or set useSSL=true and provide truststore for server certificate verification.

有说在hive配置文件中的mysql连接url中加一个参数的，

因为原因是MySQL在高版本需要指明是否进行SSL连接。

解决方案如下： 在mysql中查看有没有开启ssl

mysql> show global variables like ‘%ssl%’;

如果是have_ssl 对应disabled，

那就在hive配置文件conf/hive-site.xml中的mysql连接url中添加参数useSSL=false

hive配置MySQL时的乱码解决方案

https://blog.csdn.net/weixin_43087634/article/details/85218017

show variables like 'char%';

Hive之metastore服务启动错误解决方案org.apache.thrift.transport.TTransportException: Could not create ServerSock

https://www.pianshen.com/article/1984355270/

https://blog.csdn.net/weixin_45568892/article/details/105451958

错误org.apache.thrift.transport.TTransportException: Could not create ServerSocket on address 0.0.0.0/0.0.0.0:9083.

metastore重复启动

netstat -apn|grep 9083

执行查看linux端口命令，发现9083 端口被占用

kill进程后重新启动即可解决问题 先jps查看进程

重启hive metastore服务 

hive --service metastore &

Hive-异常处理Hive Schema version 2.3.0 does not match metastore's schema version 1.2.0 Metastore is not

https://blog.csdn.net/u014804456/article/details/77747720?utm_medium=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-3.control&depth_1-utm_source=distribute.pc_relevant.none-task-blog-BlogCommendFromMachineLearnPai2-3.control

https://github.com/apache/hive/blob/master/metastore/scripts/upgrade/mssql/upgrade-2.2.0-to-2.3.0.mssql.sql

MetaException(message:Hive Schema version 2.3.0 does not match metastore's schema version 1.2.0

https://blog.csdn.net/qq_39579408/article/details/86526757

https://blog.csdn.net/struggling_rong/article/details/82598277 

SBT命令行打包spark程序

https://zhuanlan.zhihu.com/p/65572399

下载安装配置 Spark-2.4.5 以及 sbt1.3.8 打包程序

https://blog.csdn.net/qq_21516633/article/details/105077947

安装最新版sbt工具方法和体会

http://dblab.xmu.edu.cn/blog/2546-2/#more-2546

Sbt——安装、配置、详细使用

https://blog.csdn.net/blueicex2017/article/details/104275168

scala的jar包在spark，scala，java上的执行

https://zhuanlan.zhihu.com/p/87355394

spark的动态资源配置

https://blog.csdn.net/yang735136055/article/details/100061133

Hadoop _ 疑难杂症 解决1 - WARN util.NativeCodeLoader: Unable to load native-hadoop library for your plat

https://blog.csdn.net/u010003835/article/details/81127984?utm_medium=distribute.pc_relevant_t0.none-task-blog-BlogCommendFromBaidu-1.control&depth_1-utm_source=distribute.pc_relevant_t0.none-task-blog-BlogCommendFromBaidu-1.control

hadoop 2.x安装：不能加载本地库 - java.library.path错误

https://www.cnblogs.com/erygreat/p/7223784.html

解决Hadoop启动时，没有启动datanode

http://dblab.xmu.edu.cn/blog/818-2/

关于Spark报错不能连接到Server的解决办法（Failed to connect to master master_hostname:7077）

https://blog.csdn.net/ybdesire/article/details/70666544

Spark启动的时候出现failed to launch: nice -n 0 /soft/spark/bin/spark-class org.apache.spark.deploy.worker

https://blog.csdn.net/qq_40707033/article/details/93210838

大部分人说需要在root用户下的.bashrc配置JAVA_HOME，试过之后发现还是原来的错误；

最后发现需要在/spark/sbin/spark-config.sh 加入JAVA_HOME的路径。

还有一个错误非常隐蔽，环境变量报错，可能是在/etc/profile.d目录下配置了环境变量。

Linux系统重启后/etc/hosts自动添加主机名解析

https://support.huaweicloud.com/intl/zh-cn/trouble-ecs/ecs_trouble_0320.html

https://blog.csdn.net/weixin_44706512/article/details/107332447

kafka启动方式

https://blog.csdn.net/m0_37690430/article/details/85098440

在Windows中 启动Kafka出现The Cluster ID doesn't match stored clusterId错误

https://www.jianshu.com/p/d51ef3369b37