一、编译
1.下载源码
2.编译
mvn clean install -DskipTests -DskipITs -Dscala-2.11 -Dspark3
报以下错误
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-compiler-plugin:3.8.0:compile (default-compile) on project hudi-integ-test: Compilation failure: Compilation failure: [ERROR] /Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/helpers/ZookeeperServiceProvider.java:[23,52] 程序包org.apache.hudi.common.testutils.minicluster不存在 [ERROR] /Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/helpers/ZookeeperServiceProvider.java:[39,7] 找不到符号 [ERROR] 符号: 类 ZookeeperTestService [ERROR] 位置: 类 org.apache.hudi.integ.testsuite.helpers.ZookeeperServiceProvider [ERROR] /Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/helpers/ZookeeperServiceProvider.java:[39,55] 找不到符号 [ERROR] 符号: 类 ZookeeperTestService [ERROR] 位置: 类 org.apache.hudi.integ.testsuite.helpers.ZookeeperServiceProvider [ERROR] /Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/helpers/ZookeeperServiceProvider.java:[46,7] 找不到符号 [ERROR] 符号: 类 ZookeeperTestService [ERROR] 位置: 类 org.apache.hudi.integ.testsuite.helpers.ZookeeperServiceProvider [ERROR] /Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-integ-test/src/main/java/org/apache/hudi/integ/testsuite/helpers/ZookeeperServiceProvider.java:[46,55] 找不到符号
解决：注释掉pom.xml文件中的
<module>hudi-integ-test</module> <module>packaging/hudi-integ-test-bundle</module>
再次编译即可。
[INFO] ------------------------------------------------------------------------ [INFO] Reactor Summary: [INFO] [INFO] Hudi ............................................... SUCCESS [ 4.854 s] [INFO] hudi-common ........................................ SUCCESS [ 29.059 s] [INFO] hudi-aws ........................................... SUCCESS [ 4.438 s] [INFO] hudi-timeline-service .............................. SUCCESS [ 2.899 s] [INFO] hudi-client ........................................ SUCCESS [ 0.146 s] [INFO] hudi-client-common ................................. SUCCESS [ 11.748 s] [INFO] hudi-hadoop-mr ..................................... SUCCESS [ 4.333 s] [INFO] hudi-spark-client .................................. SUCCESS [ 29.215 s] [INFO] hudi-sync-common ................................... SUCCESS [ 0.876 s] [INFO] hudi-hive-sync ..................................... SUCCESS [ 4.108 s] [INFO] hudi-spark-datasource .............................. SUCCESS [ 0.142 s] [INFO] hudi-spark-common_2.12 ............................. SUCCESS [ 14.039 s] [INFO] hudi-spark3_2.12 ................................... SUCCESS [ 9.784 s] [INFO] hudi-spark_2.12 .................................... SUCCESS [ 46.734 s] [INFO] hudi-utilities_2.12 ................................ SUCCESS [ 6.831 s] [INFO] hudi-utilities-bundle_2.12 ......................... SUCCESS [ 17.024 s] [INFO] hudi-cli ........................................... SUCCESS [ 14.301 s] [INFO] hudi-java-client ................................... SUCCESS [ 1.872 s] [INFO] hudi-flink-client .................................. SUCCESS [ 6.390 s] [INFO] hudi-spark2_2.12 ................................... SUCCESS [ 20.415 s] [INFO] hudi-dla-sync ...................................... SUCCESS [ 1.836 s] [INFO] hudi-sync .......................................... SUCCESS [ 0.068 s] [INFO] hudi-hadoop-mr-bundle .............................. SUCCESS [ 3.679 s] [INFO] hudi-hive-sync-bundle .............................. SUCCESS [ 1.424 s] [INFO] hudi-spark3.1.2-bundle_2.12 ........................ SUCCESS [ 7.476 s] [INFO] hudi-presto-bundle ................................. SUCCESS [ 4.860 s] [INFO] hudi-timeline-server-bundle ........................ SUCCESS [ 3.708 s] [INFO] hudi-hadoop-docker ................................. SUCCESS [ 0.693 s] [INFO] hudi-hadoop-base-docker ............................ SUCCESS [ 0.484 s] [INFO] hudi-hadoop-namenode-docker ........................ SUCCESS [ 0.071 s] [INFO] hudi-hadoop-datanode-docker ........................ SUCCESS [ 0.057 s] [INFO] hudi-hadoop-history-docker ......................... SUCCESS [ 0.057 s] [INFO] hudi-hadoop-hive-docker ............................ SUCCESS [ 0.694 s] [INFO] hudi-hadoop-sparkbase-docker ....................... SUCCESS [ 0.130 s] [INFO] hudi-hadoop-sparkmaster-docker ..................... SUCCESS [ 0.062 s] [INFO] hudi-hadoop-sparkworker-docker ..................... SUCCESS [ 0.072 s] [INFO] hudi-hadoop-sparkadhoc-docker ...................... SUCCESS [ 0.067 s] [INFO] hudi-hadoop-presto-docker .......................... SUCCESS [ 0.303 s] [INFO] hudi-examples ...................................... SUCCESS [ 10.590 s] [INFO] hudi-flink_2.12 .................................... SUCCESS [ 34.163 s] [INFO] hudi-kafka-connect ................................. SUCCESS [ 13.265 s] [INFO] hudi-flink-bundle_2.12 ............................. SUCCESS [ 13.849 s] [INFO] hudi-kafka-connect-bundle .......................... SUCCESS [ 11.594 s] [INFO] ------------------------------------------------------------------------ [INFO] BUILD SUCCESS [INFO] ------------------------------------------------------------------------ [INFO] Total time: 05:39 min [INFO] Finished at: 2022-02-08T11:31:28+08:00 [INFO] Final Memory: 340M/2823M [INFO] ------------------------------------------------------------------------
3.测试
cd /Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-cli
./hudi-cli.sh
SLF4J: Class path contains multiple SLF4J bindings. SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-cli/target/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: Found binding in [jar:file:/Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-cli/target/lib/log4j-slf4j-impl-2.6.2.jar!/org/slf4j/impl/StaticLoggerBinder.class] SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation. SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory] 0 [main] INFO org.springframework.beans.factory.xml.XmlBeanDefinitionReader - Loading XML bean definitions from URL [jar:file:/Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/hudi-cli/target/hudi-cli-0.10.1.jar!/META-INF/spring/spring-shell-plugin.xml] 424 [main] INFO org.springframework.context.support.GenericApplicationContext - Refreshing org.springframework.context.support.GenericApplicationContext@69ea3742: startup date [Tue Feb 08 14:02:51 CST 2022]; root of context hierarchy 649 [main] INFO org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor - JSR-330 'javax.inject.Inject' annotation found and supported for autowiring Table command getting loaded HoodieSplashScreen loaded =================================================================== * ___ ___ * * /\__\ ___ /\ \ ___ * * / / / /\__\ / \ \ /\ \ * * / /__/ / / / / /\ \ \ \ \ \ * * / \ \ ___ / / / / / \ \__\ / \__\ * * / /\ \ /\__\ / /__/ ___ / /__/ \ |__| / /\/__/ * * \/ \ \/ / / \ \ \ /\__\ \ \ \ / / / /\/ / / * * \ / / \ \ / / / \ \ / / / \ /__/ * * / / / \ \/ / / \ \/ / / \ \__\ * * / / / \ / / \ / / \/__/ * * \/__/ \/__/ \/__/ Apache Hudi CLI * * * =================================================================== Welcome to Apache Hudi CLI. Please type help if you are looking for help. hudi->version 1.0
/Users/FengZhen/Desktop/Hadoop/hudi/hudi-0.10.1/packaging此目录下就是hudi与各组件集成的jar包
Success.