编译 flink 1.12.0

参考文档： 编译Flink 1.9.0  https://www.cnblogs.com/Springmoon-venn/p/11512136.html

编译环境： ubuntu 18.05
Java : 1.8.0_271

官网下载源码包：https://mirrors.tuna.tsinghua.edu.cn/apache/flink/flink-1.12.0/flink-1.12.0-src.tgz

解压

tar -zxvf flink-1.12.0-src.tgz

ls flink-1.12.0*
flink-1.12.0-src.tgz

flink-1.12.0:
build-target              flink-java             flink-state-backends
docs                      flink-jepsen           flink-streaming-java
flink-annotations         flink-kubernetes       flink-streaming-scala
flink-clients             flink-libraries        flink-table
flink-connectors          flink-mesos            flink-tests
flink-container           flink-metrics          flink-test-utils-parent
flink-contrib             flink-ml-parent        flink-walkthroughs
flink-core                flink-optimizer        flink-yarn
flink-dist                flink-parent.iml       flink-yarn-tests
flink-docs                flink-python           LICENSE
flink-end-to-end-tests    flink-queryable-state  licenses
flink-examples            flink-quickstart       NOTICE
flink-external-resources  flink-runtime          pom.xml
flink-filesystems         flink-runtime-web      README.md
flink-formats             flink-scala            target
flink-fs-tests            flink-scala-shell      tools

进入目录，编译

mvn clean package -DskipTests 

注：flink 1.11 后就不再与 hadoop 版本绑定，不需要编译特定 hadoop 版本 的 shaded 包

flink 1.11.0 release notes : https://flink.apache.org/news/2020/07/06/release-1.11.0.html

[FLINK-11086] Flink now supports Hadoop versions above Hadoop 3.0.0. Note that the Flink project does not provide any updated “flink-shaded-hadoop-*” jars.

Users need to provide Hadoop dependencies through the HADOOP_CLASSPATH environment variable (recommended) or the lib/ folder.

编译结果：

[INFO] Reactor Summary:
[INFO] 
[INFO] Flink : Tools : Force Shading ...................... SUCCESS [ 13.192 s]
[INFO] Flink : ............................................ SUCCESS [  6.506 s]
[INFO] Flink : Annotations ................................ SUCCESS [  2.445 s]
[INFO] Flink : Test utils : ............................... SUCCESS [  0.126 s]
[INFO] Flink : Test utils : Junit ......................... SUCCESS [  1.486 s]
[INFO] Flink : Metrics : .................................. SUCCESS [  0.087 s]
[INFO] Flink : Metrics : Core ............................. SUCCESS [  0.725 s]
[INFO] Flink : Core ....................................... SUCCESS [ 26.911 s]
[INFO] Flink : Java ....................................... SUCCESS [  5.101 s]
[INFO] Flink : Queryable state : .......................... SUCCESS [  0.067 s]
[INFO] Flink : Queryable state : Client Java .............. SUCCESS [  0.830 s]
[INFO] Flink : FileSystems : .............................. SUCCESS [  0.063 s]
[INFO] Flink : FileSystems : Hadoop FS .................... SUCCESS [  8.505 s]
[INFO] Flink : Runtime .................................... SUCCESS [01:33 min]
[INFO] Flink : Scala ...................................... SUCCESS [ 27.175 s]
[INFO] Flink : FileSystems : Mapr FS ...................... SUCCESS [  0.895 s]
[INFO] Flink : FileSystems : Hadoop FS shaded ............. SUCCESS [  3.908 s]
[INFO] Flink : FileSystems : S3 FS Base ................... SUCCESS [  1.645 s]
[INFO] Flink : FileSystems : S3 FS Hadoop ................. SUCCESS [  5.757 s]
[INFO] Flink : FileSystems : S3 FS Presto ................. SUCCESS [  6.342 s]
[INFO] Flink : FileSystems : Swift FS Hadoop .............. SUCCESS [ 17.588 s]
[INFO] Flink : FileSystems : OSS FS ....................... SUCCESS [  5.269 s]
[INFO] Flink : FileSystems : Azure FS Hadoop .............. SUCCESS [  9.854 s]
[INFO] Flink : Optimizer .................................. SUCCESS [ 11.023 s]
[INFO] Flink : Connectors : ............................... SUCCESS [  0.088 s]
[INFO] Flink : Connectors : File Sink Common .............. SUCCESS [  0.340 s]
[INFO] Flink : Streaming Java ............................. SUCCESS [ 12.088 s]
[INFO] Flink : Clients .................................... SUCCESS [  2.695 s]
[INFO] Flink : Test utils : Utils ......................... SUCCESS [  1.408 s]
[INFO] Flink : Runtime web ................................ SUCCESS [11:09 min]
[INFO] Flink : Examples : ................................. SUCCESS [  0.713 s]
[INFO] Flink : Examples : Batch ........................... SUCCESS [ 16.444 s]
[INFO] Flink : Connectors : Hadoop compatibility .......... SUCCESS [ 12.183 s]
[INFO] Flink : State backends : ........................... SUCCESS [  0.089 s]
[INFO] Flink : State backends : RocksDB ................... SUCCESS [  2.446 s]
[INFO] Flink : Tests ...................................... SUCCESS [ 30.561 s]
[INFO] Flink : Streaming Scala ............................ SUCCESS [ 21.328 s]
[INFO] Flink : Connectors : HCatalog ...................... SUCCESS [  6.485 s]
[INFO] Flink : Test utils : Connectors .................... SUCCESS [  0.452 s]
[INFO] Flink : Connectors : Base .......................... SUCCESS [  0.623 s]
[INFO] Flink : Connectors : Files ......................... SUCCESS [  1.767 s]
[INFO] Flink : Table : .................................... SUCCESS [  0.070 s]
[INFO] Flink : Table : Common ............................. SUCCESS [  5.397 s]
[INFO] Flink : Table : API Java ........................... SUCCESS [  2.381 s]
[INFO] Flink : Table : API Java bridge .................... SUCCESS [  1.176 s]
[INFO] Flink : Table : API Scala .......................... SUCCESS [  7.939 s]
[INFO] Flink : Table : API Scala bridge ................... SUCCESS [  7.045 s]
[INFO] Flink : Table : SQL Parser ......................... SUCCESS [ 26.022 s]
[INFO] Flink : Libraries : ................................ SUCCESS [  0.106 s]
[INFO] Flink : Libraries : CEP ............................ SUCCESS [  3.086 s]
[INFO] Flink : Table : Planner ............................ SUCCESS [01:33 min]
[INFO] Flink : Table : SQL Parser Hive .................... SUCCESS [  6.621 s]
[INFO] Flink : Table : Runtime Blink ...................... SUCCESS [ 11.644 s]
[INFO] Flink : Table : Planner Blink ...................... SUCCESS [02:01 min]
[INFO] Flink : Formats : .................................. SUCCESS [  0.236 s]
[INFO] Flink : Formats : Json ............................. SUCCESS [  1.569 s]
[INFO] Flink : Connectors : Elasticsearch base ............ SUCCESS [  8.103 s]
[INFO] Flink : Connectors : Elasticsearch 5 ............... SUCCESS [ 13.526 s]
[INFO] Flink : Connectors : Elasticsearch 6 ............... SUCCESS [  4.407 s]
[INFO] Flink : Connectors : Elasticsearch 7 ............... SUCCESS [  3.241 s]
[INFO] Flink : Connectors : HBase base .................... SUCCESS [ 22.812 s]
[INFO] Flink : Connectors : HBase 1.4 ..................... SUCCESS [  4.806 s]
[INFO] Flink : Connectors : HBase 2.2 ..................... SUCCESS [  7.163 s]
[INFO] Flink : Formats : Hadoop bulk ...................... SUCCESS [  0.864 s]
[INFO] Flink : Formats : Orc .............................. SUCCESS [  1.793 s]
[INFO] Flink : Formats : Orc nohive ....................... SUCCESS [  1.579 s]
[INFO] Flink : Formats : Avro ............................. SUCCESS [  5.951 s]
[INFO] Flink : Formats : Parquet .......................... SUCCESS [ 14.026 s]
[INFO] Flink : Formats : Csv .............................. SUCCESS [  1.059 s]
[INFO] Flink : Connectors : Hive .......................... SUCCESS [ 12.772 s]
[INFO] Flink : Connectors : JDBC .......................... SUCCESS [  2.882 s]
[INFO] Flink : Connectors : RabbitMQ ...................... SUCCESS [  1.091 s]
[INFO] Flink : Connectors : Twitter ....................... SUCCESS [  2.206 s]
[INFO] Flink : Connectors : Nifi .......................... SUCCESS [  1.343 s]
[INFO] Flink : Connectors : Cassandra ..................... SUCCESS [  4.109 s]
[INFO] Flink : Metrics : JMX .............................. SUCCESS [  1.350 s]
[INFO] Flink : Connectors : Kafka ......................... SUCCESS [  5.280 s]
[INFO] Flink : Connectors : Google PubSub ................. SUCCESS [  3.571 s]
[INFO] Flink : Connectors : Kinesis ....................... SUCCESS [ 13.685 s]
[INFO] Flink : Connectors : SQL : Elasticsearch 6 ......... SUCCESS [  5.171 s]
[INFO] Flink : Connectors : SQL : Elasticsearch 7 ......... SUCCESS [  6.483 s]
[INFO] Flink : Connectors : SQL : HBase 1.4 ............... SUCCESS [  6.591 s]
[INFO] Flink : Connectors : SQL : HBase 2.2 ............... SUCCESS [ 11.668 s]
[INFO] Flink : Connectors : SQL : Hive 1.2.2 .............. SUCCESS [  5.588 s]
[INFO] Flink : Connectors : SQL : Hive 2.2.0 .............. SUCCESS [  5.818 s]
[INFO] Flink : Connectors : SQL : Hive 2.3.6 .............. SUCCESS [  5.212 s]
[INFO] Flink : Connectors : SQL : Hive 3.1.2 .............. SUCCESS [  7.460 s]
[INFO] Flink : Connectors : SQL : Kafka ................... SUCCESS [  0.957 s]
[INFO] Flink : Connectors : SQL : Kinesis ................. SUCCESS [  6.103 s]
[INFO] Flink : Formats : Avro confluent registry .......... SUCCESS [  2.766 s]
[INFO] Flink : Formats : Sequence file .................... SUCCESS [  0.298 s]
[INFO] Flink : Formats : Compress ......................... SUCCESS [  0.356 s]
[INFO] Flink : Formats : SQL Orc .......................... SUCCESS [  0.581 s]
[INFO] Flink : Formats : SQL Parquet ...................... SUCCESS [  0.633 s]
[INFO] Flink : Formats : SQL Avro ......................... SUCCESS [  1.828 s]
[INFO] Flink : Formats : SQL Avro Confluent Registry ...... SUCCESS [  3.294 s]
[INFO] Flink : Examples : Streaming ....................... SUCCESS [ 10.463 s]
[INFO] Flink : Examples : Table ........................... SUCCESS [  6.642 s]
[INFO] Flink : Examples : Build Helper : .................. SUCCESS [  0.119 s]
[INFO] Flink : Examples : Build Helper : Streaming Twitter  SUCCESS [  0.670 s]
[INFO] Flink : Examples : Build Helper : Streaming State machine SUCCESS [  0.526 s]
[INFO] Flink : Examples : Build Helper : Streaming Google PubSub SUCCESS [  4.241 s]
[INFO] Flink : Container .................................. SUCCESS [  0.315 s]
[INFO] Flink : Queryable state : Runtime .................. SUCCESS [  1.000 s]
[INFO] Flink : Mesos ...................................... SUCCESS [ 19.073 s]
[INFO] Flink : Kubernetes ................................. SUCCESS [  6.304 s]
[INFO] Flink : Yarn ....................................... SUCCESS [  1.964 s]
[INFO] Flink : Libraries : Gelly .......................... SUCCESS [  4.599 s]
[INFO] Flink : Libraries : Gelly scala .................... SUCCESS [ 12.661 s]
[INFO] Flink : Libraries : Gelly Examples ................. SUCCESS [  7.576 s]
[INFO] Flink : External resources : ....................... SUCCESS [  0.052 s]
[INFO] Flink : External resources : GPU ................... SUCCESS [  0.245 s]
[INFO] Flink : Metrics : Dropwizard ....................... SUCCESS [  0.333 s]
[INFO] Flink : Metrics : Graphite ......................... SUCCESS [  0.264 s]
[INFO] Flink : Metrics : InfluxDB ......................... SUCCESS [  1.697 s]
[INFO] Flink : Metrics : Prometheus ....................... SUCCESS [  1.057 s]
[INFO] Flink : Metrics : StatsD ........................... SUCCESS [  0.222 s]
[INFO] Flink : Metrics : Datadog .......................... SUCCESS [  0.332 s]
[INFO] Flink : Metrics : Slf4j ............................ SUCCESS [  0.183 s]
[INFO] Flink : Libraries : CEP Scala ...................... SUCCESS [  7.980 s]
[INFO] Flink : Table : Uber ............................... SUCCESS [  6.009 s]
[INFO] Flink : Table : Uber Blink ......................... SUCCESS [  6.234 s]
[INFO] Flink : Python ..................................... SUCCESS [ 17.610 s]
[INFO] Flink : Table : SQL Client ......................... SUCCESS [  3.707 s]
[INFO] Flink : Libraries : State processor API ............ SUCCESS [  1.865 s]
[INFO] Flink : ML : ....................................... SUCCESS [  0.074 s]
[INFO] Flink : ML : API ................................... SUCCESS [  0.388 s]
[INFO] Flink : ML : Lib ................................... SUCCESS [  1.066 s]
[INFO] Flink : ML : Uber .................................. SUCCESS [  0.429 s]
[INFO] Flink : Scala shell ................................ SUCCESS [  8.262 s]
[INFO] Flink : Dist ....................................... SUCCESS [ 41.965 s]
[INFO] Flink : Yarn Tests ................................. SUCCESS [  6.501 s]
[INFO] Flink : E2E Tests : ................................ SUCCESS [  4.415 s]
[INFO] Flink : E2E Tests : CLI ............................ SUCCESS [  0.327 s]
[INFO] Flink : E2E Tests : Parent Child classloading program SUCCESS [  0.298 s]
[INFO] Flink : E2E Tests : Parent Child classloading lib-package SUCCESS [  0.256 s]
[INFO] Flink : E2E Tests : Dataset allround ............... SUCCESS [  0.220 s]
[INFO] Flink : E2E Tests : Dataset Fine-grained recovery .. SUCCESS [  0.307 s]
[INFO] Flink : E2E Tests : Datastream allround ............ SUCCESS [  1.116 s]
[INFO] Flink : E2E Tests : Batch SQL ...................... SUCCESS [  0.258 s]
[INFO] Flink : E2E Tests : Stream SQL ..................... SUCCESS [  0.272 s]
[INFO] Flink : E2E Tests : Distributed cache via blob ..... SUCCESS [  0.161 s]
[INFO] Flink : E2E Tests : High parallelism iterations .... SUCCESS [  7.459 s]
[INFO] Flink : E2E Tests : Stream stateful job upgrade .... SUCCESS [  1.085 s]
[INFO] Flink : E2E Tests : Queryable state ................ SUCCESS [  2.125 s]
[INFO] Flink : E2E Tests : Local recovery and allocation .. SUCCESS [  0.308 s]
[INFO] Flink : E2E Tests : Elasticsearch 5 ................ SUCCESS [  6.400 s]
[INFO] Flink : E2E Tests : Elasticsearch 6 ................ SUCCESS [  4.810 s]
[INFO] Flink : Quickstart : ............................... SUCCESS [  6.150 s]
[INFO] Flink : Quickstart : Java .......................... SUCCESS [  2.474 s]
[INFO] Flink : Quickstart : Scala ......................... SUCCESS [  0.400 s]
[INFO] Flink : E2E Tests : Quickstart ..................... SUCCESS [  0.350 s]
[INFO] Flink : E2E Tests : Confluent schema registry ...... SUCCESS [  2.201 s]
[INFO] Flink : E2E Tests : Stream state TTL ............... SUCCESS [  2.968 s]
[INFO] Flink : E2E Tests : SQL client ..................... SUCCESS [  3.233 s]
[INFO] Flink : E2E Tests : File sink ...................... SUCCESS [  0.905 s]
[INFO] Flink : E2E Tests : State evolution ................ SUCCESS [  0.586 s]
[INFO] Flink : E2E Tests : RocksDB state memory control ... SUCCESS [  0.497 s]
[INFO] Flink : E2E Tests : Common ......................... SUCCESS [  1.826 s]
[INFO] Flink : E2E Tests : Metrics availability ........... SUCCESS [  0.214 s]
[INFO] Flink : E2E Tests : Metrics reporter prometheus .... SUCCESS [  0.226 s]
[INFO] Flink : E2E Tests : Heavy deployment ............... SUCCESS [  6.590 s]
[INFO] Flink : E2E Tests : Connectors : Google PubSub ..... SUCCESS [  5.169 s]
[INFO] Flink : E2E Tests : Streaming Kafka base ........... SUCCESS [  0.359 s]
[INFO] Flink : E2E Tests : Streaming Kafka ................ SUCCESS [  5.392 s]
[INFO] Flink : E2E Tests : Plugins : ...................... SUCCESS [  0.457 s]
[INFO] Flink : E2E Tests : Plugins : Dummy fs ............. SUCCESS [  0.420 s]
[INFO] Flink : E2E Tests : Plugins : Another dummy fs ..... SUCCESS [  0.277 s]
[INFO] Flink : E2E Tests : TPCH ........................... SUCCESS [  1.295 s]
[INFO] Flink : E2E Tests : Streaming Kinesis .............. SUCCESS [ 11.156 s]
[INFO] Flink : E2E Tests : Elasticsearch 7 ................ SUCCESS [  4.509 s]
[INFO] Flink : E2E Tests : Common Kafka ................... SUCCESS [  1.620 s]
[INFO] Flink : E2E Tests : TPCDS .......................... SUCCESS [  4.335 s]
[INFO] Flink : E2E Tests : Netty shuffle memory control ... SUCCESS [  0.262 s]
[INFO] Flink : E2E Tests : Python ......................... SUCCESS [  6.582 s]
[INFO] Flink : E2E Tests : HBase .......................... SUCCESS [  4.213 s]
[INFO] Flink : State backends : Heap spillable ............ SUCCESS [  1.006 s]
[INFO] Flink : Contrib : .................................. SUCCESS [  0.524 s]
[INFO] Flink : Contrib : Connectors : Wikiedits ........... SUCCESS [  0.822 s]
[INFO] Flink : FileSystems : Tests ........................ SUCCESS [  1.202 s]
[INFO] Flink : Docs ....................................... SUCCESS [  2.157 s]
[INFO] Flink : Walkthrough : .............................. SUCCESS [  0.122 s]
[INFO] Flink : Walkthrough : Common ....................... SUCCESS [  1.583 s]
[INFO] Flink : Walkthrough : Datastream Java .............. SUCCESS [  0.134 s]
[INFO] Flink : Walkthrough : Datastream Scala ............. SUCCESS [  0.154 s]
[INFO] Flink : Tools : CI : Java .......................... SUCCESS [  0.694 s]
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESS
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 30:34 min
[INFO] Finished at: 2020-12-22T18:18:39+08:00
[INFO] Final Memory: 738M/2315M
[INFO] ------------------------------------------------------------------------

和之前一样 Flink : Runtime web 包是编译的难点，因为要在线下载一些包，基本都在国外，下载慢或者根本访问不了，这次还算比较顺利，只遇到下面两个问题

nodejs 包下载不下来

Downloading https://nodejs.org/dist/v10.9.0/node-v10.9.0-linux-x64.tar.gz to /home/venn/.m2/repository/com/github/eirslett/node/10.9.0/node-10.9.0-linux-x64.tar.gz

直接访问 URL 下载，放到 mvn 目录中

$ pwd
/home/venn/.m2/repository/com/github/eirslett/node/10.9.0
$ ls
node-10.9.0-linux-x64.tar.gz

npm 执行不动

Running 'npm ci --cache-max=0 --no-save' in /home/venn/git/flink-1.12.0/flink-runtime-web/web-dashboard

直接安装 npm

apt-get install -y npm

然后重新编译，就通过了，bin 包在路径 flink-1.12.0/flink-dist/target/flink-1.12.0-bin  下面

启动个yarn 上跑的 sql demo 程序

欢迎关注Flink菜鸟公众号，会不定期更新Flink（开发技术）相关的推文