Flink安装排坑指南
Flink依赖与java环境,请先安装jdk,最好1.8以上,配置好JAVA_HOME
环境变量
首先我是在window上安装flink的,下载flink
下载之后解压到任意目录,进入bin目录运行start-cluster.bat
脚本启动flink服务,但是Task Slots却为0,查看log发现是TaskManager没有初始化成功
2020-03-14 18:28:37,888 ERROR org.apache.flink.runtime.taskexecutor.TaskManagerRunner - TaskManager initialization failed.
org.apache.flink.configuration.IllegalConfigurationException: Failed to create TaskExecutorResourceSpec
at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:72)
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.startTaskManager(TaskManagerRunner.java:356)
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.<init>(TaskManagerRunner.java:152)
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManager(TaskManagerRunner.java:308)
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.lambda$runTaskManagerSecurely$2(TaskManagerRunner.java:322)
at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.runTaskManagerSecurely(TaskManagerRunner.java:321)
at org.apache.flink.runtime.taskexecutor.TaskManagerRunner.main(TaskManagerRunner.java:287)
Caused by: org.apache.flink.configuration.IllegalConfigurationException: The required configuration option Key: 'taskmanager.cpu.cores' , default: null (fallback keys: []) is not set
at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.checkConfigOptionIsSet(TaskExecutorResourceUtils.java:90)
at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.lambda$checkTaskExecutorResourceConfigSet$0(TaskExecutorResourceUtils.java:84)
at java.util.Arrays$ArrayList.forEach(Unknown Source)
at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.checkTaskExecutorResourceConfigSet(TaskExecutorResourceUtils.java:84)
at org.apache.flink.runtime.taskexecutor.TaskExecutorResourceUtils.resourceSpecFromConfig(TaskExecutorResourceUtils.java:70)
... 7 more
2020-03-14 18:28:37,892 INFO org.apache.flink.runtime.blob.TransientBlobCache - Shutting down BLOB cache
2020-03-14 18:28:37,892 INFO org.apache.flink.runtime.blob.PermanentBlobCache - Shutting down BLOB cache
以为是没有权限创建文件夹,用管理员权限试了也报一样的错
搞了半天也没个结果,最后果断放弃了window环境。
在vmware上弄了个centos7,在centos7上搭建了jdk和flink环境,推荐centos7最少给2G内存,不然后面启动flink时会被内存不够的问题折腾死。
flink最少需要192M的内存才能启动,jdk1.8之后堆内存采用的是元空间,元空间是根据内存大小自动分配的。如果元空间给的太小,flink将会启动不起来。
The derived from fraction jvm overhead memory (19.200mb (20132659 bytes)) is less than its min value 192.000mb (201326592 bytes), min value will be used instead
在启动flink时可以通过jvm参数指定元空间大小
-XX:MetaspaceSize
,初始空间大小,达到该值就会触发垃圾收集进行类型卸载,同时GC会对该值进行调整:如果释放了大量的空间,就适当降低该值;如果释放了很少的空间,那么在不超过MaxMetaspaceSize时,适当提高该值。
flink的cofig/flink-conf.yaml
配置的jobmanager.heap.size
和taskmanager.memory.process.size
也不能超过元空间大小。它们默认大小分别是:1024m和1568m
配置方法:env.java.opts: -XX:MetaspaceSize=512m
,指定元空间为512m
如果电脑内存够大的话,推荐还是直接给centos弄个2G吧,能省很多事。
运行./bin/start-cluster.sh
,在linux上顺利启动,在win10上通过浏览器访问
win10上启动不了不知道是不是因为权限太严格导致的,希望有路过的大佬能在评论区指点一二