环境
flink 1.13.2 scala 2.12 jdk8 python 3.8 hadoop 3.3.1
linux centos 7 win7
linux hc2108 ,flink 单机 , hc2107客户端
linux hc2151-hc2156 flink集群 , hc2150客户端
问题
1 org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. 20220514
Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded. For a full list of supported file systems, please see https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/. at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:530) at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407) at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274) at org.apache.flink.api.common.io.FileOutputFormat.initializeGlobal(FileOutputFormat.java:288) at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.initializeOnMaster(InputOutputFormatVertex.java:110) at org.apache.flink.runtime.executiongraph.DefaultExecutionGraphBuilder.buildGraph(DefaultExecutionGraphBuilder.java:174) ... 19 more Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies. at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:55) at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:526)
描述
hc2108 单机 flink 运行
flink run --jobmanager hc2108:8081 $FLINK_HOME/examples/batch/WordCount.jar -input hdfs://hc2108:9000/data/T/2021/wordcount -output hdfs://hc2108:9000/out/T/2022/flink/t0514_run1
解决
在环境变量中添加:
export HADOOP_CLASSPATH=`hadoop classpath`