• b0115 flink 1.13.2 使用遇到问题


    环境

    flink 1.13.2   scala 2.12   jdk8   python 3.8   hadoop 3.3.1

    linux centos 7    win7 

    linux hc2108 ,flink 单机  , hc2107客户端

    linux hc2151-hc2156 flink集群 ,  hc2150客户端

    问题

    1 org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. 20220514

    Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Could not find a file system implementation for scheme 'hdfs'. The scheme is not directly supported by Flink and no Hadoop file system to support this scheme could be loaded. For a full list of supported file systems, please see https://ci.apache.org/projects/flink/flink-docs-stable/ops/filesystems/.
        at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:530)
        at org.apache.flink.core.fs.FileSystem.get(FileSystem.java:407)
        at org.apache.flink.core.fs.Path.getFileSystem(Path.java:274)
        at org.apache.flink.api.common.io.FileOutputFormat.initializeGlobal(FileOutputFormat.java:288)
        at org.apache.flink.runtime.jobgraph.InputOutputFormatVertex.initializeOnMaster(InputOutputFormatVertex.java:110)
        at org.apache.flink.runtime.executiongraph.DefaultExecutionGraphBuilder.buildGraph(DefaultExecutionGraphBuilder.java:174)
        ... 19 more
    Caused by: org.apache.flink.core.fs.UnsupportedFileSystemSchemeException: Hadoop is not in the classpath/dependencies.
        at org.apache.flink.core.fs.UnsupportedSchemeFactory.create(UnsupportedSchemeFactory.java:55)
        at org.apache.flink.core.fs.FileSystem.getUnguardedFileSystem(FileSystem.java:526)

    描述

    hc2108 单机  flink 运行 

    flink run --jobmanager hc2108:8081  $FLINK_HOME/examples/batch/WordCount.jar -input hdfs://hc2108:9000/data/T/2021/wordcount -output hdfs://hc2108:9000/out/T/2022/flink/t0514_run1

    解决

    在环境变量中添加:

    export HADOOP_CLASSPATH=`hadoop classpath`
    

    在这里插入图片描述

    参考

    https://blog.csdn.net/qq_43224174/article/details/122311901

  • 相关阅读:
    阅读笔记2
    阅读笔记1
    学习进度条
    第二阶段冲刺第七天站立会议
    第二阶段冲刺第六天站立会议
    第二阶段冲刺第五天站立会议
    阅读笔记4
    第二阶段冲刺第四天站立会议
    第一阶段意见评论
    第12周学习进度总结
  • 原文地址:https://www.cnblogs.com/sunzebo/p/16269488.html
Copyright © 2020-2023  润新知