• FlinkSQL/TableAPI 添加配置


    在使用FlinkSQL是向sink表中某个非空字段中插入了null值,收到异常信息

    org.apache.flink.table.api.TableException: Column 'sub_id' is NOT NULL, however, a null value is being written into it. You can set job configuration 'table.exec.sink.not-null-enforcer'='drop' to suppress this exception and drop such records silently.
        at org.apache.flink.table.runtime.operators.sink.SinkNotNullEnforcer.filter(SinkNotNullEnforcer.java:58)
        at org.apache.flink.table.runtime.operators.sink.SinkOperator.processElement(SinkOperator.java:71)
        at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71)
        at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46)
        at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26)
        at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
        at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
        at StreamExecCalc$692.processElement(Unknown Source)
        at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.pushToOperator(CopyingChainingOutput.java:71)
        at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:46)
        at org.apache.flink.streaming.runtime.tasks.CopyingChainingOutput.collect(CopyingChainingOutput.java:26)
        at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:52)
        at org.apache.flink.streaming.api.operators.CountingOutput.collect(CountingOutput.java:30)
        at org.apache.flink.streaming.api.operators.StreamSourceContexts$ManualWatermarkContext.processAndCollectWithTimestamp(StreamSourceContexts.java:310)
        at org.apache.flink.streaming.api.operators.StreamSourceContexts$WatermarkContext.collectWithTimestamp(StreamSourceContexts.java:409)
        at org.apache.flink.streaming.connectors.kafka.internals.AbstractFetcher.emitRecordsWithTimestamps(AbstractFetcher.java:352)
        at org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.partitionConsumerRecordsHandler(KafkaFetcher.java:181)
        at org.apache.flink.streaming.connectors.kafka.internals.KafkaFetcher.runFetchLoop(KafkaFetcher.java:137)
        at org.apache.flink.streaming.connectors.kafka.FlinkKafkaConsumerBase.run(FlinkKafkaConsumerBase.java:761)
        at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:100)
        at org.apache.flink.streaming.api.operators.StreamSource.run(StreamSource.java:63)
        at org.apache.flink.streaming.runtime.tasks.SourceStreamTask$LegacySourceFunctionThread.run(SourceStreamTask.java:215)

    解决方式:

     执行后SQL语句能正常运行。类似的参数设置

    SET execution.result-mode=table;
    
    SET execution.result-mode=tableau;
    
    SET table.sql-dialect=hive;

    除了在SQL中使用这种设定方式,在TalbeAPI中可以可以设定一些参数:

    // instantiate table environment
    val tEnv: TableEnvironment = ...
    
    // access flink configuration
    val configuration = tEnv.getConfig().getConfiguration()
    // set low-level key-value options
    configuration.setString("table.exec.mini-batch.enabled", "true")
    configuration.setString("table.exec.mini-batch.allow-latency", "5 s")
    configuration.setString("table.exec.mini-batch.size", "5000")

    参考:

    https://ci.apache.org/projects/flink/flink-docs-release-1.12/zh/dev/table/streaming/query_configuration.html

    https://www.cnblogs.com/qiu-hua/p/14053999.html

  • 相关阅读:
    Thinphp+nginx配置伪静态
    Potyczki Algorythmiczne 2013
    接下来一段时间会对大家进行网络通信的魔鬼训练理解socket
    项目中怎样做技术选型
    面试官问我:你做事仔细吗?
    【编程一生】2021年总结数据可视化技巧
    一个反直觉的sql
    CURD系统怎么做出技术含量惊艳面试官
    深入理解函数式编程
    历史文章分类汇总
  • 原文地址:https://www.cnblogs.com/yoyowin/p/14899227.html
Copyright © 2020-2023  润新知