Apache Hudi 源码分析 HoodieTableSink

可以看出，分为几种，

bulk_insert

append

default, upsert

compact，clean

Pipelines这部分，注释写的非常不错

bulk_insert

先按照partition path进行shuffle分区，然后再按照partition path进行排序，一个partition path的record都聚合在一起

这样再按照每个partition path去写文件，有效降低了小文件

  /**
   * Bulk insert the input dataset at once.
   *
   * <p>By default, the input dataset would shuffle by the partition path first then
   * sort by the partition path before passing around to the write function.
   * The whole pipeline looks like the following:
   *
   * <pre>
   *      | input1 | ===\     /=== |sorter| === | task1 | (p1, p2)
   *                   shuffle
   *      | input2 | ===/     \=== |sorter| === | task2 | (p3, p4)
   *
   *      Note: Both input1 and input2's dataset come from partitions: p1, p2, p3, p4
   * </pre>
   *
   * <p>The write task switches to new file handle each time it receives a record
   * from the different partition path, the shuffle and sort would reduce small files.
   *
   * <p>The bulk insert should be run in batch execution mode.
   *
   * @param conf       The configuration
   * @param rowType    The input row type
   * @param dataStream The input data stream
   * @return the bulk insert data stream sink
   */

可以看到代码逻辑上也是分成3块，

    final String[] partitionFields = FilePathUtils.extractPartitionKeys(conf);
    if (partitionFields.length > 0) {
      RowDataKeyGen rowDataKeyGen = RowDataKeyGen.instance(conf, rowType);
      if (conf.getBoolean(FlinkOptions.WRITE_BULK_INSERT_SHUFFLE_INPUT)) {

        // shuffle by partition keys，分区
        // use #partitionCustom instead of #keyBy to avoid duplicate sort operations,
        // see BatchExecutionUtils#applyBatchExecutionSettings for details.
        Partitioner<String> partitioner = (key, channels) ->
            KeyGroupRangeAssignment.assignKeyToParallelOperator(key, StreamGraphGenerator.DEFAULT_LOWER_BOUND_MAX_PARALLELISM, channels);
        dataStream = dataStream.partitionCustom(partitioner, rowDataKeyGen::getPartitionPath);
      }
      if (conf.getBoolean(FlinkOptions.WRITE_BULK_INSERT_SORT_INPUT)) {
        SortOperatorGen sortOperatorGen = new SortOperatorGen(rowType, partitionFields);
        // sort by partition keys，排序
        dataStream = dataStream
            .transform("partition_key_sorter",
                TypeInformation.of(RowData.class),
                sortOperatorGen.createSortOperator())
            .setParallelism(conf.getInteger(FlinkOptions.WRITE_TASKS));
        ExecNodeUtil.setManagedMemoryWeight(dataStream.getTransformation(),
            conf.getInteger(FlinkOptions.WRITE_SORT_MEMORY) * 1024L * 1024L);
      }
    }
    return dataStream
        .transform("hoodie_bulk_insert_write",
            TypeInformation.of(Object.class),
            operatorFactory) //写入operator
        // follow the parallelism of upstream operators to avoid shuffle
        .setParallelism(conf.getInteger(FlinkOptions.WRITE_TASKS))
        .addSink(DummySink.INSTANCE) //最后加上DummySink
        .name("dummy");

这里operatorFactory最终生成的是

BulkInsertWriteFunction

  @Override
  public void processElement(I value, Context ctx, Collector<Object> out) throws IOException {
    this.writerHelper.write((RowData) value);
  }

BucketBulkInsertWriterHelper

如果record是连续的，可以直接重用lastFileId对应的file handle，比较高效，所以前面需要排序

  public void write(RowData tuple) throws IOException {
    try {
      RowData record = tuple.getRow(1, this.recordArity);
      String recordKey = keyGen.getRecordKey(record);
      String partitionPath = keyGen.getPartitionPath(record);
      String fileId = tuple.getString(0).toString();
      if ((lastFileId == null) || !lastFileId.equals(fileId)) {
        LOG.info("Creating new file for partition path " + partitionPath);
        handle = getRowCreateHandle(partitionPath, fileId);
        lastFileId = fileId;
      }
      handle.write(recordKey, partitionPath, record); //单行写入parquet文件
    } catch (Throwable throwable) {
      LOG.error("Global error thrown while trying to write records in HoodieRowDataCreateHandle", throwable);
      throw throwable;
    }
  }

否则需要调用，getRowCreateHandle

  private HoodieRowDataCreateHandle getRowCreateHandle(String partitionPath) throws IOException {
    if (!handles.containsKey(partitionPath)) { // handles里面没有现成的
      // if records are sorted, we can close all existing handles
      if (isInputSorted) { //
        close(); //如果Sorted，没有必要cache之前的handle，所以可以close掉
      }
      HoodieRowDataCreateHandle rowCreateHandle = new HoodieRowDataCreateHandle(hoodieTable, writeConfig, partitionPath, getNextFileId(),
          instantTime, taskPartitionId, taskId, taskEpochId, rowType);
      handles.put(partitionPath, rowCreateHandle); //创建一个新的加入到handles里面
    } else if (!handles.get(partitionPath).canWrite()) { //虽然已经有handle，但是已经写满
      // even if there is a handle to the partition path, it could have reached its max size threshold. So, we close the handle here and
      // create a new one.
      writeStatusList.add(handles.remove(partitionPath).close()); //所以把当前的handle关掉，并放入writeStatus中
      HoodieRowDataCreateHandle rowCreateHandle = new HoodieRowDataCreateHandle(hoodieTable, writeConfig, partitionPath, getNextFileId(),
          instantTime, taskPartitionId, taskId, taskEpochId, rowType);
      handles.put(partitionPath, rowCreateHandle); //创建新的handle加入
    }
    return handles.get(partitionPath); //从handles取到handle
  }

可以看到这里写文件也是单条写入的，

之所以叫BulkInsert，

因为BulkInsertWriteOperator实现BoundedOneInput

public class BulkInsertWriteOperator<I>
    extends AbstractWriteOperator<I>
    implements BoundedOneInput {

BoundedOneInput中主要是endInput函数，

  /**
   * End input action for batch source.
   */
  public void endInput() {
    final List<WriteStatus> writeStatus = this.writerHelper.getWriteStatuses(this.taskID);

    final WriteMetadataEvent event = WriteMetadataEvent.builder()
        .taskID(taskID)
        .instantTime(this.writerHelper.getInstantTime())
        .writeStatus(writeStatus)
        .lastBatch(true)
        .endInput(true)
        .build();
    this.eventGateway.sendEventToCoordinator(event);
  }

可以看到，这里会给coordinator发生event来更新meta

所以是bulkinsert，因为当所有数据写完后，需要手动调用一次endInput来更新元数据

append

不考虑update和去重

所以流程很简单直接写

这里画了shuffle，但是没有特意去partition，如果并行度相同应该没有shuffle的过程

这种方式会产生大量的小文件，没有按partition分区，也没有排序

主要是没有分区，没有排序不会有太大的影响文件数目，因为只有在snapshot的时候才会产生新文件

  /**
   * Insert the dataset with append mode(no upsert or deduplication).
   *
   * <p>The input dataset would be rebalanced among the write tasks:
   *
   * <pre>
   *      | input1 | ===\     /=== | task1 | (p1, p2, p3, p4)
   *                   shuffle
   *      | input2 | ===/     \=== | task2 | (p1, p2, p3, p4)
   *
   *      Note: Both input1 and input2's dataset come from partitions: p1, p2, p3, p4
   * </pre>
   *
   * <p>The write task switches to new file handle each time it receives a record
   * from the different partition path, so there may be many small files.
   *
   * @param conf       The configuration
   * @param rowType    The input row type
   * @param dataStream The input data stream
   * @param bounded    Whether the input stream is bounded
   * @return the appending data stream sink
   */

  public static DataStreamSink<Object> append(
      Configuration conf,
      RowType rowType,
      DataStream<RowData> dataStream,
      boolean bounded) {
    WriteOperatorFactory<RowData> operatorFactory = AppendWriteOperator.getFactory(conf, rowType); //

    return dataStream
        .transform("hoodie_append_write", TypeInformation.of(Object.class), operatorFactory) //直接Append Operator写入
        .uid("uid_hoodie_stream_write" + conf.getString(FlinkOptions.TABLE_NAME))
        .setParallelism(conf.getInteger(FlinkOptions.WRITE_TASKS))
        .addSink(DummySink.INSTANCE)
        .name("dummy");
  }

AppendWriteFunction

首先对于processElement的逻辑和BulkInsert一样，没有区别

不一样的，Append除了支持batch，即boundedOneInput接口，在endInput中flushData

也支持streaming的情况，在snapshotState中，flushData

  @Override
  public void snapshotState() {
    // Based on the fact that the coordinator starts the checkpoint first,
    // it would check the validity.
    // wait for the buffer data flush out and request a new instant
    flushData(false);
  }

  @Override
  public void processElement(I value, Context ctx, Collector<Object> out) throws Exception {
    if (this.writerHelper == null) {
      initWriterHelper();
    }
    this.writerHelper.write((RowData) value);
  }

  /**
   * End input action for batch source.
   */
  public void endInput() {
    flushData(true);
    this.writeStatuses.clear();
  }

这个flush的逻辑和BulkInsert中的基本一致

  private void flushData(boolean endInput) {
    final List<WriteStatus> writeStatus;
    final String instant;
    if (this.writerHelper != null) { //
      writeStatus = this.writerHelper.getWriteStatuses(this.taskID); //
      instant = this.writerHelper.getInstantTime(); //
    } else {
      writeStatus = Collections.emptyList();
      instant = instantToWrite(false);
      LOG.info("No data to write in subtask [{}] for instant [{}]", taskID, instant);
    }
    final WriteMetadataEvent event = WriteMetadataEvent.builder()
        .taskID(taskID)
        .instantTime(instant)
        .writeStatus(writeStatus)
        .lastBatch(true)
        .endInput(endInput)
        .build(); //
    this.eventGateway.sendEventToCoordinator(event); //
    // nullify the write helper for next ckp
    this.writerHelper = null;
    this.writeStatuses.addAll(writeStatus);
    // blocks flushing until the coordinator starts a new instant
    this.confirming = true;
  }

需要注意的是，在getWriteStatuses的时候，会close file handle

所以一次snapshot会产生一批新的文件

  public List<HoodieInternalWriteStatus> getHoodieWriteStatuses() throws IOException {
    close();
    return writeStatusList;
  }

这里BulkInsert和Append两种模式的设计不太清晰，大部分逻辑雷同

BulkInsert，有按partitionPath分区和排序，但是只支持batch

Append，直接写，但是支持streaming和batch

为啥不能合成一个？

bootstrap

有两种，boundedBootstrap和streamBootstrap

  /**
   * Constructs bootstrap pipeline.
   * The bootstrap operator loads the existing data index (primary key to file id mapping),
   * then send the indexing data set to subsequent operator(usually the bucket assign operator).
   *
   * @param conf               The configuration
   * @param rowType            The row type
   * @param defaultParallelism The default parallelism
   * @param dataStream         The data stream
   * @param bounded            Whether the source is bounded
   * @param overwrite          Whether it is insert overwrite
   */
  public static DataStream<HoodieRecord> bootstrap(
      Configuration conf,
      RowType rowType,
      int defaultParallelism,
      DataStream<RowData> dataStream,
      boolean bounded,
      boolean overwrite) {
    final boolean globalIndex = conf.getBoolean(FlinkOptions.INDEX_GLOBAL_ENABLED);
    if (overwrite || OptionsResolver.isBucketIndexType(conf)) {
      return rowDataToHoodieRecord(conf, rowType, dataStream); //只是转换成HoodieRecord
    } else if (bounded && !globalIndex && OptionsResolver.isPartitionedTable(conf)) {
      return boundedBootstrap(conf, rowType, defaultParallelism, dataStream); //
    } else {
      return streamBootstrap(conf, rowType, defaultParallelism, dataStream, bounded); //
    }
  }

streamBootstrap

  private static DataStream<HoodieRecord> streamBootstrap(
      Configuration conf,
      RowType rowType,
      int defaultParallelism,
      DataStream<RowData> dataStream,
      boolean bounded) {
    DataStream<HoodieRecord> dataStream1 = rowDataToHoodieRecord(conf, rowType, dataStream); //转化成HoodieRecord

    if (conf.getBoolean(FlinkOptions.INDEX_BOOTSTRAP_ENABLED) || bounded) {
      dataStream1 = dataStream1
          .transform(
              "index_bootstrap",
              TypeInformation.of(HoodieRecord.class),
              new BootstrapOperator<>(conf)) //使用BootstrapOperator
          .setParallelism(conf.getOptional(FlinkOptions.INDEX_BOOTSTRAP_TASKS).orElse(defaultParallelism))
          .uid("uid_index_bootstrap_" + conf.getString(FlinkOptions.TABLE_NAME));
    }

    return dataStream1;
  }

BootstrapOperator

主要逻辑在initializeState里面，processElement只是forword record

state中，要保存lastPendingInstant，最新的没有完成的instant

主要的逻辑是preLoadIndexRecords

  @Override
  public void snapshotState(StateSnapshotContext context) throws Exception {
    lastInstantTime = this.ckpMetadata.lastPendingInstant(); //
    instantState.update(Collections.singletonList(lastInstantTime));
  }

  @Override
  public void initializeState(StateInitializationContext context) throws Exception {
    ListStateDescriptor<String> instantStateDescriptor = new ListStateDescriptor<>(
        "instantStateDescriptor",
        Types.STRING
    );
    instantState = context.getOperatorStateStore().getListState(instantStateDescriptor);

    if (context.isRestored()) {
      Iterator<String> instantIterator = instantState.get().iterator(); //
      if (instantIterator.hasNext()) {
        lastInstantTime = instantIterator.next(); //
      }
    }

    this.hadoopConf = StreamerUtil.getHadoopConf();
    this.writeConfig = StreamerUtil.getHoodieClientConfig(this.conf, true);
    this.hoodieTable = FlinkTables.createTable(writeConfig, hadoopConf, getRuntimeContext());
    this.ckpMetadata = CkpMetadata.getInstance(hoodieTable.getMetaClient().getFs(), this.writeConfig.getBasePath());
    this.aggregateManager = getRuntimeContext().getGlobalAggregateManager();

    preLoadIndexRecords(); //
  }

  @Override
  @SuppressWarnings("unchecked")
  public void processElement(StreamRecord<I> element) throws Exception {
    output.collect((StreamRecord<O>) element);
  }

preLoadIndexRecords

对于这个table下的所有partition，满足Pattern，即需要加载index的，调用loadRecords

然后等待所有并发的task都完成loadrecords，这步不完成init就不结束，job不会正式开始

  /**
   * Load the index records before {@link #processElement}.
   */
  protected void preLoadIndexRecords() throws Exception {
    String basePath = hoodieTable.getMetaClient().getBasePath();
    int taskID = getRuntimeContext().getIndexOfThisSubtask();
    LOG.info("Start loading records in table {} into the index state, taskId = {}", basePath, taskID);
    for (String partitionPath : FSUtils.getAllFoldersWithPartitionMetaFile(FSUtils.getFs(basePath, hadoopConf), basePath)) {
      if (pattern.matcher(partitionPath).matches()) {
        loadRecords(partitionPath);
      }
    }// wait for the other bootstrap tasks finish bootstrapping.
    waitForBootstrapReady(getRuntimeContext().getIndexOfThisSubtask());
  }

  /**
   * Wait for other bootstrap tasks to finish the index bootstrap.
   */
  private void waitForBootstrapReady(int taskID) {
    int taskNum = getRuntimeContext().getNumberOfParallelSubtasks();
    int readyTaskNum = 1;
    while (taskNum != readyTaskNum) {
      try {
        readyTaskNum = aggregateManager.updateGlobalAggregate(BootstrapAggFunction.NAME, taskID, new BootstrapAggFunction());
        LOG.info("Waiting for other bootstrap tasks to complete, taskId = {}.", taskID);

        TimeUnit.SECONDS.sleep(5);
      } catch (Exception e) {
        LOG.warn("Update global task bootstrap summary error", e);
      }
    }
  }

loadRecords

  protected void loadRecords(String partitionPath) throws Exception {

    HoodieTimeline commitsTimeline = this.hoodieTable.getMetaClient().getCommitsTimeline(); //
    if (!StringUtils.isNullOrEmpty(lastInstantTime)) {
      commitsTimeline = commitsTimeline.findInstantsAfter(lastInstantTime); //lastInstantTime是state保存，读取after的timeline
    }
    Option<HoodieInstant> latestCommitTime = commitsTimeline.filterCompletedInstants().lastInstant(); //找出timeline中完成instant中最新的一个

    if (latestCommitTime.isPresent()) {
      BaseFileUtils fileUtils = BaseFileUtils.getInstance(this.hoodieTable.getBaseFileFormat());
      Schema schema = new TableSchemaResolver(this.hoodieTable.getMetaClient()).getTableAvroSchema();

      List<FileSlice> fileSlices = this.hoodieTable.getSliceView()
          .getLatestFileSlicesBeforeOrOn(partitionPath, latestCommitTime.get().getTimestamp(), true)
          .collect(toList()); //Before，因为后面都是没有commit的，所以读前面的，读出所有的fileSlices

      for (FileSlice fileSlice : fileSlices) { //
        if (!shouldLoadFile(fileSlice.getFileId(), maxParallelism, parallelism, taskID)) {
          continue;
        }
        LOG.info("Load records from {}.", fileSlice);

        // load parquet records
        fileSlice.getBaseFile().ifPresent(baseFile -> { //先读BaseFile，parquet格式
          // filter out crushed files
          if (!isValidFile(baseFile.getFileStatus())) {
            return;
          }
          try (ClosableIterator<HoodieKey> iterator = fileUtils.getHoodieKeyIterator(this.hadoopConf, new Path(baseFile.getPath()))) {
            iterator.forEachRemaining(hoodieKey -> {
              output.collect(new StreamRecord(new IndexRecord(generateHoodieRecord(hoodieKey, fileSlice))));
            }); //最终输出indexRecord，hoodiekey和fileSlice的匹配，这里会输出这个fileslice中所有的recordKey
          }
        });

        // load avro log records
        List<String> logPaths = fileSlice.getLogFiles() //后读logfile
            // filter out crushed files
            .filter(logFile -> isValidFile(logFile.getFileStatus()))
            .map(logFile -> logFile.getPath().toString())
            .collect(toList());
        HoodieMergedLogRecordScanner scanner = FormatUtils.logScanner(logPaths, schema, latestCommitTime.get().getTimestamp(),
            writeConfig, hadoopConf); //生成对logfile的scanner

        try { //输出和上面一样，也是indexRecord
          for (String recordKey : scanner.getRecords().keySet()) {
            output.collect(new StreamRecord(new IndexRecord(generateHoodieRecord(new HoodieKey(recordKey, partitionPath), fileSlice))));
          }
        } catch (Exception e) {
          throw new HoodieException(String.format("Error when loading record keys from files: %s", logPaths), e);
        } finally {
          scanner.close();
        }
      }
    }

boundedBootstrap

  /**
   * Constructs bootstrap pipeline for batch execution mode.
   * The indexing data set is loaded before the actual data write
   * in order to support batch UPSERT.
   */
  private static DataStream<HoodieRecord> boundedBootstrap(
      Configuration conf,
      RowType rowType,
      int defaultParallelism,
      DataStream<RowData> dataStream) {
    final RowDataKeyGen rowDataKeyGen = RowDataKeyGen.instance(conf, rowType);
    // shuffle by partition keys
    dataStream = dataStream
        .keyBy(rowDataKeyGen::getPartitionPath); //加上key，以partitionPath为key

    return rowDataToHoodieRecord(conf, rowType, dataStream) //转换成HoodieRecord
        .transform(
            "batch_index_bootstrap",
            TypeInformation.of(HoodieRecord.class),
            new BatchBootstrapOperator<>(conf)) //BathBootStrap
        .setParallelism(conf.getOptional(FlinkOptions.INDEX_BOOTSTRAP_TASKS).orElse(defaultParallelism))
        .uid("uid_batch_index_bootstrap_" + conf.getString(FlinkOptions.TABLE_NAME));
  }

BatchBootstrapOperator

和上面不同在于，

这里，preLoadIndexRecords是空的，不做事情

然后在processElement的时候，动态对于record对应的partitionPath进行load

  @Override
  protected void preLoadIndexRecords() {
    // no operation
  }

  @Override
  @SuppressWarnings("unchecked")
  public void processElement(StreamRecord<I> element) throws Exception {
    final HoodieRecord<?> record = (HoodieRecord<?>) element.getValue();
    final String partitionPath = record.getKey().getPartitionPath();

    if (haveSuccessfulCommits && !partitionPathSet.contains(partitionPath)) {
      loadRecords(partitionPath);
      partitionPathSet.add(partitionPath);
    }

    // send the trigger record
    output.collect((StreamRecord<O>) element);
  }

所以最终bootstrap，除了output正常的record流，还会output indexRecord流

hoodieStreamWrite

先按照recordKey shuffle

然后assign bucket

再按照bucket id，即file group，shuffle，保证一个file group只会在一个task中写入，避免冲突

  /**
   * The streaming write pipeline.
   *
   * <p>The input dataset shuffles by the primary key first then
   * shuffles by the file group ID before passing around to the write function.
   * The whole pipeline looks like the following:
   *
   * <pre>
   *      | input1 | ===\     /=== | bucket assigner | ===\     /=== | task1 |
   *                   shuffle(by PK)                    shuffle(by bucket ID)
   *      | input2 | ===/     \=== | bucket assigner | ===/     \=== | task2 |
   *
   *      Note: a file group must be handled by one write task to avoid write conflict.
   * </pre>
   *
   * <p>The bucket assigner assigns the inputs to suitable file groups, the write task caches
   * and flushes the data set to disk.
   *
   * @param conf               The configuration
   * @param defaultParallelism The default parallelism
   * @param dataStream         The input data stream
   * @return the stream write data stream pipeline
   */

代码逻辑比较清晰，

      WriteOperatorFactory<HoodieRecord>
              operatorFactory = StreamWriteOperator.getFactory(conf); //
      return dataStream
          // Key-by record key, to avoid multiple subtasks write to a bucket at the same time
          .keyBy(HoodieRecord::getRecordKey) //按照record key进行shuffle
          .transform(
              "bucket_assigner",
              TypeInformation.of(HoodieRecord.class),
              new KeyedProcessOperator<>(new BucketAssignFunction<>(conf))) //分配bucket
          .uid("uid_bucket_assigner_" + conf.getString(FlinkOptions.TABLE_NAME))
          .setParallelism(conf.getOptional(FlinkOptions.BUCKET_ASSIGN_TASKS).orElse(defaultParallelism))
          // shuffle by fileId(bucket id)
          .keyBy(record -> record.getCurrentLocation().getFileId()) //按照bucketID进行shuffle
          .transform("stream_write", TypeInformation.of(Object.class), operatorFactory) //流式写入
          .uid("uid_stream_write" + conf.getString(FlinkOptions.TABLE_NAME))
          .setParallelism(conf.getInteger(FlinkOptions.WRITE_TASKS));

BucketAssignFunction

有两个流，一个是从bootstrap过来的indexRecord，一个是正常的数据record

所以对于，indexRecord逻辑很简单，利用KeyedStateStore来存每个recordKey对应的location

对于正常的record，

几种情况，

如果当前的PartitionPath和老的不一样，

    - 如果是globalIndex，需要先创建一条deleteRecord，然后再在当前partitionPath，getNewRecordLocation，即分配一个新的file group

    - 如果不是globalIndex，那index是partition内部的，不用管其他的partition，所以直接getNewRecordLocation

如果location一样，那么直接update

  private void processRecord(HoodieRecord<?> record, Collector<O> out) throws Exception {
    // 1. put the record into the BucketAssigner;
    // 2. look up the state for location, if the record has a location, just send it out;
    // 3. if it is an INSERT, decide the location using the BucketAssigner then send it out.
    final HoodieKey hoodieKey = record.getKey();
    final String recordKey = hoodieKey.getRecordKey();
    final String partitionPath = hoodieKey.getPartitionPath();
    final HoodieRecordLocation location;

    // Only changing records need looking up the index for the location,
    // append only records are always recognized as INSERT.
    HoodieRecordGlobalLocation oldLoc = indexState.value();
    if (isChangingRecords && oldLoc != null) {
      // Set up the instant time as "U" to mark the bucket as an update bucket.
      if (!Objects.equals(oldLoc.getPartitionPath(), partitionPath)) {
        if (globalIndex) {
          // if partition path changes, emit a delete record for old partition path,
          // then update the index state using location with new partition path.
          HoodieRecord<?> deleteRecord = new HoodieAvroRecord<>(new HoodieKey(recordKey, oldLoc.getPartitionPath()),
              payloadCreation.createDeletePayload((BaseAvroPayload) record.getData()));
          deleteRecord.setCurrentLocation(oldLoc.toLocal("U"));
          deleteRecord.seal();
          out.collect((O) deleteRecord);
        }
        location = getNewRecordLocation(partitionPath);
      } else {
        location = oldLoc.toLocal("U");
        this.bucketAssigner.addUpdate(partitionPath, location.getFileId());
      }
    } else {
      location = getNewRecordLocation(partitionPath);
    }
    // always refresh the index
    if (isChangingRecords) {
      updateIndexState(partitionPath, location);
    }
    record.setCurrentLocation(location);
    out.collect((O) record);
  }

getNewRecordLocation，其实是调用了bucketAssigner.addInsert

下面就看下，

bucketAssigner.addInsert和bucketAssigner.addUpdate

addInsert

这里注意，newFileAssignState和bucketInfoMap在每次checkpoint的时候会clear掉

因为cp的时候，所有的file会close掉

  public BucketInfo addInsert(String partitionPath) {
    // for new inserts, compute buckets depending on how many records we have for each partition
    SmallFileAssign smallFileAssign = getSmallFileAssign(partitionPath);

    // 先找小文件，如果有可以assign，就update该文件
    if (smallFileAssign != null && smallFileAssign.assign()) {
      return new BucketInfo(BucketType.UPDATE, smallFileAssign.getFileId(), partitionPath);
    }

    // 看下当前partitionPath上，是否有现成可以assign的文件，如果有拿过来直接update
    if (newFileAssignStates.containsKey(partitionPath)) {
      NewFileAssignState newFileAssignState = newFileAssignStates.get(partitionPath);
      if (newFileAssignState.canAssign()) {
        newFileAssignState.assign();
        final String key = StreamerUtil.generateBucketKey(partitionPath, newFileAssignState.fileId);
        if (bucketInfoMap.containsKey(key)) {
          // the newFileAssignStates is cleaned asynchronously when received the checkpoint success notification,
          // the records processed within the time range:
          // (start checkpoint, checkpoint success(and instant committed))
          // should still be assigned to the small buckets of last checkpoint instead of new one.

          // the bucketInfoMap is cleaned when checkpoint starts.

          // A promotion: when the HoodieRecord can record whether it is an UPDATE or INSERT,
          // we can always return an UPDATE BucketInfo here, and there is no need to record the
          // UPDATE bucket through calling #addUpdate.
          return bucketInfoMap.get(key);
        }
        return new BucketInfo(BucketType.UPDATE, newFileAssignState.fileId, partitionPath);
      }
    }
    //如果没有，就要新创建bucket
    BucketInfo bucketInfo = new BucketInfo(BucketType.INSERT, createFileIdOfThisTask(), partitionPath);
    final String key = StreamerUtil.generateBucketKey(partitionPath, bucketInfo.getFileIdPrefix());
    bucketInfoMap.put(key, bucketInfo);
    NewFileAssignState newFileAssignState = new NewFileAssignState(bucketInfo.getFileIdPrefix(), writeProfile.getRecordsPerBucket()); //
    newFileAssignState.assign();
    newFileAssignStates.put(partitionPath, newFileAssignState);
    return bucketInfo;
  }

addUpdate

这个逻辑就很简单了，对于update已经知道fileid，就直接生成bucketInfo就可以

  public BucketInfo addUpdate(String partitionPath, String fileIdHint) {
    final String key = StreamerUtil.generateBucketKey(partitionPath, fileIdHint);
    if (!bucketInfoMap.containsKey(key)) {
      BucketInfo bucketInfo = new BucketInfo(BucketType.UPDATE, fileIdHint, partitionPath);
      bucketInfoMap.put(key, bucketInfo);
    }
    // else do nothing because the bucket already exists.
    return bucketInfoMap.get(key);
  }

StreamWriteFunction

可以看到，这里核心逻辑非常简单

processElement只是buffer

当checkpoint的时候去flush所有的buckets

  @Override
  public void open(Configuration parameters) throws IOException {
    this.tracer = new TotalSizeTracer(this.config);
    initBuffer();
    initWriteFunction();
  }

  @Override
  public void snapshotState() {
    // Based on the fact that the coordinator starts the checkpoint first,
    // it would check the validity.
    // wait for the buffer data flush out and request a new instant
    flushRemaining(false);
  }

  @Override
  public void processElement(I value, ProcessFunction<I, Object>.Context ctx, Collector<Object> out) throws Exception {
    bufferRecord((HoodieRecord<?>) value);
  }

bufferRecord

把element里面的data加到对应的bucket的buffer中去，

然后会额外做两个判断，flushBucket和flushBuffer

如果bucket过大，就flush掉该bucket；如果buffer太大，从里面找到最大的bucket去flush

  /**
   * Buffers the given record.
   *
   * <p>Flush the data bucket first if the bucket records size is greater than
   * the configured value {@link FlinkOptions#WRITE_BATCH_SIZE}.
   *
   * <p>Flush the max size data bucket if the total buffer size exceeds the configured
   * threshold {@link FlinkOptions#WRITE_TASK_MAX_SIZE}.
   *
   * @param value HoodieRecord
   */
  protected void bufferRecord(HoodieRecord<?> value) {
    final String bucketID = getBucketID(value); //

    DataBucket bucket = this.buckets.computeIfAbsent(bucketID,
        k -> new DataBucket(this.config.getDouble(FlinkOptions.WRITE_BATCH_SIZE), value)); //
    final DataItem item = DataItem.fromHoodieRecord(value); //

    bucket.records.add(item); //

    boolean flushBucket = bucket.detector.detect(item); //
    boolean flushBuffer = this.tracer.trace(bucket.detector.lastRecordSize); //
    if (flushBucket) {
      if (flushBucket(bucket)) {
        this.tracer.countDown(bucket.detector.totalSize);
        bucket.reset();
      }
    } else if (flushBuffer) {
      // find the max size bucket and flush it out
      List<DataBucket> sortedBuckets = this.buckets.values().stream()
          .sorted((b1, b2) -> Long.compare(b2.detector.totalSize, b1.detector.totalSize))
          .collect(Collectors.toList());
      final DataBucket bucketToFlush = sortedBuckets.get(0);
      if (flushBucket(bucketToFlush)) {
        this.tracer.countDown(bucketToFlush.detector.totalSize);
        bucketToFlush.reset();
      } else {
        LOG.warn("The buffer size hits the threshold {}, but still flush the max size data bucket failed!", this.tracer.maxBufferSize);
      }
    }
  }

flushRemaining

  private void flushRemaining(boolean endInput) {
    this.currentInstant = instantToWrite(hasData()); //获取instant

    final List<WriteStatus> writeStatus;
    if (buckets.size() > 0) {
      writeStatus = new ArrayList<>();
      this.buckets.values()
          // The records are partitioned by the bucket ID and each batch sent to
          // the writer belongs to one bucket.
          .forEach(bucket -> {
            List<HoodieRecord> records = bucket.writeBuffer();
            if (records.size() > 0) {
              if (config.getBoolean(FlinkOptions.PRE_COMBINE)) {
                records = FlinkWriteHelper.newInstance().deduplicateRecords(records, (HoodieIndex) null, -1); //去重，在bucket所缓存的records中
              }
              bucket.preWrite(records); //
              writeStatus.addAll(writeFunction.apply(records, currentInstant)); //利用writeFunction进行真实写入
              records.clear();
              bucket.reset();
            }
          });
    } else {
      LOG.info("No data to write in subtask [{}] for instant [{}]", taskID, currentInstant);
      writeStatus = Collections.emptyList();
    }
    final WriteMetadataEvent event = WriteMetadataEvent.builder()
        .taskID(taskID)
        .instantTime(currentInstant)
        .writeStatus(writeStatus)
        .lastBatch(true)
        .endInput(endInput)
        .build(); //将writeStatus发给coordinator，更新meta

    this.eventGateway.sendEventToCoordinator(event);
    this.buckets.clear();
    this.tracer.reset();
    this.writeClient.cleanHandles();
    this.writeStatuses.addAll(writeStatus);
    // blocks flushing until the coordinator starts a new instant
    this.confirming = true;
  }

这里的writeFunction，

我们可以挑个看下，

HoodieFlinkWriteClient.upsert

HoodieFlinkCopyOnWriteTable.upsert

FlinkUpsertCommitActionExecutor.execute

FlinkWriteHelper.write

BaseFlinkCommitActionExecutor.execute

 protected Iterator<List<WriteStatus>> handleUpdateInternal(HoodieMergeHandle<?, ?, ?, ?> upsertHandle, String fileId)
      throws IOException {
    if (upsertHandle.getOldFilePath() == null) {
      throw new HoodieUpsertException(
          "Error in finding the old file path at commit " + instantTime + " for fileId: " + fileId);
    } else {
      FlinkMergeHelper.newInstance().runMerge(table, upsertHandle); //最终调用
    }
    return Collections.singletonList(upsertHandle.writeStatuses()).iterator();
  }

runMerge

producer，readerIterator读取record

consumer，调用mergeHandle

      ThreadLocal<BinaryEncoder> encoderCache = new ThreadLocal<>();
      ThreadLocal<BinaryDecoder> decoderCache = new ThreadLocal<>();
      //producer-consumer模式
      wrapper = new BoundedInMemoryExecutor<>(table.getConfig().getWriteBufferLimitBytes(), new IteratorBasedQueueProducer<>(readerIterator),
          Option.of(new UpdateHandler(mergeHandle)), record -> {
        if (!externalSchemaTransformation) {
          return record;
        }
        return transformRecordBasedOnNewSchema(gReader, gWriter, encoderCache, decoderCache, (GenericRecord) record);
      });
      wrapper.execute();

mergeHandle

HoodieFlinkWriteClient.upsert中生成，

 getOrCreateWriteHandle

    final boolean isDelta = table.getMetaClient().getTableType().equals(HoodieTableType.MERGE_ON_READ);
    final HoodieWriteHandle<?, ?, ?, ?> writeHandle;
    if (isDelta) { //merge on Read
      writeHandle = new FlinkAppendHandle<>(config, instantTime, table, partitionPath, fileID, recordItr,
          table.getTaskContextSupplier());
    } else if (loc.getInstantTime().equals("I")) { //insert
      writeHandle = new FlinkCreateHandle<>(config, instantTime, table, partitionPath,
          fileID, table.getTaskContextSupplier());
    } else { //update
      writeHandle = insertClustering
          ? new FlinkConcatHandle<>(config, instantTime, table, recordItr, partitionPath,
              fileID, table.getTaskContextSupplier())
          : new FlinkMergeHandle<>(config, instantTime, table, recordItr, partitionPath,
              fileID, table.getTaskContextSupplier());
    }

分成几种，

FlinkAppendHandle，FlinkCreateHandle，FlinkMergeHandle，基本重用HoodieAppendHandle，HoodieCreateHandle，HoodieMergeHandle

StreamWriteOperatorCoordinator

首先是Coordinator，所以要接收Operator发来的消息，

分三种，endInput，bootstrap，writeMeta

  @Override
  public void handleEventFromOperator(int i, OperatorEvent operatorEvent) {
    ValidationUtils.checkState(operatorEvent instanceof WriteMetadataEvent,
        "The coordinator can only handle WriteMetaEvent");
    WriteMetadataEvent event = (WriteMetadataEvent) operatorEvent;

    if (event.isEndInput()) {
      // handle end input event synchronously
      handleEndInputEvent(event); //
    } else {
      executor.execute(
          () -> {
            if (event.isBootstrap()) {
              handleBootstrapEvent(event); //
            } else {
              handleWriteMetaEvent(event); //
            }
          }, "handle write metadata event for instant %s", this.instant
      );
    }
  }

EventBuffer，数组，大小等于task的并发度，一个task对应于一个slot

所以通过EventBuffer，就可以很容易看出来各个task的event的情况

this.eventBuffer = new WriteMetadataEvent[this.parallelism];

3个handle event的逻辑

  private void handleBootstrapEvent(WriteMetadataEvent event) {
    this.eventBuffer[event.getTaskID()] = event; //把event放入到buffer
    if (Arrays.stream(eventBuffer).allMatch(evt -> evt != null && evt.isBootstrap())) { //如果所有的task都收到了bootstrap event
      // start to initialize the instant.
      initInstant(event.getInstantTime()); //初始化新的instant
    }
  }

  private void handleEndInputEvent(WriteMetadataEvent event) {
    addEventToBuffer(event); //放入buffer
    if (allEventsReceived()) { //如果所有task都收到EndInput
      // start to commit the instant.
      commitInstant(this.instant); //commit该instant
      // The executor thread inherits the classloader of the #handleEventFromOperator
      // caller, which is a AppClassLoader.
      Thread.currentThread().setContextClassLoader(getClass().getClassLoader());
      // sync Hive synchronously if it is enabled in batch mode.
      syncHive();
    }
  }

  private void handleWriteMetaEvent(WriteMetadataEvent event) {
    // the write task does not block after checkpointing(and before it receives a checkpoint success event),
    // if it checkpoints succeed then flushes the data buffer again before this coordinator receives a checkpoint
    // success event, the data buffer would flush with an older instant time.
    ValidationUtils.checkState(
        HoodieTimeline.compareTimestamps(this.instant, HoodieTimeline.GREATER_THAN_OR_EQUALS, event.getInstantTime()),
        String.format("Receive an unexpected event for instant %s from task %d",
            event.getInstantTime(), event.getTaskID()));

    addEventToBuffer(event); //写Meta的event直接buffer
  }

然后，

initInstant

  private void initInstant(String instant) {
    HoodieTimeline completedTimeline =
        StreamerUtil.createMetaClient(conf).getActiveTimeline().filterCompletedInstants(); //找出所有完成的instant
    executor.execute(() -> {
      if (instant.equals("") || completedTimeline.containsInstant(instant)) {
        // the last instant committed successfully
        reset(); //如果当前instant已经完成，那就清空eventBuffer
      } else {
        LOG.info("Recommit instant {}", instant);
        commitInstant(instant); //如果没有完成，先commit该instant
      }
      // starts a new instant
      startInstant(); //start一个新的instant
      // upgrade downgrade
      this.writeClient.upgradeDowngrade(this.instant);
    }, "initialize instant %s", instant);
  }

startInstant

  private void startInstant() {
    // put the assignment in front of metadata generation,
    // because the instant request from write task is asynchronous.
    this.instant = this.writeClient.startCommit(tableState.commitAction, this.metaClient); //在Timeline上创建新的instant
    this.metaClient.getActiveTimeline().transitionRequestedToInflight(tableState.commitAction, this.instant); //将instant的状态迁移成inflight
    this.ckpMetadata.startInstant(this.instant); //创建start instant对应的cp文件
  }

commitInstant，这个是在EndInputEvent中，即是batch写入的时候触发

  private boolean commitInstant(String instant, long checkpointId) {

    List<WriteStatus> writeResults = Arrays.stream(eventBuffer)
        .filter(Objects::nonNull)
        .map(WriteMetadataEvent::getWriteStatuses)
        .flatMap(Collection::stream)
        .collect(Collectors.toList()); //从Eventbuffer中提取出writeResults

    if (writeResults.size() == 0) { //如果没有results，直接ack
      sendCommitAckEvents(checkpointId);
      return false;
    }
    doCommit(instant, writeResults); //commit
    return true;
  }

doCommit

  private void doCommit(String instant, List<WriteStatus> writeResults) {
      final Map<String, List<String>> partitionToReplacedFileIds = tableState.isOverwrite
          ? writeClient.getPartitionToReplacedFileIds(tableState.operationType, writeResults)
          : Collections.emptyMap();
      boolean success = writeClient.commit(instant, writeResults, Option.of(checkpointCommitMetadata),
          tableState.commitAction, partitionToReplacedFileIds); //commit Metadata
      if (success) {
        reset();
        this.ckpMetadata.commitInstant(instant); //创建commit instant的cp file
        LOG.info("Commit instant [{}] success!", instant);
      } else {
        throw new HoodieException(String.format("Commit instant [%s] failed!", instant));
      }
  }

HoodieFlinkWriteClient

  public boolean commit(String instantTime, List<WriteStatus> writeStatuses, Option<Map<String, String>> extraMetadata, String commitActionType, Map<String, List<String>> partitionToReplacedFileIds) {
    List<HoodieWriteStat> writeStats = writeStatuses.parallelStream().map(WriteStatus::getStat).collect(Collectors.toList());
    return commitStats(instantTime, writeStats, extraMetadata, commitActionType, partitionToReplacedFileIds);
  }

BaseHoodieWriteClient

  public boolean commitStats(String instantTime, List<HoodieWriteStat> stats, Option<Map<String, String>> extraMetadata,
                             String commitActionType, Map<String, List<String>> partitionToReplaceFileIds) {
    // Create a Hoodie table which encapsulated the commits and files visible
    HoodieTable table = createTable(config, hadoopConf); //
    HoodieCommitMetadata metadata = CommitUtils.buildMetadata(stats, partitionToReplaceFileIds,
        extraMetadata, operationType, config.getWriteSchema(), commitActionType); //
    HoodieInstant inflightInstant = new HoodieInstant(State.INFLIGHT, table.getMetaClient().getCommitActionType(), instantTime);
    HeartbeatUtils.abortIfHeartbeatExpired(instantTime, table, heartbeatClient, config);
    this.txnManager.beginTransaction(Option.of(inflightInstant),
        lastCompletedTxnAndMetadata.isPresent() ? Option.of(lastCompletedTxnAndMetadata.get().getLeft()) : Option.empty());
    try {
      preCommit(inflightInstant, metadata); //主要是resolve写冲突
      commit(table, commitActionType, instantTime, metadata, stats); //commit，将instant存入timeline
      // 删除marker，自动clean，自动archive
      postCommit(table, metadata, instantTime, extraMetadata, false);
      LOG.info("Committed " + instantTime);
      releaseResources();
    } catch (IOException e) {
      throw new HoodieCommitException("Failed to complete commit " + config.getBasePath() + " at time " + instantTime, e);
    } finally {
      this.txnManager.endTransaction(Option.of(inflightInstant)); //更改meta前后需要加减锁
    }
    // do this outside of lock since compaction, clustering can be time taking and we don't need a lock for the entire execution period
    runTableServicesInline(table, metadata, extraMetadata); //执行定义的table services，如compaction或clustering
    emitCommitMetrics(instantTime, metadata, commitActionType);

再者，要响应checkpoint

对于流式写入，需要在每次checkpoint的时候去commitInstant，并且开启新的instant

  @Override
  public void notifyCheckpointComplete(long checkpointId) {
    executor.execute(
        () -> {
          // The executor thread inherits the classloader of the #notifyCheckpointComplete
          // caller, which is a AppClassLoader.
          Thread.currentThread().setContextClassLoader(getClass().getClassLoader());
          // for streaming mode, commits the ever received events anyway,
          // the stream write task snapshot and flush the data buffer synchronously in sequence,
          // so a successful checkpoint subsumes the old one(follows the checkpoint subsuming contract)
          final boolean committed = commitInstant(this.instant, checkpointId); //

          if (tableState.scheduleCompaction) {
            // if async compaction is on, schedule the compaction
            CompactionUtil.scheduleCompaction(metaClient, writeClient, tableState.isDeltaTimeCompaction, committed); //
          }

          if (committed) {
            // start new instant.
            startInstant(); //
            // sync Hive if is enabled
            syncHiveAsync();
          }
        }, "commits the instant %s", this.instant
    );
  }

  @Override
  public void notifyCheckpointAborted(long checkpointId) {
    if (checkpointId == this.checkpointId) {
      executor.execute(() -> {
        this.ckpMetadata.abortInstant(this.instant); //
      }, "abort instant %s", this.instant);
    }
  }