如何在Kafka和Flink环境下测试性能? [英] How performance can be tested in Kafka and Flink environment?

问题描述

如何以 kafka 作为输入源对 Flink 进行性能测试.另外，请推荐是否有适用于这种情况的性能测试工具.

How is performance tested for Flink with kafka as input source. Also, recommend if any performance test tools are available for this case.

推荐答案

Flink 包括吞吐量(numRecordsInPerSecond 和 numRecordsOutPerSecond)和 延迟.

Flink includes metrics for both throughput (numRecordsInPerSecond and numRecordsOutPerSecond) and latency.

如果您想更仔细地测量端到端的延迟，您可以在接收器(或其他终端节点)中添加自定义指标，将事件中的时间戳与当前时间进行比较.看起来像这样:

If you want to more carefully measure latency end-to-end, you can add a custom metric in a sink (or other terminal node) that compares timestamps in your events to the current time. That would look something like this:

public class LatencyMeasuringSink<T> extends RichSinkFunction<T> {
  private transient DescriptiveStatisticsHistogram eventTimeLag;
  private static final int EVENT_TIME_LAG_WINDOW_SIZE = 10_000;

  @Override
  public void open(Configuration parameters) throws Exception {
    super.open(parameters);

    eventTimeLag = getRuntimeContext().getMetricGroup().histogram("eventTimeLag",
            new DescriptiveStatisticsHistogram(EVENT_TIME_LAG_WINDOW_SIZE));
  }

  @Override
  public void invoke(T dataPoint, Context context) throws Exception {
    eventTimeLag.update(System.currentTimeMillis() - dataPoint.getTimeStampMs());
  }
}

您可能希望将 Kafka 生产者配置为将 LogAppendTime 时间戳放在您的事件中，并将其用作比较的基础.当然，这假设所涉及的不同机器中的时钟同步得足够好，以便此测量有意义 - 或者您可以在单台机器上运行测试.

You might want to configure your Kafka producer to put LogAppendTime timestamps in your events, and use those as the basis for comparison. This assumes, of course, that the clocks in the different machines involved are synchronized well enough for this measurement to be meaningful -- or you might run the tests on a single machine.

FLIP-83:Flink 端到端性能测试框架也可能引起关注.这是正在进行中的工作.

FLIP-83: Flink End-to-end Performance Testing Framework may also be of interest. This is work in progress.

这篇关于如何在Kafka和Flink环境下测试性能?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！