Spark Streaming-获取批处理级别的性能统计信息 [英] Spark Streaming - obtain batch-level performance stats

问题描述

我正在设置一个 Apache Spark 集群以执行实时流计算，并希望通过跟踪各种指标(例如批大小，批处理时间等)来监视部署的性能. Spark Streaming 程序是用 Scala

I'm setting up an Apache Spark cluster to perform realtime streaming computations and would like to monitor the performance of the deployment by tracking various metrics like sizes of batches, batch processing times, etc. My Spark Streaming program is written in Scala

问题

1. 火花监控REST API 描述列出了各种可用的端点.但是，我找不到暴露批处理级别信息的端点.有没有一种方法可以获取已为应用程序运行的所有Spark批次的列表以及其他按批次的详细信息，例如:
   • 每批事件数
   • 处理时间
   • 计划延迟
   • 退出状态:，即批处理是否成功

1. The Spark monitoring REST API description lists the various endpoints available. However, I couldn't find endpoints that expose batch-level info. Is there a way to get a list of all the Spark batches that have been run for an application and other per-batch details such as follows:
   • Number of events per batch
   • Processing time
   • Scheduling delay
   • Exit status: ie, whether the batch was processed successfully or not

预先感谢

推荐答案

如果您对1.没有运气，那么对2.会有帮助.

If you have no luck with 1., this will help with 2.:

ssc.addStreamingListener(new JobListener());

// ...

class JobListener implements StreamingListener {

    @Override
    public void onBatchCompleted(StreamingListenerBatchCompleted batchCompleted) {

        System.out.println("Batch completed, Total delay :" + batchCompleted.batchInfo().totalDelay().get().toString() +  " ms");

    }

   /*

   snipped other methods

   */


}

来自 batchCompleted.batchInfo()

numRecords

batchTime ， processsingStartTime ， processingEndTime

schedulingDelay

outputOperationInfos

• numRecords
• batchTime, processsingStartTime, processingEndTime
• schedulingDelay
• outputOperationInfos

希望您可以从这些属性中获得所需的内容.

Hopefully you can get what you need from those properties.

这篇关于Spark Streaming-获取批处理级别的性能统计信息的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！