如何检索输出大小和从 Spark UI 写入的记录等指标? [英] How to retrieve Metrics like Output Size and Records Written from Spark UI?

问题描述

如何在任务或作业完成后立即在控制台(Spark Shell 或 Spark 提交作业)上收集这些指标.

How do I collect these metrics on a console (Spark Shell or Spark submit job) right after the task or job is done.

我们使用 Spark 将数据从 Mysql 加载到 Cassandra，它非常庞大(例如:~200 GB 和 600M 行).当任务完成后，我们要验证spark过程到底做了多少行?我们可以从 Spark UI 获取数字，但是我们如何从 spark shell 或 spark-submit 作业中检索该数字(写入的输出记录").

We are using Spark to load data from Mysql to Cassandra and it is quite huge (ex: ~200 GB and 600M rows). When the task the done, we want to verify how many rows exactly did spark process? We can get the number from Spark UI, but how can we retrieve that number ("Output Records Written") from spark shell or in spark-submit job.

从 Mysql 加载到 Cassandra 的示例命令.

Sample Command to load from Mysql to Cassandra.

val pt = sqlcontext.read.format("jdbc").option("url", "jdbc:mysql://...:3306/...").option("driver", "com.mysql.jdbc.Driver").option("dbtable", "payment_types").option("user", "hadoop").option("password", "...").load()

pt.save("org.apache.spark.sql.cassandra",SaveMode.Overwrite,options = Map( "table" -> "payment_types", "keyspace" -> "test"))

我想检索上述任务的所有 Spark UI 指标，主要是输出大小和写入的记录.

I want to retrieve all the Spark UI metrics on the above task mainly Output size and Records Written.

请帮忙.

感谢您的时间！

推荐答案

找到了答案.您可以使用 SparkListener 获取统计信息.

Found the answer. You can get the stats by using SparkListener.

如果您的工作没有输入或输出指标，您可能会得到 None.get 异常，您可以通过提供 if stmt 来安全地忽略这些异常.

If your job has no input or output metrics you might get None.get exceptions which you can safely ignore by providing if stmt.

sc.addSparkListener(new SparkListener() {
  override def onTaskEnd(taskEnd: SparkListenerTaskEnd) {
    val metrics = taskEnd.taskMetrics
    if(metrics.inputMetrics != None){
      inputRecords += metrics.inputMetrics.get.recordsRead}
    if(metrics.outputMetrics != None){
      outputWritten += metrics.outputMetrics.get.recordsWritten }
  }
})

请找到以下示例.

import org.apache.spark.SparkContext
import org.apache.spark.SparkConf
import com.datastax.spark.connector._
import org.apache.spark.sql._
import org.apache.spark.storage.StorageLevel
import org.apache.spark.scheduler.{SparkListener, SparkListenerTaskEnd}

val conf = new SparkConf()
.set("spark.cassandra.connection.host", "...")
.set("spark.driver.allowMultipleContexts","true")
.set("spark.master","spark://....:7077")
.set("spark.driver.memory","1g")
.set("spark.executor.memory","10g")
.set("spark.shuffle.spill","true")
.set("spark.shuffle.memoryFraction","0.2")
.setAppName("CassandraTest")
sc.stop
val sc = new SparkContext(conf)
val sqlcontext = new org.apache.spark.sql.SQLContext(sc)

var outputWritten = 0L

sc.addSparkListener(new SparkListener() {
  override def onTaskEnd(taskEnd: SparkListenerTaskEnd) {
    val metrics = taskEnd.taskMetrics
    if(metrics.inputMetrics != None){
      inputRecords += metrics.inputMetrics.get.recordsRead}
    if(metrics.outputMetrics != None){
      outputWritten += metrics.outputMetrics.get.recordsWritten }
  }
})

val bp = sqlcontext.read.format("jdbc").option("url", "jdbc:mysql://...:3306/...").option("driver", "com.mysql.jdbc.Driver").option("dbtable", "bucks_payments").option("partitionColumn","id").option("lowerBound","1").option("upperBound","14596").option("numPartitions","10").option("fetchSize","100000").option("user", "hadoop").option("password", "...").load()
bp.save("org.apache.spark.sql.cassandra",SaveMode.Overwrite,options = Map( "table" -> "bucks_payments", "keyspace" -> "test"))

println("outputWritten",outputWritten)

结果:

scala> println("outputWritten",outputWritten)
(outputWritten,16383)

这篇关于如何检索输出大小和从 Spark UI 写入的记录等指标?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！