如何分析 pyspark 工作 [英] How to profile pyspark jobs

问题描述

我想了解 pyspark 代码中的分析.

如下:https://github.com/apache/spark/pull/2351

<预><代码>>>>sc._conf.set("spark.python.profile", "true")>>>rdd = sc.parallelize(range(100)).map(str)>>>rdd.count()100>>>sc.show_profiles()============================================================RDD id=1 的配置文件============================================================0.001 秒内 284 次函数调用(276 次原始调用)排序方式:内部时间、累计时间ncalls tottime percall cumtime percall filename:lineno(function)4 0.000 0.000 0.000 0.000 serializers.py:198(load_stream)4 0.000 0.000 0.000 0.000 {减少}12/4 0.000 0.000 0.001 0.000 rdd.py:2092(pipeline_func)4 0.000 0.000 0.000 0.000 {cPickle.loads}4 0.000 0.000 0.000 0.000 {cPickle.dumps}104 0.000 0.000 0.000 0.000 rdd.py:852(<genexpr>)8 0.000 0.000 0.000 0.000 serializers.py:461(read_int)12 0.000 0.000 0.000 0.000 rdd.py:303(func)

以上效果很好.但是，如果我执行以下操作:

from pyspark.sql import HiveContext从 pyspark 导入 SparkConf从 pyspark 导入 SparkContextconf = SparkConf().setAppName("myapp").set("spark.python.profile","true")sc = SparkContext(conf=conf)sqlContext = HiveContext(sc)df=sqlContext.sql("select * from myhivetable")df.count()sc.show_profiles()

这并没有给我任何东西.我得到计数​​但 show_profiles() 给我 None

感谢任何帮助

解决方案

当您使用 Spark SQL 时，没有要分析的 Python 代码.唯一的 Python 是调用 Scala 引擎.其他一切都在 Java 虚拟机上执行.

I want to understand profiling in pyspark codes.

Following this: https://github.com/apache/spark/pull/2351

>>> sc._conf.set("spark.python.profile", "true")
>>> rdd = sc.parallelize(range(100)).map(str)
>>> rdd.count()
100
>>> sc.show_profiles()
============================================================
Profile of RDD<id=1>
============================================================
         284 function calls (276 primitive calls) in 0.001 seconds

   Ordered by: internal time, cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
        4    0.000    0.000    0.000    0.000 serializers.py:198(load_stream)
        4    0.000    0.000    0.000    0.000 {reduce}
     12/4    0.000    0.000    0.001    0.000 rdd.py:2092(pipeline_func)
        4    0.000    0.000    0.000    0.000 {cPickle.loads}
        4    0.000    0.000    0.000    0.000 {cPickle.dumps}
      104    0.000    0.000    0.000    0.000 rdd.py:852(<genexpr>)
        8    0.000    0.000    0.000    0.000 serializers.py:461(read_int)
       12    0.000    0.000    0.000    0.000 rdd.py:303(func)

Above works great. But If I do something like below:

from pyspark.sql import HiveContext
from pyspark import SparkConf
from pyspark import SparkContext
conf = SparkConf().setAppName("myapp").set("spark.python.profile","true")
sc   = SparkContext(conf=conf)
sqlContext = HiveContext(sc)

df=sqlContext.sql("select * from myhivetable")
df.count()
sc.show_profiles()

This does not give me anything. I get the count but show_profiles() give me None

Any help appreciated

解决方案

There is no Python code to profile when you use Spark SQL. The only Python is to call Scala engine. Everything else is executed on Java Virtual Machine.

这篇关于如何分析 pyspark 工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！