Pyspark-如何在RDD操作中检查变量 [英] Pyspark - How to inspect variables within RDD operations

问题描述

我曾经使用IntelliJ在Scala Spark中进行开发.通过设置断点，我能够在调试模式下检查变量内容.像这样

I used to develop in Scala Spark using IntelliJ. I was able to inspect variable contents under debug mode by setting break point. Like this

我最近使用pyspark和pycharm启动了一个新项目.我发现代码不会在Spark操作的断点处停止，如下所示.

I recently start a new project using pyspark with pycharm. I found code does not stop at break point in Spark operations, like below.

另一个问题是提示提示没有从地图"功能提供正确的提示.似乎IDE不知道来自"map"函数的变量仍然是RDD，我想它与python函数有关，未定义返回类型.

And another question is the prompt hint does not give right hint for instance from "map" function. Seems IDE does not know the variable from "map" function is still RDD, my guess is it related to python function does not define return type.

对于PySpark开发人员，我感到这些天真问题.任何帮助都会很棒，谢谢！

I feel these naive question for PySpark developers. Any help would be great, thank you!

推荐答案

"...代码不会在Spark操作的断点处停止，如下所示..."-请您说明一下您的PyCharm版本和操作系统是什么?

"...code does not stop at break point in Spark operations, like below..." - Could you please clarify what is your PyCharm version and OS?

另外一个问题是提示提示无法从"map"功能提供正确的提示.似乎IDE不知道"map"功能的变量仍然是rdd ..."-我相信它与此功能请求 https://youtrack.jetbrains.com/issue/PY-29811

"And another question is the prompt hint does not give right hint for instance from "map" function. Seems IDE does not know the variable from "map" function is still rdd..." - I believe it is related to this feature request https://youtrack.jetbrains.com/issue/PY-29811

这篇关于Pyspark-如何在RDD操作中检查变量的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！