将 DataFrame show() 的结果保存到 pyspark 中的字符串 [英] Saving result of DataFrame show() to string in pyspark

问题描述

我想在pyspark中捕获show的结果，类似于此处 和此处.我无法使用 pyspark 找到解决方案，只能使用 scala.

I would like to capture the result of show in pyspark, similar to here and here. I was not able to find a solution with pyspark, only scala.

df.show()
#+----+-------+
#| age|   name|
#+----+-------+
#|null|Michael|
#|  30|   Andy|
#|  19| Justin|
#+----+-------+

最终目的是在我的 logger.info 中将其捕获为字符串我试过 logger.info(df.show()) 它只会显示在控制台上.

The ultimate purpose is to capture this as string inside my logger.info I tried logger.info(df.show()) which will only display on console.

推荐答案

您可以使用与链接 在pyspark中捕获explain()的结果.只需检查 源代码show() 并观察它正在调用 self._jdf.showString().

You can build a helper function using the same approach as shown in post you linked Capturing the result of explain() in pyspark. Just examine the source code for show() and observe that it is calling self._jdf.showString().

答案取决于您使用的 spark 版本，因为 show() 的参数数量随时间而变化.

The answer depends on which version of spark you are using, as the number of arguments to show() has changed over time.

在 2.3 版本中，添加了 vertical 参数.

In version 2.3, the vertical argument was added.

def getShowString(df, n=20, truncate=True, vertical=False):
    if isinstance(truncate, bool) and truncate:
        return(df._jdf.showString(n, 20, vertical))
    else:
        return(df._jdf.showString(n, int(truncate), vertical))

Spark 版本 1.5 到 2.2

从 1.5 版开始，添加了 truncate 参数.

def getShowString(df, n=20, truncate=True):
    if isinstance(truncate, bool) and truncate:
        return(df._jdf.showString(n, 20))
    else:
        return(df._jdf.showString(n, int(truncate)))

Spark 版本 1.3 到 1.4

show 函数在 1.3 版本中首次引入.

Spark Versions 1.3 through 1.4

The show function was first introduced in version 1.3.

def getShowString(df, n=20):
    return(df._jdf.showString(n))

<小时>

现在使用辅助函数如下:

---

Now use the helper function as follows:

x = getShowString(df)  # default arguments
print(x)
#+----+-------+
#| age|   name|
#+----+-------+
#|null|Michael|
#|  30|   Andy|
#|  19| Justin|
#+----+-------+

或者在您的情况下:

logger.info(getShowString(df))

这篇关于将 DataFrame show() 的结果保存到 pyspark 中的字符串的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！