改善PySpark DataFrame.show输出以适合Jupyter笔记本 [英] Improve PySpark DataFrame.show output to fit Jupyter notebook

问题描述

在Jupyter笔记本中使用PySpark，Spark的 DataFrame.show 的输出与熊猫数据帧的显示方式相比，技术含量较低.我一直以为很好，它能完成工作"，直到得到:

Using PySpark in a Jupyter notebook, the output of Spark's DataFrame.show is low-tech compared to how Pandas DataFrames are displayed. I thought "Well, it does the job", until I got this:

输出未调整为笔记本电脑的宽度，因此线条难看.有没有一种方法可以自定义?甚至更好的是，有没有一种方法可以获取熊猫风格的输出(显然无需转换为 pandas.DataFrame )?

The output is not adjusted to the width of the notebook, so that the lines wrap in an ugly way. Is there a way to customize this? Even better, is there a way to get output Pandas-style (without converting to pandas.DataFrame obviously)?

推荐答案

在处理了具有很多列的表之后，我决定要做的最好的事情就是使用数据:

After playing around with my table which has a lot of columns I decided the best thing to do to get a feel for the data is to use:

df.show(n=5, truncate=False, vertical=True)

这将垂直显示它而不会被截断，这是我能想到的最清晰的视图.

This displays it vertically without truncation and is the cleanest viewing I can come up with.

这篇关于改善PySpark DataFrame.show输出以适合Jupyter笔记本的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！