改进 PySpark DataFrame.show 输出以适合 Jupyter 笔记本 [英] Improve PySpark DataFrame.show output to fit Jupyter notebook

问题描述

在 Jupyter notebook 中使用 PySpark，与 Pandas DataFrames 的显示方式相比，Spark 的 DataFrame.show 的输出技术含量较低.我想嗯，它可以完成工作"，直到我得到这个:

输出未调整到笔记本的宽度，因此线条以丑陋的方式环绕.有没有办法自定义这个?更好的是，有没有办法获得 Pandas 风格的输出(显然不转换为 pandas.DataFrame)?

解决方案

这现在

Using PySpark in a Jupyter notebook, the output of Spark's DataFrame.show is low-tech compared to how Pandas DataFrames are displayed. I thought "Well, it does the job", until I got this:

The output is not adjusted to the width of the notebook, so that the lines wrap in an ugly way. Is there a way to customize this? Even better, is there a way to get output Pandas-style (without converting to pandas.DataFrame obviously)?

解决方案

This is now possible natively as of Spark 2.4.0 by setting spark.sql.repl.eagerEval.enabled to True:

这篇关于改进 PySpark DataFrame.show 输出以适合 Jupyter 笔记本的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！