spark-将数据框转换为列表以提高性能 [英] spark - Converting dataframe to list improving performance

问题描述

我需要隐蔽Spark数据框的一列以列出以便稍后用于matplotlib

I need to covert a column of the Spark dataframe to list to use later for matplotlib

df.toPandas()[col_name].values.tolist()

此操作似乎需要高性能，大约需要18秒 还有其他方法可以改善性能吗?

it looks like there is high performance overhead this operation takes around 18sec is there other way to do that or improve the perfomance?

推荐答案

如果您确实需要本地列表，则可以在此处执行很多操作，但一种改进是仅收集单个列而不是整个DataFrame:

If you really need a local list there is not much you can do here but one improvement is to collect only a single column not a whole DataFrame:

df.select(col_name).flatMap(lambda x: x).collect()

这篇关于spark-将数据框转换为列表以提高性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！