在PySpark中爆炸 [英] Explode in PySpark
问题描述
我想从包含单词列表的DataFrame转换为每个单词都在其自己行中的DataFrame.
I would like to transform from a DataFrame that contains lists of words into a DataFrame with each word in its own row.
如何在DataFrame的列上爆炸?
How do I do explode on a column in a DataFrame?
这是我的一些尝试的示例,您可以在其中取消注释每个代码行并获取以下注释中列出的错误.我在带有Spark 1.6.1的Python 2.7中使用PySpark.
Here is an example with some of my attempts where you can uncomment each code line and get the error listed in the following comment. I use PySpark in Python 2.7 with Spark 1.6.1.
from pyspark.sql.functions import split, explode
DF = sqlContext.createDataFrame([('cat \n\n elephant rat \n rat cat', )], ['word'])
print 'Dataset:'
DF.show()
print '\n\n Trying to do explode: \n'
DFsplit_explode = (
DF
.select(split(DF['word'], ' '))
# .select(explode(DF['word'])) # AnalysisException: u"cannot resolve 'explode(word)' due to data type mismatch: input to function explode should be array or map type, not StringType;"
# .map(explode) # AttributeError: 'PipelinedRDD' object has no attribute 'show'
# .explode() # AttributeError: 'DataFrame' object has no attribute 'explode'
).show()
# Trying without split
print '\n\n Only explode: \n'
DFsplit_explode = (
DF
.select(explode(DF['word'])) # AnalysisException: u"cannot resolve 'explode(word)' due to data type mismatch: input to function explode should be array or map type, not StringType;"
).show()
请咨询
推荐答案
explode
和split
是SQL函数.两者都在SQL Column
上运行. split
将Java正则表达式作为第二个参数.如果要在任意空格上分离数据,则需要这样的东西:
explode
and split
are SQL functions. Both operate on SQL Column
. split
takes a Java regular expression as a second argument. If you want to separate data on arbitrary whitespace you'll need something like this:
df = sqlContext.createDataFrame(
[('cat \n\n elephant rat \n rat cat', )], ['word']
)
df.select(explode(split(col("word"), "\s+")).alias("word")).show()
## +--------+
## | word|
## +--------+
## | cat|
## |elephant|
## | rat|
## | rat|
## | cat|
## +--------+
这篇关于在PySpark中爆炸的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!