从Pyspark DataFrame中的选定行获取特定字段 [英] Getting specific field from chosen Row in Pyspark DataFrame
本文介绍了从Pyspark DataFrame中的选定行获取特定字段的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我有一个通过 pyspark 从JSON文件中通过
I have a Spark DataFrame built through pyspark from a JSON file as
sc = SparkContext()
sqlc = SQLContext(sc)
users_df = sqlc.read.json('users.json')
现在,我要访问 chosen_user 数据,这是其_id字段.我能做
Now, I want to access a chosen_user data, where this is its _id field. I can do
print users_df[users_df._id == chosen_user].show()
,这给了我完整的用户行.但是,假设我只想要Row中的一个特定字段,例如用户性别,我将如何获得它?
and this gives me the full Row of the user. But suppose I just want one specific field in the Row, say the user gender, how would I obtain it?
推荐答案
只需过滤并选择:
result = users_df.where(users_df._id == chosen_user).select("gender")
或col
from pyspark.sql.functions import col
result = users_df.where(col("_id") == chosen_user).select(col("gender"))
最后PySpark Row
只是具有某些扩展名的tuple
,因此您可以例如flatMap
:
Finally PySpark Row
is just a tuple
with some extensions so you can for example flatMap
:
result.rdd.flatMap(list).first()
或map
带有以下内容:
result.rdd.map(lambda x: x.gender).first()
这篇关于从Pyspark DataFrame中的选定行获取特定字段的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文