从Pyspark DataFrame中的选定行获取特定字段 [英] Getting specific field from chosen Row in Pyspark DataFrame

问题描述

我有一个通过 pyspark 从JSON文件中通过

I have a Spark DataFrame built through pyspark from a JSON file as

sc = SparkContext()
sqlc = SQLContext(sc)

users_df = sqlc.read.json('users.json')

现在，我要访问 chosen_user 数据，这是其_id字段.我能做

Now, I want to access a chosen_user data, where this is its _id field. I can do

print users_df[users_df._id == chosen_user].show()

，这给了我完整的用户行.但是，假设我只想要Row中的一个特定字段，例如用户性别，我将如何获得它?

and this gives me the full Row of the user. But suppose I just want one specific field in the Row, say the user gender, how would I obtain it?

推荐答案

只需过滤并选择:

result = users_df.where(users_df._id == chosen_user).select("gender")

或col

from pyspark.sql.functions import col

result = users_df.where(col("_id") == chosen_user).select(col("gender"))

最后PySpark Row只是具有某些扩展名的tuple，因此您可以例如flatMap:

Finally PySpark Row is just a tuple with some extensions so you can for example flatMap:

result.rdd.flatMap(list).first()

或map带有以下内容:

result.rdd.map(lambda x: x.gender).first()

这篇关于从Pyspark DataFrame中的选定行获取特定字段的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！