在同一调用中从Spark Dataframes拆分方法中选择数组元素? [英] Select array element from Spark Dataframes split method in same call?
本文介绍了在同一调用中从Spark Dataframes拆分方法中选择数组元素?的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!
问题描述
我正在拆分HTTP请求以查看元素,我想知道是否有一种方法可以指定我希望在同一调用中查看的元素,而无需执行其他操作.
I'm splitting an HTTP request to look at the elements, and I was wondering if there was a way to specify the element I'd like to look at in the same call without having to do another operation.
例如:
from pyspark.sql import functions as fn
df.select(fn.split(df.http_request, '/').alias('http'))
给我一个新的Dataframe
,它具有如下这样的数组行:
gives me a new Dataframe
with rows of arrays like this:
+--------------------+
| http|
+--------------------+
|[, courses, 26420...|
我想要索引1(课程)中的项目,而不必执行另一个select
语句来指定df.select(df.http[1])
或其他内容.这可能吗?
I want the item in index 1 (courses) without having to then do another select
statement to specify df.select(df.http[1])
or whatever. Is this possible?
推荐答案
那么您可以定义UDF
:
from pyspark.sql.functions import *
from pyspark.sql.types import *
def getter(column, index):
return column[index]
getterUDF = udf(getter, StringType())
df.select(getterUDF(split(df.http_request, '/').alias('http'), lit(1)))
您还可以使用@max推荐的getItem
方法
You could also use the getItem
method recommended by @max
df.select(F.split(df.http_request, '/').alias('http').getItem(1))
这篇关于在同一调用中从Spark Dataframes拆分方法中选择数组元素?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!
查看全文