在同一个调用中从 Spark Dataframes 拆分方法中选择数组元素? [英] Select array element from Spark Dataframes split method in same call?

问题描述

我正在拆分 HTTP 请求以查看元素，我想知道是否有一种方法可以指定我想在同一个调用中查看的元素，而无需执行其他操作.

I'm splitting an HTTP request to look at the elements, and I was wondering if there was a way to specify the element I'd like to look at in the same call without having to do another operation.

例如:

from pyspark.sql import functions as fn

df.select(fn.split(df.http_request, '/').alias('http'))

给我一​​个新的 Dataframe，其中包含如下所示的数组行:

gives me a new Dataframe with rows of arrays like this:

+--------------------+
|                http|
+--------------------+
|[, courses, 26420...|

我想要索引 1(课程)中的项目，而不必再执行另一个 select 语句来指定 df.select(df.http[1]) 或其他任何内容.这可能吗?

I want the item in index 1 (courses) without having to then do another select statement to specify df.select(df.http[1]) or whatever. Is this possible?

推荐答案

好吧，你可以定义一个 UDF:

Well you could define a UDF:

from pyspark.sql.functions import *
from pyspark.sql.types import *

def getter(column, index):
    return column[index]

getterUDF = udf(getter, StringType())

df.select(getterUDF(split(df.http_request, '/').alias('http'), lit(1)))

你也可以使用@max推荐的getItem方法

You could also use the getItem method recommended by @max

df.select(F.split(df.http_request, '/').alias('http').getItem(1))

这篇关于在同一个调用中从 Spark Dataframes 拆分方法中选择数组元素?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！