将数组传递给Python Spark Lit函数 [英] Passing Array to Python Spark Lit Function

问题描述

假设我有一个numpy数组，其中包含数字1-10.因此，a为[1 2 3 4 5 6 7 8 9 10].

Let's say I have a numpy array a that contains the numbers 1-10. So a is [1 2 3 4 5 6 7 8 9 10].

现在，我还有一个Python Spark数据框，要将numpy数组添加到该数据框.我认为一列文字可以胜任.因此，我执行以下操作:

Now, I also have a Python Spark dataframe to which I want to add my numpy array a. I figure that a column of literals will do the job. So I do the following:

df = df.withColumn("NewColumn", F.lit(a))

这不起作用.错误是不支持的文字类型类java.util.ArrayList".

This doesn't work. The error is "Unsupported literal type class java.util.ArrayList".

现在，如果我只尝试数组的一个元素，如下所示，它将起作用.

Now, if I try just one element of the array, as follows, it works.

df = df.withColumn("NewColumn", F.lit(a[0]))

有没有办法可以做我想做的事情?我已经完成了几天要完成的任务，这是我最近完成的任务.我看了所有相关的Stack Overflow问题，但没有得到我一直在寻找的答案.任何帮助表示赞赏.谢谢.

Is there a way I can do what I'm trying? I've been working on the task I want to complete for days and this is the closest I've come to finishing it. I have looked at all related Stack Overflow questions but I didn't get quite the answer I was looking for. Any help is appreciated. Thanks.

推荐答案

for数组内置函数中的循环

您可以将array 内置函数用作

a = [1,2,3,4,5,6,7,8,9,10]
df = spark.createDataFrame([['a b c d e f g h i j '],], ['col1'])
df = df.withColumn("NewColumn", F.array([F.lit(x) for x in a]))
df.show(truncate=False)

您应该获得

+--------------------+-------------------------------+
|col1                |NewColumn                      |
+--------------------+-------------------------------+
|a b c d e f g h i j |[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]|
+--------------------+-------------------------------+
root
 |-- col1: string (nullable = true)
 |-- NewColumn: array (nullable = false)
 |    |-- element: integer (containsNull = false)

使用udf函数

#udf function
def arrayUdf():
    return a
callArrayUdf = F.udf(arrayUdf, T.ArrayType(T.IntegerType()))

#calling udf function
df = df.withColumn("NewColumn", callArrayUdf())

输出与for循环方式相同

output is same as with for loop way

已更新

我要粘贴下面提供的@pault的评论

I am pasting @pault's comment given below

您可以使用map隐藏循环:df.withColumn("NewColumn", F.array(map(F.lit, a)))

You can hide the loop using map: df.withColumn("NewColumn", F.array(map(F.lit, a)))

这篇关于将数组传递给Python Spark Lit函数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！