将数组传递给 Spark Lit 函数 [英] Passing Array to Spark Lit function
问题描述
假设我有一个包含数字 1-10 的 numpy 数组 a
:[1 2 3 4 5 6 7 8 9 10]
我还有一个 Spark 数据框,我想向其中添加我的 numpy 数组 a
.我认为一列文字可以完成这项工作.这不起作用:
df = df.withColumn(NewColumn", F.lit(a))
<块引用>
不支持的文字类型类 java.util.ArrayList
但这有效:
df = df.withColumn(NewColumn", F.lit(a[0]))
怎么做?
之前的示例 DF:
col1 |
---|
a b c d e f g h i j |
预期结果:
col1 | 新列 |
---|---|
a b c d e f g h i j | 1 2 3 4 5 6 7 8 9 10 |
Spark array
中的列表解析a = [1,2,3,4,5,6,7,8,9,10]df = spark.createDataFrame([['a b c d e f g h i j '],], ['col1'])df = df.withColumn(NewColumn", F.array([F.lit(x) for x in a]))df.show(截断=假)df.printSchema()# +--------------------+-------------------------------+# |col1 |新列|# +--------------------+-------------------------------+# |a b c d e f g h i j |[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]|# +--------------------+-------------------------------+# 根# |-- col1: string (nullable = true)# |-- NewColumn: 数组 (nullable = false)# ||-- 元素:整数(containsNull = false)
@pault 评论 (Python 2.7):
<块引用>您可以使用 map
隐藏循环:df.withColumn(NewColumn", F.array(map(F.lit, a)))
@abegehr 添加了 Python 3 版本:
<块引用>df.withColumn(NewColumn", F.array(*map(F.lit, a)))
Spark 的 udf
#定义UDFdef arrayUdf():返回一个callArrayUdf = F.udf(arrayUdf, T.ArrayType(T.IntegerType()))# 调用UDFdf = df.withColumn(NewColumn", callArrayUdf())
输出是一样的.
Let's say I have a numpy array a
that contains the numbers 1-10:
[1 2 3 4 5 6 7 8 9 10]
I also have a Spark dataframe to which I want to add my numpy array a
. I figure that a column of literals will do the job. This doesn't work:
df = df.withColumn("NewColumn", F.lit(a))
Unsupported literal type class java.util.ArrayList
But this works:
df = df.withColumn("NewColumn", F.lit(a[0]))
How to do it?
Example DF before:
col1 |
---|
a b c d e f g h i j |
Expected result:
col1 | NewColumn |
---|---|
a b c d e f g h i j | 1 2 3 4 5 6 7 8 9 10 |
List comprehension inside Spark's array
a = [1,2,3,4,5,6,7,8,9,10]
df = spark.createDataFrame([['a b c d e f g h i j '],], ['col1'])
df = df.withColumn("NewColumn", F.array([F.lit(x) for x in a]))
df.show(truncate=False)
df.printSchema()
# +--------------------+-------------------------------+
# |col1 |NewColumn |
# +--------------------+-------------------------------+
# |a b c d e f g h i j |[1, 2, 3, 4, 5, 6, 7, 8, 9, 10]|
# +--------------------+-------------------------------+
# root
# |-- col1: string (nullable = true)
# |-- NewColumn: array (nullable = false)
# | |-- element: integer (containsNull = false)
@pault commented (Python 2.7):
You can hide the loop using
map
:
df.withColumn("NewColumn", F.array(map(F.lit, a)))
@ abegehr added Python 3 version:
df.withColumn("NewColumn", F.array(*map(F.lit, a)))
Spark's udf
# Defining UDF
def arrayUdf():
return a
callArrayUdf = F.udf(arrayUdf, T.ArrayType(T.IntegerType()))
# Calling UDF
df = df.withColumn("NewColumn", callArrayUdf())
Output is the same.
这篇关于将数组传递给 Spark Lit 函数的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!