Pyspark-拆分一列并采用n个元素 [英] Pyspark - Split a column and take n elements

问题描述

我想取一列并使用字符分割字符串.像往常一样，我知道方法split将返回一个列表，但是在编码时，我发现返回的对象仅具有方法getItem或getField，并具有API中的以下描述:

I want to take a column and split a string using a character. As per usual, I understood that the method split would return a list, but when coding I found that the returning object had only the methods getItem or getField with the following descriptions from the API:

@since(1.3)   
def getItem(self, key):
    """
    An expression that gets an item at position ``ordinal`` out of a list,
    or gets an item by key out of a dict.


@since(1.3)
def getField(self, name):
    """
    An expression that gets a field by name in a StructField.

很明显，这不符合我的要求，例如对于"A_B_C_D"列中的文本，我想在两个不同的列中将"A_B_C_"和"D"分开.

Obviously this doesnt meet my requirements, for example for the text within the column "A_B_C_D" I would like to split between "A_B_C_" and "D" in two different columns.

这是我正在使用的代码

from pyspark.sql.functions import regexp_extract, col, split
df_test=spark.sql("SELECT * FROM db_test.table_test")
#Applying the transformations to the data

split_col=split(df_test['Full_text'],'_')
df_split=df_test.withColumn('Last_Item',split_col.getItem(3))

查找示例:

from pyspark.sql import Row
from pyspark.sql.functions import regexp_extract, col, split
l = [("Item1_Item2_ItemN"),("FirstItem_SecondItem_LastItem"),("ThisShouldBeInTheFirstColumn_ThisShouldBeInTheLastColumn")]
rdd = sc.parallelize(l)
datax = rdd.map(lambda x: Row(fullString=x))
df = sqlContext.createDataFrame(datax)
split_col=split(df['fullString'],'_')
df=df.withColumn('LastItemOfSplit',split_col.getItem(2))

结果:

fullString                                                LastItemOfSplit
Item1_Item2_ItemN                                            ItemN
FirstItem_SecondItem_LastItem                                LastItem
ThisShouldBeInTheFirstColumn_ThisShouldBeInTheLastColumn     null

我的预期结果是总是有最后一个项目

My expected result would be having always the last item

fullString                                                LastItemOfSplit
Item1_Item2_ItemN                                            ItemN
FirstItem_SecondItem_LastItem                                LastItem
ThisShouldBeInTheFirstColumn_ThisShouldBeInTheLastColumn  ThisShouldBeInTheLastColumn

推荐答案

您可以使用getItem(size - 1)从数组中获取最后一项:

You can use getItem(size - 1) to get the last item from the arrays:

示例:

df = spark.createDataFrame([[['A', 'B', 'C', 'D']], [['E', 'F']]], ['split'])
df.show()
+------------+
|       split|
+------------+
|[A, B, C, D]|
|      [E, F]|
+------------+

import pyspark.sql.functions as F
df.withColumn('lastItem', df.split.getItem(F.size(df.split) - 1)).show()
+------------+--------+
|       split|lastItem|
+------------+--------+
|[A, B, C, D]|       D|
|      [E, F]|       F|
+------------+--------+

针对您的情况:

from pyspark.sql.functions import regexp_extract, col, split, size
df_test=spark.sql("SELECT * FROM db_test.table_test")
#Applying the transformations to the data

split_col=split(df_test['Full_text'],'_')
df_split=df_test.withColumn('Last_Item',split_col.getItem(size(split_col) - 1))

这篇关于Pyspark-拆分一列并采用n个元素的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！