如何从PySpark中的字符串获取列表 [英] How to get a List from a String in PySpark
问题描述
在PySpark中是否存在与 eval 类似的功能.
Is there something like an eval function equivalent in PySpark.
我正在尝试将Python代码转换为PySpark
I am trying to convert Python code into PySpark
我正在查询一个数据框，并且其中一个列具有如下所示的数据，但其格式为字符串格式.
I am Querying a Dataframe and one of the Column has the Data as shown below but in String Format.
[{u'date': u'2015-02-08', u'by': u'abc@gg.com', u'value': u'NA'}, {u'date': u'2016-02-08', u'by': u'dfg@yaa.com', u'value': u'applicable'}, {u'date': u'2017-02-08', u'by': u'wrwe@hot.com', u'value': u'ufc'}]
假定"x"是在数据框中保存此值的列.
Assume that 'x' is the column which holds this value in the Dataframe.
现在我想传递该字符串列'x'并获取列表，以便我可以将其传递给mapPartition函数.
Now i want to pass in that String column 'x' and get the List so that i can pass it to mapPartition function.
我想避免迭代驱动程序上的每一行，这就是我这样想的原因.
I want to avoid iterating to each row on my Driver that's the reason i am thinking this way.
在Python中使用eval()函数(如果使用的话):我得到以下输出:
In Python using eval() function if used: I get below output:
x = "[{u'date': u'2015-02-08', u'by': u'abc@gg.com', u'value': u'NA'}, {u'date': u'2016-02-08', u'by': u'dfg@yaa.com', u'value': u'applicable'}, {u'date': u'2017-02-08', u'by': u'wrwe@hot.com', u'value': u'ufc'}]"
list = eval(x)
for i in list: print i
输出:(这也是我在PySpark中想要的)
{u'date': u'2015-02-08', u'by': u'abc@gg.com', u'value': u'NA'}
{u'date': u'2016-02-08', u'by': u'dfg@yaa.com', u'value': u'applicable'}
{u'date': u'2017-02-08', u'by': u'wrwe@hot.com', u'value': u'ufc'}
如何在PySpark中做到这一点?
How to do this in PySpark ??
推荐答案
使用 from_json 函数将json字符串转换为实际json ，您可以从中受益.为此，您将必须定义与您的json字符串匹配的 schema .最后，像使用 eval 一样，使用 explode 函数将 struct数组分隔到不同的行.
You can benefit by using from_json function to convert your json string to actual json. For that you will have to define a schema matching to your json string. And finally use explode function to separate the struct array to different rows as you did with eval.
如果您有数据
x = "[{u'date': u'2015-02-08', u'by': u'abc@gg.com', u'value': u'NA'}, {u'date': u'2016-02-08', u'by': u'dfg@yaa.com', u'value': u'applicable'}, {u'date': u'2017-02-08', u'by': u'wrwe@hot.com', u'value': u'ufc'}]"
然后创建数据框架
df = sqlContext.createDataFrame([(x,),], ["x"])
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|x |
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|[{u'date': u'2015-02-08', u'by': u'abc@gg.com', u'value': u'NA'}, {u'date': u'2016-02-08', u'by': u'dfg@yaa.com', u'value': u'applicable'}, {u'date': u'2017-02-08', u'by': u'wrwe@hot.com', u'value': u'ufc'}]|
+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
root
|-- x: string (nullable = true)
使用json
正如我所解释的，您将需要一个 schema ， regexp_replace 函数， from_json 函数和 explode 函数
As I had explained, you would need a schema, regexp_replace function, from_json function and explode function as
from pyspark.sql import types as T
schema = T.ArrayType(T.StructType([T.StructField('date', T.StringType()), T.StructField('by', T.StringType()), T.StructField('value', T.StringType())]))
from pyspark.sql import functions as F
df = df.withColumn("x", F.explode(F.from_json(F.regexp_replace(df['x'], "(u')", "'"), schema=schema)))
应该给您
+-----------------------------------+
|x |
+-----------------------------------+
|[2015-02-08,abc@gg.com,NA] |
|[2016-02-08,dfg@yaa.com,applicable]|
|[2017-02-08,wrwe@hot.com,ufc] |
+-----------------------------------+
root
|-- x: struct (nullable = true)
| |-- date: string (nullable = true)
| |-- by: string (nullable = true)
| |-- value: string (nullable = true)
如果您需要问题中提到的json字符串，则可以将 to_json 函数用作
df = df.withColumn("x", F.to_json(df['x']))
这会给你
+-------------------------------------------------------------+
|x |
+-------------------------------------------------------------+
|{"date":"2015-02-08","by":"abc@gg.com","value":"NA"} |
|{"date":"2016-02-08","by":"dfg@yaa.com","value":"applicable"}|
|{"date":"2017-02-08","by":"wrwe@hot.com","value":"ufc"} |
+-------------------------------------------------------------+
仅使用字符串
如果您不想遍历json的所有复杂性，则可以简单地使用字符串.为此，您需要嵌套 regex_replace ， split 和 explode 函数作为
If you don't want to go through all the complexities of jsons then you can simply work with strings. For that you would need nested regex_replace, split and explode functions as
from pyspark.sql import functions as F
df = df.withColumn("x", F.explode(F.split(F.regexp_replace(F.regexp_replace(F.regexp_replace(df['x'], "(u')", "'"), "[\\[\\]\s]", ""), "},\\{", "};&;{"), ";&;")))
应该给您
+-------------------------------------------------------------+
|x |
+-------------------------------------------------------------+
|{'date':'2015-02-08','by':'abc@gg.com','value':'NA'} |
|{'date':'2016-02-08','by':'dfg@yaa.com','value':'applicable'}|
|{'date':'2017-02-08','by':'wrwe@hot.com','value':'ufc'} |
+-------------------------------------------------------------+
这篇关于如何从PySpark中的字符串获取列表的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
