由于数据类型不匹配 PySpark 无法解析列 [英] cannot resolve column due to data type mismatch PySpark
本文介绍了由于数据类型不匹配 PySpark 无法解析列的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！
问题描述
在 PySpark 中遇到的错误:
Error being faced in PySpark:
pyspark.sql.utils.AnalysisException: "cannot resolve '`result_set`.`dates`.`trackers`['token']' due to data type mismatch: argument 2 requires integral type, however, ''token'' is of string type.;;\n'Project [result_parameters#517, result_set#518, <lambda>(result_set#518.dates.trackers[token]) AS result_set.dates.trackers.token#705]\n+- Relation[result_parameters#517,result_set#518] json\n"
数据结构:
-- result_set: struct (nullable = true)
| |-- currency: string (nullable = true)
| |-- dates: array (nullable = true)
| | |-- element: struct (containsNull = true)
| | | |-- date: string (nullable = true)
| | | |-- trackers: array (nullable = true)
| | | | |-- element: struct (containsNull = true)
| | | | | |-- countries: array (nullable = true)
| | | | | | |-- element: struct (containsNull = true)
| | | | | | | |-- country: string (nullable = true)
| | | | | | | |-- os_names: array (nullable = true)
| | | | | | | | |-- element: struct (containsNull = true)
| | | | | | | | | |-- kpi_values: array (nullable = true)
| | | | | | | | | | |-- element: double (containsNull = true)
| | | | | | | | | |-- os_name: string (nullable = true)
| | | | | |-- token: string (nullable = true)
| |-- name: string (nullable = true)
| |-- token: string (nullable = true)
我正在尝试创建一个视图来显示货币、日期和代币:
I am trying to create a view to show currency, date, and token:
df.select('result_set.currency', 'result_set.dates.date', 'result_set.dates.trackers.token').show()
数据样本:
"result_set": {
"token": "abcdef",
"name": "Facebook",
"currency": "EUR",
"dates": [
{
"date": "2020-03-11",
"trackers": [
{
"token": "12345",
"countries": [
{
"country": "am",
"os_names": [
{
"os_name": "android",
"kpi_values": [
0,
0,
0,
0,
0,
0,
1,
0,
0
]
}
]
},
我正在尝试根据 json 数据中的一些级别创建一个视图.
I am trying to create a view based on a few of these levels within the json data.
更新:
复制令牌
df.selectExpr('result_set.currency','explode(result_set.dates)').\
select("*","col.*").\
selectExpr("explode(trackers)","*").\
selectExpr("currency","date","explode(trackers)").\
select("currency","date","col.*").\
selectExpr("currency","date","token", "explode(countries)").\
select("currency","date","token", "col.*").\
selectExpr("currency","date","token", "country", "explode(os_names)").\
select("currency","date","token", "country", "col.*").\
selectExpr("currency","date","token", "country", "os_name", "explode(kpi_values)").\
show(20)
在进行了一些爆炸后，现在令牌重复了 8 次.
After doing some explodes, now the token repeats 8 times.
推荐答案
当您访问结构数组时，我们需要从 array<中给出哪个元素/strong>我们需要访问即0,1,2..等
- 如果我们需要选择数组的所有元素，那么我们需要使用
explode().
示例:
df.printSchema()
#root
# |-- result_set: struct (nullable = true)
# | |-- currency: string (nullable = true)
# | |-- dates: array (nullable = true)
# | | |-- element: struct (containsNull = true)
# | | | |-- date: string (nullable = true)
# | | | |-- trackers: array (nullable = true)
# | | | | |-- element: struct (containsNull = true)
# | | | | | |-- countries: array (nullable = true)
# | | | | | | |-- element: struct (containsNull = true)
# | | | | | | | |-- country: string (nullable = true)
# | | | | | | | |-- os_names: array (nullable = true)
# | | | | | | | | |-- element: struct (containsNull = true)
# | | | | | | | | | |-- kpi_values: array (nullable = true)
# | | | | | | | | | | |-- element: long (containsNull = true)
# | | | | | | | | | |-- os_name: string (nullable = true)
# | | | | | |-- token: string (nullable = true)
# | |-- name: string (nullable = true)
# | |-- token: string (nullable = true)
#accessing token,date from array
df.selectExpr('result_set.dates.trackers[0].token','result_set.currency', 'result_set.dates.date').show()
#+--------------------------------------------------+--------+------------+
#|result_set.dates.trackers AS trackers#194[0].token|currency| date|
#+--------------------------------------------------+--------+------------+
#| [12345]| EUR|[2020-03-11]|
#+--------------------------------------------------+--------+------------+
#accessing first elements from dates, trackers array and extracting date,token values
df.selectExpr('result_set.dates[0].trackers[0].token as token','result_set.currency', 'result_set.dates[0].date as date').show()
#+-----+--------+----------+
#|token|currency| date|
#+-----+--------+----------+
#|12345| EUR|2020-03-11|
#+-----+--------+----------+
#if you need to select all elements of array then we need to explode the array and select the data
df.selectExpr('result_set.currency','explode(result_set.dates)').\
select("*","col.*").\
selectExpr("explode(trackers)","*").\
selectExpr("currency","date","explode(trackers)").\
select("currency","date","col.*").\
select("currency","date","token").\
show()
#+--------+----------+-----+
#|currency| date|token|
#+--------+----------+-----+
#| EUR|2020-03-11|12345|
#+--------+----------+-----+
这篇关于由于数据类型不匹配 PySpark 无法解析列的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
查看全文
