为什么 createDataFrame 对列重新排序? [英] Why does createDataFrame reorder the columns?

问题描述

假设我正在从一个没有架构的列表中创建一个数据框:

Suppose I am creating a data frame from a list without a schema:

data = [Row(c=0, b=1, a=2), Row(c=10, b=11, a=12)]
df = spark.createDataFrame(data)
df.show()

+---+---+---+
|  a|  b|  c|
+---+---+---+
|  2|  1|  0|
| 12| 11| 10|
+---+---+---+

为什么列按字母顺序重新排序?
我可以在不添加架构的情况下保留列的原始顺序吗?

Why are the columns reordered in alphabet order ?
Can I preserve the original order of columns without adding a schema ?

推荐答案

为什么列按字母顺序重新排序?

Why are the columns reordered in alphabet order ?

因为 Row 是用 **kwargs 创建的 按名称对参数进行排序.

Because Row created with **kwargs sorts the arguments by name.

此设计选择是解决PEP 468.请查看 SPARK-12467 进行讨论.

This design choice is required to address the issues described in PEP 468. Please check SPARK-12467 for a discussion.

我可以在不添加架构的情况下保留列的原始顺序吗?

Can I preserve the original order of columns without adding a schema ?

不适用于 **kwargs.你可以使用普通的元组:

Not with **kwargs. You can use plain tuples:

df = spark.createDataFrame([(0, 1, 2), (10, 11, 12)], ["c", "b", "a"])

或 namedtuple:

from collections import namedtuple

CBA = namedtuple("CBA", ["c", "b", "a"])
spark.createDataFrame([CBA(0, 1, 2), CBA(10, 11, 12)])

这篇关于为什么 createDataFrame 对列重新排序?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！