如果Spark中的数据帧是不可变的，为什么我们可以使用withColumn()之类的操作对其进行修改? [英] If dataframes in Spark are immutable, why are we able to modify it with operations such as withColumn()?

问题描述

这可能是一个愚蠢的问题，源于我的无知.我已经在PySpark上工作了几周，并且没有太多的编程经验.

This is probably a stupid question originating from my ignorance. I have been working on PySpark for a few weeks now and do not have much programming experience to start with.

我的理解是，在Spark中，RDD，数据帧和数据集都是不可变的-再次理解，这意味着您无法更改数据.如果是这样，为什么我们可以使用withColumn()编辑数据框的现有列?

My understanding is that in Spark, RDDs, Dataframes, and Datasets are all immutable - which, again I understand, means you cannot change the data. If so, why are we able to edit a Dataframe's existing column using withColumn()?

推荐答案

按照Spark体系结构，DataFrame建立在本质上是不可变的RDD之上，因此数据框架本质上也是不可变的.

As per Spark Architecture DataFrame is built on top of RDDs which are immutable in nature, Hence Data frames are immutable in nature as well.

关于withColumn或与此相关的任何其他操作，当您对DataFrame应用此类操作时，它将生成一个新的数据框架，而不是更新现有的数据框架.

Regarding the withColumn or any other operation for that matter, when you apply such operations on DataFrames it will generate a new data frame instead of updating the existing data frame.

但是，当您使用动态类型的语言python时，您将覆盖先前引用的值.因此，当您执行以下语句时

However, When you are working with python which is dynamically typed language you overwrite the value of the previous reference. Hence when you are executing below statement

df = df.withColumn()

它将生成另一个数据帧，并将其分配给引用"df".

It will generate another dataframe and assign it to reference "df".

为了进行验证，可以使用rdd的id()方法获取数据框的唯一标识符.

In order to verify the same, you can use id() method of rdd to get the unique identifier of your dataframe.

df.rdd.id()

将为您的数据框提供唯一的标识符.

will give you unique identifier for your dataframe.

希望以上说明对您有帮助.

I hope the above explanation helps.

此致

Neeraj

这篇关于如果Spark中的数据帧是不可变的，为什么我们可以使用withColumn()之类的操作对其进行修改?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！