在派生自其他列的数据框中添加新列 (Spark) [英] Adding a new column in Data Frame derived from other columns (Spark)

问题描述

我使用的是 Spark 1.3.0 和 Python.我有一个数据框，我希望添加一个从其他列派生的附加列.像这样，

I'm using Spark 1.3.0 and Python. I have a dataframe and I wish to add an additional column which is derived from other columns. Like this,

>>old_df.columns
[col_1, col_2, ..., col_m]

>>new_df.columns
[col_1, col_2, ..., col_m, col_n]

哪里

col_n = col_3 - col_4

如何在 PySpark 中执行此操作?

How do I do this in PySpark?

推荐答案

一种方法是使用 withColumn 方法:

One way to achieve that is to use withColumn method:

old_df = sqlContext.createDataFrame(sc.parallelize(
    [(0, 1), (1, 3), (2, 5)]), ('col_1', 'col_2'))

new_df = old_df.withColumn('col_n', old_df.col_1 - old_df.col_2)

或者，您可以在已注册的表上使用 SQL:

Alternatively you can use SQL on a registered table:

old_df.registerTempTable('old_df')
new_df = sqlContext.sql('SELECT *, col_1 - col_2 AS col_n FROM old_df')

这篇关于在派生自其他列的数据框中添加新列 (Spark)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！