Spark Dataframe :How to add a index Column : Aka Distributed Data Index [英] Spark Dataframe :How to add a index Column : Aka Distributed Data Index

问题描述

我从 csv 文件中读取数据，但没有索引.

I read data from a csv file ,but don't have index.

我想从 1 添加一列到行号.

I want to add a column from 1 to row's number.

我该怎么办，谢谢(scala)

What should I do,Thanks (scala)

推荐答案

使用 Scala，您可以使用:

With Scala you can use:

import org.apache.spark.sql.functions._ 

df.withColumn("id",monotonicallyIncreasingId)

你可以参考这个示例 和 Scala 文档.

You can refer to this exemple and scala docs.

使用 Pyspark，您可以使用:

With Pyspark you can use:

from pyspark.sql.functions import monotonically_increasing_id 

df_index = df.select("*").withColumn("id", monotonically_increasing_id())

这篇关于Spark Dataframe :How to add a index Column : Aka Distributed Data Index的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！