如何制作整数索引行? [英] How to make an integer index row?

问题描述

我有一个DataFrame:

I have a DataFrame:

+-----+--------+---------+
|  usn|log_type|item_code|
+-----+--------+---------+
|    0|      11|    I0938|
|  916|      19|    I0009|
|  916|      51|    I1097|
|  916|      19|    C0723|
|  916|      19|    I0010|
|  916|      19|    I0010|
|12331|      19|    C0117|
|12331|      19|    C0117|
|12331|      19|    I0009|
|12331|      19|    I0009|
|12331|      19|    I0010|
|12838|      19|    I1067|
|12838|      19|    I1067|
|12838|      19|    C1083|
|12838|      11|    B0250|
|12838|      19|    C1346|
+-----+--------+---------+

我想要独特的item_code并为每个item_code创建一个索引，如下所示:

And I want distinct item_code and make an index for each item_code like this:

+---------+------+
|item_code| numId|
+---------+------+
|    I0938|   0  |
|    I0009|   1  |
|    I1097|   2  |
|    C0723|   3  |
|    I0010|   4  |
|    C0117|   5  | 
|    I1067|   6  |
|    C1083|   7  |
|    B0250|   8  | 
|    C1346|   9  |
+---------+------+

我不使用monotonically_increasing_id，因为它返回了bigint.

I don't use monotonically_increasing_id because it returns a bigint.

推荐答案

使用monotanicallly_increasing_id仅保证数字在增加，不能保证起始数字和连续编号.如果要确保得到0、1、2、3，...，则可以使用RDD函数zipWithIndex().

Using monotanicallly_increasing_id only guarantees that the numbers are increasing, the starting number and consecutive numbering is not guaranteed. If you want to be sure to get 0,1,2,3,... you can use the RDD function zipWithIndex().

由于我对python和spark不太熟悉，因此下面的示例使用的是scala，但转换起来应该很容易.

Since I'm not too familiar with spark together with python, the below example is using scala but it should be easy to convert it.

val spark = SparkSession.builder.getOrCreate()
import spark.implicits._

val df = Seq("I0938","I0009","I1097","C0723","I0010","I0010",
    "C0117","C0117","I0009","I0009","I0010","I1067",
    "I1067","C1083","B0250","C1346")
  .toDF("item_code")

val df2 = df.distinct.rdd
  .map{case Row(item: String) => item}
  .zipWithIndex()
  .toDF("item_code", "numId")

哪个会给您所要求的结果:

Which will give you the requested result:

+---------+-----+
|item_code|numId|
+---------+-----+
|    I0010|    0|
|    I1067|    1|
|    C0117|    2|
|    I0009|    3|
|    I1097|    4|
|    C1083|    5|
|    I0938|    6|
|    C0723|    7|
|    B0250|    8|
|    C1346|    9|
+---------+-----+

这篇关于如何制作整数索引行?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！