将RDD转换为列联表:Pyspark [英] Converting RDD to Contingency Table: Pyspark

问题描述

当前，我正在尝试将RDD转换为列队表，以便使用pyspark.ml.clustering.KMeans模块，它将数据帧作为输入.

Currently I am trying to convert an RDD to a contingency table in-order to use the pyspark.ml.clustering.KMeans module, which takes a dataframe as input.

当我执行myrdd.take(K)时，(其中K是某个数字)结构如下:

When I do myrdd.take(K),(where K is some number) the structure looks as follows:

[[u'user1'，('itm1'，3)，...，('itm2'，1)]，[u'user2'，('itm1'，7)，...， ('itm2'，4)]，...，[u'usern'，('itm2'，2)，...，('itm3'，10)]]

[[u'user1',('itm1',3),...,('itm2',1)], [u'user2',('itm1',7),..., ('itm2',4)],...,[u'usern',('itm2',2),...,('itm3',10)]]

其中每个列表都包含一个实体作为第一个元素，以及该实体以元组形式喜欢的所有项目及其计数的集合.

Where each list contains an entity as the first element and the set of all items and their counts that was liked by this entity in the form of tuple.

现在，我的目标是将以上内容转换为类似于以下列联表的火花DataFrame.

Now, my objective is to convert the above into a spark DataFrame that resembles the following contingency table.

+----------+------+----+-----+
|entity    |itm1  |itm2|itm3 |
+----------+------+----+-----+
|    user1 |     3|   1|    0|
|    user2 |     7|   4|    0|
|    usern |     0|   2|   10|
+----------+------+----+-----+

我使用了以下链接中引用的df.stat.crosstab方法:

I have used the df.stat.crosstab method as cited in the following link :

统计和Apache Spark中带有DataFrames的数学函数-4.交叉制表(列联表)

它几乎接近我想要的.

and it is almost close to what I want.

但是，如果像上面的元组那样还有一个计数字段，即('itm1',3)如何将该值 3 合并(或添加)到列联表(或实体-项目矩阵).

But if there is one more count field like in the above tuple i.e., ('itm1',3) how to incorporate (or add) this value 3 into the final result of the contingency table (or entity-item matrix).

当然，我将上述RDD的列表转换为矩阵并将其写为csv文件，然后以DataFrame的形式读回，这是很长的路要走的.

Of course, I take the long route by converting the above list of RDD into a matrix and write them as csv file and then read back as a DataFrame.

是否有使用DataFrame进行操作的更简单方法?

Is there a simpler way to do it using DataFrame ?

推荐答案

使用createDataFrame()方法将RDD转换为pyspark数据框.

Convert RDD to pyspark dataframe by using createDataFrame() method.

使用交叉表方法后，请使用show方法.请参考以下示例:

Use show method after using crosstab method. Please refer following example:

cf = train_predictions.crosstab("prediction","label_col")

以表格格式显示它:

cf.show()

输出:

+--------------------+----+----+
|prediction_label_col| 0.0| 1.0|
+--------------------+----+----+
|                 1.0| 752|1723|
|                 0.0|1830| 759|
+--------------------+----+----+

这篇关于将RDD转换为列联表:Pyspark的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！