如何获得倒排索引? [英] How can I get inverted index?

问题描述

我是新来的Spark.如何通过使用Spark获取csv文件的反向索引?我有CSV文件

I'new using Spark. How can I get inverted index for csv file by using Spark? I have csv file

df.show()
+--------+--------------------+--------------------+----------+
|      id|               title|                tags|closeddate|
+--------+--------------------+--------------------+----------+
|48702270|                null|                null|      null|
|48702455|                null|                null|      null|
|48702469|                null|                null|      null|
|48704166|                null|                null|      null|
|48704172|                null|                null|      null|
|48704243|Transfer files fr...|<java><android><f...|      null|
|48704263|for each loops as...|<java><java-8><mu...|      null|

如何获取显示顶部标签的倒排索引作为示例?

How can I get the inverted index that shows top tags as an example?

推荐答案

首先，您需要将标签列转换为数组，如下所示:

First you need to convert the tags column to array, like this:

from pyspark.sql.types import *
def read_tags_raw(tags_string): 
    return tags_string.strip('>').strip('<').split('><') if tags_string else []

read_tags = udf(read_tags_raw, ArrayType(StringType()))
df_valid_tags = df.withColumn('tags', read_tags('tags'))

然后您调用explode并按标签分组以创建倒排索引:

Then you call explode and group by tag to create inverted index:

df_valid_tags.select(explode('tags').alias('tag'), 'id') \
    .groupBy('tag').agg(collect_list('id').alias('ids'))

祝您使用SO数据库好运；-)

Good luck working with SO database ;-)

这篇关于如何获得倒排索引?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！