如何保留 Spark HashingTF() 函数的输入键或索引? [英] how do I preserve the key or index of input to Spark HashingTF() function?

问题描述

基于 1.4 的 Spark 文档(https://spark.apache.org/docs/1.4.0/mllib-feature-extraction.html) 我正在编写一个 TF-IDF 示例，用于将文本文档转换为值向量.给出的示例显示了如何做到这一点，但输入是一个 RDD，没有键.这意味着我的输出 RDD 不再包含索引或键来引用原始文档.例子是这样的:

Based on the Spark documentation for 1.4 (https://spark.apache.org/docs/1.4.0/mllib-feature-extraction.html) I'm writing a TF-IDF example for converting text documents to vectors of values. The example given shows how this can be done but the input is a RDD of tokens with no keys. This means that my output RDD no longer contains an index or key to refer back to the original document. The example is this:

documents = sc.textFile("...").map(lambda line: line.split(" "))

hashingTF = HashingTF()
tf = hashingTF.transform(documents)

我想做这样的事情:

documents = sc.textFile("...").map(lambda line: (UNIQUE_LINE_KEY, line.split(" ")))

hashingTF = HashingTF()
tf = hashingTF.transform(documents)

并使结果 tf 变量在某处包含 UNIQUE_LINE_KEY 值.我只是错过了一些明显的东西吗?从示例来看，似乎没有什么好方法可以将 document RDD 与 tf RDD 链接起来.

and have the resulting tf variable contain the UNIQUE_LINE_KEY value somewhere. Am I just missing something obvious? From the examples it appears there is no good way to link the document RDD with the tf RDD.

推荐答案

如果您使用提交后的 Spark 版本 85b96372cf0fd055f89fc639f45c1f2cb02a378f(包括 1.4)，并使用 ml API HashingTF(需要DataFrame 输入而不是普通的 RDD)，其输出中的原始列.希望有帮助！

If you use a version of Spark from after commit 85b96372cf0fd055f89fc639f45c1f2cb02a378f (this includes the 1.4), and use the ml API HashingTF (requires DataFrame input instead of plain RDDs), the original columns in its output. Hope that helps!

这篇关于如何保留 Spark HashingTF() 函数的输入键或索引?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！