使用Word2VecModel.transform（）不会在地图功能工作 [英] using Word2VecModel.transform() does not work in map function

问题描述

我已经建立了使用引发Word2Vec模型并将其保存为一个模型。现在，我想在另一个code为脱机模式中使用它。我已经加载了模型，并将其用于一个字（如你好）的present载体，它工作得很好。但是，我需要调用它在使用RDD地图很多的话。

I have built a Word2Vec model using Spark and save it as a model. Now, I want to use it in another code as offline model. I have loaded the model and used it to present vector of a word (e.g. Hello) and it works well. But, I need to call it for many words in an RDD using map.

当我打电话model.transform（）的地图功能，它抛出这个错误：

When I call model.transform() in a map function, it throws this error:

看来，你正试图从广播中引用SparkContext
  例外：看来，你正试图从广播的变量，动作或transforamtion参考SparkContext。 SparkContext只能在驱动程序中使用，而不是在code它工人运行。欲了解更多信息，请参阅SPARK-5063。

"It appears that you are attempting to reference SparkContext from a broadcast " Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

在code：

from pyspark import SparkContext
from pyspark.mllib.feature import Word2Vec
from pyspark.mllib.feature import Word2VecModel

sc = SparkContext('local[4]',appName='Word2Vec')

model=Word2VecModel.load(sc, "word2vecModel")

x= model.transform("Hello")
print(x[0]) # it works fine and returns [0.234, 0.800,....]

y=sc.parallelize([['Hello'],['test']])
y.map(lambda w: model.transform(w[0])).collect() #it throws the error

我真的会AP preciate你的帮助。

I will really appreciate your help.

推荐答案

这是一个预期的行为。像其他 MLlib 模型Python对象就位于斯卡拉模型的包装和实际加工委托给它的JVM对应。由于Py4J网关不在员工都可以访问（见如何使用Java / Scala的功能，从一个动作或一个转型？）你不能从操作或改造调用Java / Scala的方法。

It is an expected behavior. Like other MLlib models Python object is just a wrapper around Scala model and actual processing is delegated to its JVM counterpart. Since Py4J gateway is not accessible on workers (see How to use Java/Scala function from an action or a transformation?) you cannot call Java / Scala method from an action or transformation.

通常MLlib模型提供了一个辅助方法，它可以直接在RDDS工作，但它在这里不是这样的。 Word2VecModel 提供返回的话矢量地图，但不幸的是它是一个 JavaMap getVectors 方法，所以它不会转变里面工作。你可以尝试这样的事：

Typically MLlib models provide a helper method which can work directly on RDDs but it is not the case here. Word2VecModel provides getVectors method which returns a map from words to vector but unfortunately it is a JavaMap so it won't work inside transformation. You could try something like this:

from pyspark.mllib.linalg import DenseVector

vectors_ = model.getVectors() # py4j.java_collections.JavaMap
vectors = {k: DenseVector([x for x in vectors_.get(k)])
    for k in vectors_.keys()}

让Python字典，但它会非常慢。另一种选择是转储这个对象到磁盘上可以由Python的消费形式，但它需要一些修修补补Py4J，这是更好地避免这种情况。反而让读取模型作为数据框：

to get Python dictionary but it will be extremely slow. Another option is to dump this object to disk in a form that can be consumed by Python but it requires some tinkering with Py4J and it is better to avoid this. Instead lets read model as a DataFrame:

lookup = sqlContext.read.parquet("path_to_word2vec_model/data").alias("lookup")

，我们会得到以下结构：

and we'll get a following structure:

lookup.printSchema()
## root
## |-- word: string (nullable = true)
## |-- vector: array (nullable = true)
## |    |-- element: float (containsNull = true)

可用于映射的话，例如向量通过加入：

from pyspark.sql.functions import col

words = sc.parallelize([('hello', ), ('test', )]).toDF(["word"]).alias("words")

words.join(lookup, col("words.word") == col("lookup.word"))

## +-----+-----+--------------------+
## | word| word|              vector|
## +-----+-----+--------------------+
## |hello|hello|[-0.030862354, -0...|
## | test| test|[-0.13154022, 0.2...|
## +-----+-----+--------------------+

如果数据适合驱动器/工人的内存，你可以尝试来收集和广播图：

If data fits into driver / worker memory you can try to collect and map with broadcast:

lookup_bd = sc.broadcast(lookup.rdd.collectAsMap())
rdd = sc.parallelize([['Hello'],['test']])
rdd.map(lambda ws: [lookup_bd.value.get(w) for w in ws])

这篇关于使用Word2VecModel.transform（）不会在地图功能工作的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！