使用 Word2VecModel.transform() 在地图功能中不起作用 [英] using Word2VecModel.transform() does not work in map function

查看:19
本文介绍了使用 Word2VecModel.transform() 在地图功能中不起作用的处理方法,对大家解决问题具有一定的参考价值,需要的朋友们下面随着小编来一起学习吧!

问题描述

我使用 Spark 构建了一个 Word2Vec 模型并将其保存为模型.现在,我想在另一个代码中使用它作为离线模型.我已经加载了模型并用它来呈现一个词的向量(例如你好),它运行良好.但是,我需要在 RDD 中使用 map 调用很多词.

I have built a Word2Vec model using Spark and save it as a model. Now, I want to use it in another code as offline model. I have loaded the model and used it to present vector of a word (e.g. Hello) and it works well. But, I need to call it for many words in an RDD using map.

当我在 map 函数中调用 model.transform() 时,它抛出这个错误:

When I call model.transform() in a map function, it throws this error:

看来您正试图从广播中引用 SparkContext"例外:您似乎试图从广播变量、操作或转换中引用 SparkContext.SparkContext 只能在驱动程序上使用,不能在它运行在工作线程上的代码中使用.有关详细信息,请参阅 SPARK-5063.

"It appears that you are attempting to reference SparkContext from a broadcast " Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.

代码:

from pyspark import SparkContext
from pyspark.mllib.feature import Word2Vec
from pyspark.mllib.feature import Word2VecModel

sc = SparkContext('local[4]',appName='Word2Vec')

model=Word2VecModel.load(sc, "word2vecModel")

x= model.transform("Hello")
print(x[0]) # it works fine and returns [0.234, 0.800,....]

y=sc.parallelize([['Hello'],['test']])
y.map(lambda w: model.transform(w[0])).collect() #it throws the error

非常感谢您的帮助.

推荐答案

这是预期的行为.与其他 MLlib 模型一样,Python 对象只是 Scala 模型的包装器,实际处理委托给它的 JVM 对应物.由于工作人员无法访问 Py4J 网关(请参阅如何从操作或转换中使用 Java/Scala 函数?)您无法从操作或转换中调用 Java/Scala 方法.

It is an expected behavior. Like other MLlib models Python object is just a wrapper around Scala model and actual processing is delegated to its JVM counterpart. Since Py4J gateway is not accessible on workers (see How to use Java/Scala function from an action or a transformation?) you cannot call Java / Scala method from an action or transformation.

通常 MLlib 模型提供了一个辅助方法,可以直接在 RDD 上工作,但这里不是这种情况.Word2VecModel 提供了 getVectors 方法,它返回一个从单词到向量的映射,但不幸的是它是一个 JavaMap 所以它不能在转换中工作.你可以试试这样的:

Typically MLlib models provide a helper method which can work directly on RDDs but it is not the case here. Word2VecModel provides getVectors method which returns a map from words to vector but unfortunately it is a JavaMap so it won't work inside transformation. You could try something like this:

from pyspark.mllib.linalg import DenseVector

vectors_ = model.getVectors() # py4j.java_collections.JavaMap
vectors = {k: DenseVector([x for x in vectors_.get(k)])
    for k in vectors_.keys()}

获取 Python 字典,但速度会非常慢.另一种选择是以 Python 可以使用的形式将此对象转储到磁盘,但这需要对 Py4J 进行一些修改,最好避免这种情况.相反,让我们将模型作为 DataFrame 读取:

to get Python dictionary but it will be extremely slow. Another option is to dump this object to disk in a form that can be consumed by Python but it requires some tinkering with Py4J and it is better to avoid this. Instead lets read model as a DataFrame:

lookup = sqlContext.read.parquet("path_to_word2vec_model/data").alias("lookup")

我们将得到以下结构:

lookup.printSchema()
## root
## |-- word: string (nullable = true)
## |-- vector: array (nullable = true)
## |    |-- element: float (containsNull = true)

可用于将单词映射到向量,例如通过 join:

which can be used to map words to vectors for example through join:

from pyspark.sql.functions import col

words = sc.parallelize([('hello', ), ('test', )]).toDF(["word"]).alias("words")

words.join(lookup, col("words.word") == col("lookup.word"))

## +-----+-----+--------------------+
## | word| word|              vector|
## +-----+-----+--------------------+
## |hello|hello|[-0.030862354, -0...|
## | test| test|[-0.13154022, 0.2...|
## +-----+-----+--------------------+

如果数据适合驱动程序/工作程序内存,您可以尝试使用广播进行收集和映射:

If data fits into driver / worker memory you can try to collect and map with broadcast:

lookup_bd = sc.broadcast(lookup.rdd.collectAsMap())
rdd = sc.parallelize([['Hello'],['test']])
rdd.map(lambda ws: [lookup_bd.value.get(w) for w in ws])

这篇关于使用 Word2VecModel.transform() 在地图功能中不起作用的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!

查看全文
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆