使用 Word2VecModel.transform() 在地图功能中不起作用 [英] using Word2VecModel.transform() does not work in map function
问题描述
我使用 Spark 构建了一个 Word2Vec 模型并将其保存为模型.现在,我想在另一个代码中使用它作为离线模型.我已经加载了模型并用它来呈现一个词的向量(例如你好),它运行良好.但是,我需要在 RDD 中使用 map 调用很多词.
I have built a Word2Vec model using Spark and save it as a model. Now, I want to use it in another code as offline model. I have loaded the model and used it to present vector of a word (e.g. Hello) and it works well. But, I need to call it for many words in an RDD using map.
当我在 map 函数中调用 model.transform() 时,它抛出这个错误:
When I call model.transform() in a map function, it throws this error:
看来您正试图从广播中引用 SparkContext"例外:您似乎试图从广播变量、操作或转换中引用 SparkContext.SparkContext 只能在驱动程序上使用,不能在它运行在工作线程上的代码中使用.有关详细信息,请参阅 SPARK-5063.
"It appears that you are attempting to reference SparkContext from a broadcast " Exception: It appears that you are attempting to reference SparkContext from a broadcast variable, action, or transforamtion. SparkContext can only be used on the driver, not in code that it run on workers. For more information, see SPARK-5063.
代码:
from pyspark import SparkContext
from pyspark.mllib.feature import Word2Vec
from pyspark.mllib.feature import Word2VecModel
sc = SparkContext('local[4]',appName='Word2Vec')
model=Word2VecModel.load(sc, "word2vecModel")
x= model.transform("Hello")
print(x[0]) # it works fine and returns [0.234, 0.800,....]
y=sc.parallelize([['Hello'],['test']])
y.map(lambda w: model.transform(w[0])).collect() #it throws the error
非常感谢您的帮助.
推荐答案
这是预期的行为.与其他 MLlib
模型一样,Python 对象只是 Scala 模型的包装器,实际处理委托给它的 JVM 对应物.由于工作人员无法访问 Py4J 网关(请参阅如何从操作或转换中使用 Java/Scala 函数?)您无法从操作或转换中调用 Java/Scala 方法.
It is an expected behavior. Like other MLlib
models Python object is just a wrapper around Scala model and actual processing is delegated to its JVM counterpart. Since Py4J gateway is not accessible on workers (see How to use Java/Scala function from an action or a transformation?) you cannot call Java / Scala method from an action or transformation.
通常 MLlib 模型提供了一个辅助方法,可以直接在 RDD 上工作,但这里不是这种情况.Word2VecModel
提供了 getVectors
方法,它返回一个从单词到向量的映射,但不幸的是它是一个 JavaMap
所以它不能在转换中工作.你可以试试这样的:
Typically MLlib models provide a helper method which can work directly on RDDs but it is not the case here. Word2VecModel
provides getVectors
method which returns a map from words to vector but unfortunately it is a JavaMap
so it won't work inside transformation. You could try something like this:
from pyspark.mllib.linalg import DenseVector
vectors_ = model.getVectors() # py4j.java_collections.JavaMap
vectors = {k: DenseVector([x for x in vectors_.get(k)])
for k in vectors_.keys()}
获取 Python 字典,但速度会非常慢.另一种选择是以 Python 可以使用的形式将此对象转储到磁盘,但这需要对 Py4J 进行一些修改,最好避免这种情况.相反,让我们将模型作为 DataFrame 读取:
to get Python dictionary but it will be extremely slow. Another option is to dump this object to disk in a form that can be consumed by Python but it requires some tinkering with Py4J and it is better to avoid this. Instead lets read model as a DataFrame:
lookup = sqlContext.read.parquet("path_to_word2vec_model/data").alias("lookup")
我们将得到以下结构:
lookup.printSchema()
## root
## |-- word: string (nullable = true)
## |-- vector: array (nullable = true)
## | |-- element: float (containsNull = true)
可用于将单词映射到向量,例如通过 join
:
which can be used to map words to vectors for example through join
:
from pyspark.sql.functions import col
words = sc.parallelize([('hello', ), ('test', )]).toDF(["word"]).alias("words")
words.join(lookup, col("words.word") == col("lookup.word"))
## +-----+-----+--------------------+
## | word| word| vector|
## +-----+-----+--------------------+
## |hello|hello|[-0.030862354, -0...|
## | test| test|[-0.13154022, 0.2...|
## +-----+-----+--------------------+
如果数据适合驱动程序/工作程序内存,您可以尝试使用广播进行收集和映射:
If data fits into driver / worker memory you can try to collect and map with broadcast:
lookup_bd = sc.broadcast(lookup.rdd.collectAsMap())
rdd = sc.parallelize([['Hello'],['test']])
rdd.map(lambda ws: [lookup_bd.value.get(w) for w in ws])
这篇关于使用 Word2VecModel.transform() 在地图功能中不起作用的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!