Mahout rowSimilarity [英] Mahout rowSimilarity

问题描述

我试图计算维基百科文件之间的行相似性。我有tf-idf向量，格式为 Key类：class org.apache.hadoop.io.Text值类：class org.apache.mahout.math.VectorWritable 。我从这里快速浏览文本分析： https://cwiki.apache.org/confluence/display/MAHOUT/Quick+tour+of+text+analysis+using+the+Mahout+command+line

我创建了一个mahout矩阵，如下所示：

  mahout rowid \ 
 -i wikipedia-vectors / tfidf-vectors / part-r-00000 
 -o wikipedia-matrix 
  

我得到了生成行和列的数字：

  vectors.RowIdJob：用4587604行写出矩阵和14121544列到维基百科矩阵/矩阵
  

矩阵格式为键class：class org.apache.hadoop.io.IntWritable Value Class：class org.apache.mahout.math.VectorWritable

我也有使用以下格式的 docIndex 文档：键类：class org.ap ache.hadoop.io.IntWritable Value Class：class org.apache.hadoop.io.Text

然后，当我运行rowsimilarity作业

  mahout rowsimilarity 
 -i wikipedia-matrix / matrix 
 -o wikipedia-similarity 
  - r 4587604 
  - 相似性类别SIMILARITY_COSINE 
 -m 50 
 -ess 
  

我收到以下错误：

  13/08/25 15:18:18信息mapred.JobClient：Task Id ：attempt_201308161435_0364_m_000001_1，状态：FAILED 
 java.lang.ClassCastException：org.apache.hadoop.io.Text无法转换为org.apache.mahout.math.VectorWritable $ b $在org.apache.mahout.math .hadoop.similarity.cooccurrence.RowSimilarityJob $ VectorNormMapper.map（RowSimilarityJob.java:183）
 at org.apache.hadoop.mapreduce.Mapper.run（Mapper.java:144）
 at org.apache .hadoop.mapred.MapTask.runNewMapper（MapTask.java:648）
 at org.apache.hadoop.mapred.MapTask.run（MapTask.java ：322）
 at org.apache.hadoop.mapred.Child $ 4.run（Child.java:266）
 at java.security.AccessController.doPrivileged（Native Method）$ b $ javax。 security.auth.Subject.doAs（Subject.java:396）
 at org.apache.hadoop.security.UserGroupInformation.doAs（UserGroupInformation.java:1278）
 at org.apache.hadoop.mapred。 Child.main（Child.java:260）
  

有人可以帮我解决这个错误吗？当输入矩阵的格式为时，我不确定这是 org.apache.hadoop.io.Text > Key class：class org.apache .hadoop.io.IntWritable Value Class：class org.apache.mahout.math.VectorWritable

非常感谢。

Best，
Dragan

解决方案

我使用以下命令：

  hadoop jar mahout-examples-0.9-SNAPSHOT.jar 
 org.apache.mahout.math.hadoop。 similarity.cooccurrence.RowSimilarityJob 
 -i / user / dmilchev / wikipedia-matrix / matrix 
 -o / user / dmilchev / wikipedia-similarity 
 -r 4587604 --similarityClassname SIMILARITY_COSINE -m 50  - ess 
  

并且我没有收到任何错误。

I am trying to compute row similarity between wikipedia documents. I have the tf-idf vectors in format Key class: class org.apache.hadoop.io.Text Value Class: class org.apache.mahout.math.VectorWritable. I am following the quick tour of text analysis from here: https://cwiki.apache.org/confluence/display/MAHOUT/Quick+tour+of+text+analysis+using+the+Mahout+command+line

I created a mahout matrix as follows:

mahout rowid \
   -i wikipedia-vectors/tfidf-vectors/part-r-00000
   -o wikipedia-matrix

I got the the number generated rows and columns:

vectors.RowIdJob: Wrote out matrix with 4587604 rows and 14121544 columns to wikipedia-matrix/matrix

The matrix is of format Key class: class org.apache.hadoop.io.IntWritable Value Class: class org.apache.mahout.math.VectorWritable

I also have a docIndex document with the following format: Key class: class org.apache.hadoop.io.IntWritable Value Class: class org.apache.hadoop.io.Text

Then when I run the rowsimilarity job

mahout rowsimilarity 
   -i wikipedia-matrix/matrix 
   -o wikipedia-similarity 
   -r 4587604
   --similarityClassname SIMILARITY_COSINE
   -m 50
   -ess

I am getting the following error:

13/08/25 15:18:18 INFO mapred.JobClient: Task Id : attempt_201308161435_0364_m_000001_1, Status : FAILED
java.lang.ClassCastException: org.apache.hadoop.io.Text cannot be cast to org.apache.mahout.math.VectorWritable
    at org.apache.mahout.math.hadoop.similarity.cooccurrence.RowSimilarityJob$VectorNormMapper.map(RowSimilarityJob.java:183)
    at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
    at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:648)
    at org.apache.hadoop.mapred.MapTask.run(MapTask.java:322)
    at org.apache.hadoop.mapred.Child$4.run(Child.java:266)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:396)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1278)
    at org.apache.hadoop.mapred.Child.main(Child.java:260)

Could someone please help me with the error? I am not sure from where is this org.apache.hadoop.io.Text when the input matrix is of format Key class: class org.apache.hadoop.io.IntWritable Value Class: class org.apache.mahout.math.VectorWritable

Thank you very much.

Best, Dragan

解决方案

I solved it using the following command:

hadoop jar mahout-examples-0.9-SNAPSHOT.jar 
org.apache.mahout.math.hadoop.similarity.cooccurrence.RowSimilarityJob 
-i /user/dmilchev/wikipedia-matrix/matrix 
-o /user/dmilchev/wikipedia-similarity 
-r 4587604 --similarityClassname SIMILARITY_COSINE -m 50 -ess 

and I did not get any error.

这篇关于Mahout rowSimilarity的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！