我如何构建/运行这个简单的 Mahout 程序而不会出现异常? [英] How do I build/run this simple Mahout program without getting exceptions?
问题描述
我想运行我在 Mahout In Action 中找到的这段代码:
I would like to run this code which I found in Mahout In Action:
package org.help;
import java.io.IOException;
import java.util.ArrayList;
import java.util.List;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.FileSystem;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.SequenceFile;
import org.apache.hadoop.io.Text;
import org.apache.mahout.math.DenseVector;
import org.apache.mahout.math.NamedVector;
import org.apache.mahout.math.VectorWritable;
public class SeqPrep {
public static void main(String args[]) throws IOException{
List<NamedVector> apples = new ArrayList<NamedVector>();
NamedVector apple;
apple = new NamedVector(new DenseVector(new double[]{0.11, 510, 1}), "small round green apple");
apples.add(apple);
Configuration conf = new Configuration();
FileSystem fs = FileSystem.get(conf);
Path path = new Path("appledata/apples");
SequenceFile.Writer writer = new SequenceFile.Writer(fs, conf, path, Text.class, VectorWritable.class);
VectorWritable vec = new VectorWritable();
for(NamedVector vector : apples){
vec.set(vector);
writer.append(new Text(vector.getName()), vec);
}
writer.close();
SequenceFile.Reader reader = new SequenceFile.Reader(fs, new Path("appledata/apples"), conf);
Text key = new Text();
VectorWritable value = new VectorWritable();
while(reader.next(key, value)){
System.out.println(key.toString() + " , " + value.get().asFormatString());
}
reader.close();
}
}
我编译它:
$ javac -classpath :/usr/local/hadoop-1.0.3/hadoop-core-1.0.3.jar:/home/hduser/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT.jar:/home/hduser/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT-job.jar:/home/hduser/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT-sources.jar -d myjavac/ SeqPrep.java
我打它:
$ jar -cvf SeqPrep.jar -C myjavac/ .
现在我想在我的本地 hadoop 节点上运行它.我试过了:
Now I'd like to run it on my local hadoop node. I've tried:
hadoop jar SeqPrep.jar org.help.SeqPrep
但我明白了:
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/math/Vector
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
所以我尝试使用 libjars 参数:
So I tried using the libjars parameter:
$ hadoop jar SeqPrep.jar org.help.SeqPrep -libjars /home/hduser/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT.jar -libjars /home/hduser/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT-job.jar -libjars /home/hduser/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT-sources.jar -libjars /home/hduser/mahout/trunk/math/target/mahout-math-0.8-SNAPSHOT.jar -libjars /home/hduser/mahout/trunk/math/target/mahout-math-0.8-SNAPSHOT-sources.jar
遇到了同样的问题.我不知道还能尝试什么.
and got the same problem. I don't know what else to try.
我的最终目标是能够将 hadoop fs 上的 .csv 文件读取到稀疏矩阵中，然后将其乘以随机向量.
My eventual goal is to be able to read a .csv file on the hadoop fs into a sparse matrix and then multiply it by a random vector.
看起来 Razvan 得到了它(注意:请参阅下面的另一种不会干扰您的 hadoop 安装的方法).供参考:
edit: Looks like Razvan got it (note: see below for another way to do this that does not mess with your hadoop installation). For reference:
$ find /usr/local/hadoop-1.0.3/. |grep mah
/usr/local/hadoop-1.0.3/./lib/mahout-core-0.8-SNAPSHOT-tests.jar
/usr/local/hadoop-1.0.3/./lib/mahout-core-0.8-SNAPSHOT.jar
/usr/local/hadoop-1.0.3/./lib/mahout-core-0.8-SNAPSHOT-job.jar
/usr/local/hadoop-1.0.3/./lib/mahout-core-0.8-SNAPSHOT-sources.jar
/usr/local/hadoop-1.0.3/./lib/mahout-math-0.8-SNAPSHOT-sources.jar
/usr/local/hadoop-1.0.3/./lib/mahout-math-0.8-SNAPSHOT-tests.jar
/usr/local/hadoop-1.0.3/./lib/mahout-math-0.8-SNAPSHOT.jar
然后:
$hadoop jar SeqPrep.jar org.help.SeqPrep
small round green apple , small round green apple:{0:0.11,1:510.0,2:1.0}
我正在尝试在不将 mahout jar 复制到 hadoop lib/
edit: I'm trying to do this without copying the mahout jars into the hadoop lib/
$ rm /usr/local/hadoop-1.0.3/lib/mahout-*
当然还有:
hadoop jar SeqPrep.jar org.help.SeqPrep
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/mahout/math/Vector
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
Caused by: java.lang.ClassNotFoundException: org.apache.mahout.math.Vector
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
当我尝试 mahout 作业文件时:
and when I try the mahout job file:
$hadoop jar ~/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT-job.jar org.help.SeqPrep
Exception in thread "main" java.lang.ClassNotFoundException: org.help.SeqPrep
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
如果我尝试包含我制作的 .jar 文件:
If I try to include the .jar file I made:
$ hadoop jar ~/mahout/trunk/core/target/mahout-core-0.8-SNAPSHOT-job.jar SeqPrep.jar org.help.SeqPrep
Exception in thread "main" java.lang.ClassNotFoundException: SeqPrep.jar
显然我一次只能发送一个 jar 到 hadoop.这意味着我需要将我制作的类添加到 mahout 核心作业文件中:
edit: Apparently I can only send one jar at a time to hadoop. This means I need to add the class I made into the mahout core job file:
~/mahout/trunk/core/target$ cp mahout-core-0.8-SNAPSHOT-job.jar mahout-core-0.8-SNAPSHOT-job.jar_backup
~/mahout/trunk/core/target$ cp ~/workspace/seqprep/bin/org/help/SeqPrep.class .
~/mahout/trunk/core/target$ jar uf mahout-core-0.8-SNAPSHOT-job.jar SeqPrep.class
然后:
~/mahout/trunk/core/target$ hadoop jar mahout-core-0.8-SNAPSHOT-job.jar org.help.SeqPrep
Exception in thread "main" java.lang.ClassNotFoundException: org.help.SeqPrep
好的，现在我可以在不搞乱我的 hadoop 安装的情况下做到这一点.我在之前的编辑中错误地更新了 .jar.应该是:
edit: Ok, now I can do it without messing with my hadoop installation. I was updating the .jar wrong in that previous edit. It should be:
~/mahout/trunk/core/target$ jar uf mahout-core-0.8-SNAPSHOT-job.jar org/help/SeqPrep.class
然后:
~/mahout/trunk/core/target$ hadoop jar mahout-core-0.8-SNAPSHOT-job.jar org.help.SeqPrep
small round green apple , small round green apple:{0:0.11,1:510.0,2:1.0}
推荐答案
您需要使用 Mahout 提供的作业"JAR 文件.它打包了所有依赖项.您还需要将您的课程添加到其中.这就是所有 Mahout 示例的工作方式.您不应该将 Mahout jar 放在 Hadoop 库中，因为那样会在 Hadoop 中安装"过深的程序.
You need to use the "job" JAR file provided by Mahout. It packages up all the dependencies. You need to add your classes to it too. This is how all the Mahout examples work. You shouldn't put Mahout jars in the Hadoop lib since that sort of "installs" a program too deeply in Hadoop.
这篇关于我如何构建/运行这个简单的 Mahout 程序而不会出现异常?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
