线程“main"中的异常java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration [英] Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration

问题描述

我使用的是 Hadoop 1.0.3 和 HBase 0.94.22.我正在尝试运行映射器程序以从 Hbase 表中读取值并将它们输出到文件中.我收到以下错误:

I am using Hadoop 1.0.3 and HBase 0.94.22. I am trying to run a mapper program to read values from a Hbase table and output them to a file. I am getting the following error:

Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
    at java.lang.Class.forName0(Native Method)
    at java.lang.Class.forName(Class.java:340)
    at org.apache.hadoop.util.RunJar.main(RunJar.java:149)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.hbase.HBaseConfiguration
    at java.net.URLClassLoader$1.run(URLClassLoader.java:372)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass(URLClassLoader.java:360)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    at java.lang.ClassLoader.loadClass(ClassLoader.java:357)

代码如下

import java.io.IOException;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.client.Result;
import org.apache.hadoop.hbase.client.Scan;
import org.apache.hadoop.hbase.filter.FirstKeyOnlyFilter;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapreduce.TableMapReduceUtil;
import org.apache.hadoop.hbase.mapreduce.TableMapper;
import org.apache.hadoop.hbase.util.Bytes;
import org.apache.hadoop.io.IntWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Job;
import org.apache.hadoop.mapreduce.lib.output.FileOutputFormat;


    public class Test {

    static class TestMapper extends TableMapper<Text, IntWritable> {
        private static final IntWritable one = new IntWritable(1);

        public void map(ImmutableBytesWritable row, Result value, Context context) throws    IOException, InterruptedException
        {
            ImmutableBytesWritable userkey = new ImmutableBytesWritable(row.get(), 0 , Bytes.SIZEOF_INT);
            String key =Bytes.toString(userkey.get());
            context.write(new Text(key), one);

        }
    }


    public static void main(String[] args) throws Exception {

        HBaseConfiguration conf = new HBaseConfiguration();
        Job job = new Job(conf, "hbase_freqcounter");
        job.setJarByClass(Test.class);
        Scan scan = new Scan();

        FileOutputFormat.setOutputPath(job, new Path(args[0]));
        String columns = "data";
        scan.addFamily(Bytes.toBytes(columns));
        scan.setFilter(new FirstKeyOnlyFilter());
        TableMapReduceUtil.initTableMapperJob("test",scan, TestMapper.class, Text.class, IntWritable.class, job);
        job.setOutputKeyClass(Text.class);
        job.setOutputValueClass(IntWritable.class);
        System.exit(job.waitForCompletion(true)?0:1);

    }

}

我将上面的代码导出到一个 jar 文件中，然后在命令行上我使用下面的命令来运行上面的代码.

I get the above code exported to a jar file and on the command line I use the below command to run the above code.

hadoop jar/home/testdb.jar 测试

hadoop jar /home/testdb.jar test

其中 test 是映射器结果应写入的文件夹.

where test is the folder to which the mapper results should be written.

我检查了一些其他链接，例如 由以下原因引起:java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException 建议在类路径中包含 zookeeper 文件，但是在 eclipse 中创建项目时，我已经包含了 hbase 的 lib 目录中的 zookeeper 文件.我包含的文件是zookeeper-3.4.5.jar.Ans 也访问了这个链接 HBase - java.lang.NoClassDefFoundError in java，但我使用映射器类从 hbase 表而不是任何客户端 API 中获取值.我知道我在某个地方犯了一个错误，伙计们，你们能帮帮我吗??

I have checked a few other links like Caused by: java.lang.ClassNotFoundException: org.apache.zookeeper.KeeperException where it has been suggested to include the zookeeper file in the classpath, but while creating the project in eclipse I have already included zookeeper file from the lib directory of hbase. The file I have included is zookeeper-3.4.5.jar. Ans also visited this link too HBase - java.lang.NoClassDefFoundError in java , but I am using a mapper class to get the values from the hbase table not any client API. I know I am making a mistake somewhere, guys could you please help me out ??

我注意到另一个奇怪的事情，当我删除主函数中除了第一行HBaseConfiguration conf = new HBaseConfiguration();"之外的所有代码，然后将代码导出到jar文件并尝试编译jar文件作为 hadoop jar test.jar 我仍然得到同样的错误.看来要么我错误地定义了 conf 变量，要么我的环境有问题.

I have noted another strange thing, when I remove all of the code in the main function except the first line " HBaseConfiguration conf = new HBaseConfiguration();", then export the code to a jar file and try to compile the jar file as hadoop jar test.jar I still get the same error. It seems either I am defining the conf variable incorrectly or there is some issue with my environment.

推荐答案

我解决了这个问题，我没有在 hadoop-env.sh 文件中添加 hbase 类路径.下面是我为使工作正常而添加的内容.

I got the fix to the problem, I had not added the hbase classpath in the hadoop-env.sh file. Below is the one I added to make the job work.

$ export HADOOP_CLASSPATH=$HBASE_HOME/hbase-0.94.22.jar:
    $HBASE_HOME/hbase-0.94.22-test.jar:
    $HBASE_HOME/conf:
    ${HBASE_HOME}/lib/zookeeper-3.4.5.jar:
    ${HBASE_HOME}/lib/protobuf-java-2.4.0a.jar:
    ${HBASE_HOME}/lib/guava-11.0.2.jar

这篇关于线程“main"中的异常java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！