找不到hbase.mapreduce.TableOutputFormat [英] hbase.mapreduce.TableOutputFormat not found

问题描述

我正在使用hdp 2.6和hbase1.1.2.

I'm using hdp 2.6 and hbase1.1.2.

当我从不在hdp群集中的服务器提交映射减少作业时，
我遇到以下异常:

When I submit a map-reduce job from a server which is not in the hdp cluster,
I got the following exception:

Class org.apache.hadoop.hbase.mapreduce.TableOutputFormat not found.

我的映射器在本地作业模式下正确完成，并且我确实确定此类在我的类路径中，并且群集中的lib也有此jar.

My mapper finished correctly under local job mode and I'm really sure that this class is in my classpath and lib in cluster has this jar either.

我用Google搜索了其他人执行的几个步骤:
1.使用-libjars
2.将hbase_classpath添加到hadoop.env并重新启动集群
3.将hbase-master/lib添加到yarn-site.xml并重新启动集群
但是这些不起作用.请帮助我，因为我只能使用Mr来完成此任务.

I googled several steps that others did:
1. use -libjars
2. add hbase_classpath to hadoop.env and restart the cluster
3. add hbase-master/lib to yarn-site.xml and restart the cluster
But these don't work. Plz help me cause I can only use mr to do this task.

这是我的代码:

package com.test.service;

import org.apache.hadoop.hbase.HBaseConfiguration;
import org.apache.hadoop.hbase.mapreduce.ImportTsv;
import org.apache.hadoop.security.UserGroupInformation;
import org.apache.hadoop.util.ToolRunner;
import org.springframework.beans.factory.annotation.Value;
import org.springframework.stereotype.Service;

@Service
public class Tsv2HBaseService {
    private org.apache.hadoop.conf.Configuration  config;
    private String principle;
    private String keytab;

    Tsv2HBaseService(@Value("${hbase.kerberos.REALM}") String principle,
                     @Value("${hbase.kerberos.keytab}") String keytab) {
        this.config = HBaseConfiguration.create();
        this.principle = principle;
        this.keytab = keytab;
    }

    public int importMultiKeyWithTableTSV(String table, String tsvfile, String className) throws Exception {
        String args[] = {
                "-Dimporttsv.mapper.class=" + className,
                "-DtableStructure.file=" + tsvfile.replace(".csv", ".xml"),
                table,
                tsvfile,
                "-libjars ./hbase-client-1.1.2.jar,./hbase-server-1.1.2.jar,./hbase-protocol-1.1.2.jar,./hbase-common-1.1.2.jar"
        };

        config.set("hadoop.security.authentication" , "Kerberos");
        UserGroupInformation.setConfiguration(this.config);
        UserGroupInformation.loginUserFromKeytab(this.principle, this.keytab);
        return ToolRunner.run(config, new ImportTsv(), args);
    }

}

package com.test.web;
import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.hbase.KeyValue;
import org.apache.hadoop.hbase.client.Put;
import org.apache.hadoop.hbase.io.ImmutableBytesWritable;
import org.apache.hadoop.hbase.mapreduce.CellCreator;
import org.apache.hadoop.hbase.util.Base64;
import org.apache.hadoop.io.LongWritable;
import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapreduce.Mapper;
import org.slf4j.Logger;
import org.slf4j.LoggerFactory;

import java.io.IOException;

public class GRZHXXTsvImporterMapper extends Mapper<LongWritable, Text, ImmutableBytesWritable, Put> {

    private static final Logger logger = LoggerFactory.getLogger(Tsv2HbaseController.class);
    @Override
    public void map(LongWritable offset, Text value, Mapper<LongWritable, Text, ImmutableBytesWritable, Put>.Context context)
            throws IOException {
        try {
            String[] row = value.toString().split(",");
            ImmutableBytesWritable rowKey = new ImmutableBytesWritable((row[0]+row[1]+row[2]).getBytes());
            Put put = new Put(rowKey.copyBytes());
            KeyValue kv3 = new KeyValue(rowKey.copyBytes(), "PUBLIC".getBytes(),"GRJCJS".getBytes(), row[3].getBytes());
            put.add(kv3);
            // Write user table put
            context.write(rowKey, put);

        } catch (Exception e) {
            e.printStackTrace();
        }
    }
}

hadoop-env.sh:

...lots of settings...
HADOOP_CLASSPATH=${HADOOP_CLASSPATH}{JAVA_JDBC_LIBS}${HBASE_CLASSPATH}
...lots of settings...

yarn-site.xml:

...lots of settings...
yarn.application.classpath=.......,/usr/hdp/current/hbase-master/lib/*
...lots of settings...

推荐答案

最后我找到了解决方案.

Finally I found the solution.

将hbase-master添加到下面的类路径中:yarn-site.xmlmapred-site.xml

Add hbase-master to classpath below: yarn-site.xml mapred-site.xml

应该同时在集群和您自己的jar资源中修改xml.

both xml should be modified in cluster and your own jar resources.

这篇关于找不到hbase.mapreduce.TableOutputFormat的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！