Hadoop:java.lang.Exception:java.lang.RuntimeException:配置对象时出错 [英] Hadoop: java.lang.Exception: java.lang.RuntimeException: Error in configuring object

问题描述

首先感谢您的帮助.在地图类中，我实例化了另一个类WebPageToText.我的第一个问题:在Hadoop中运行代码时，map类中的打印内容是否会出现?第二个问题:请帮助我解决此错误.

Thanks for your help first. In the map class, I instantiated another class WebPageToText. My first question: will the print in the map class appear when running the code in Hadoop? 2nd question: Please help me with this error.

我一直遇到这个问题:

    14/04/02 20:39:35 INFO util.NativeCodeLoader: Loaded the native-hadoop library
    14/04/02 20:39:36 WARN snappy.LoadSnappy: Snappy native library is available
    14/04/02 20:39:36 INFO snappy.LoadSnappy: Snappy native library loaded
    14/04/02 20:39:36 INFO mapred.FileInputFormat: Total input paths to process : 1
    14/04/02 20:39:36 INFO mapred.JobClient: Running job: job_local1947041074_0001
    14/04/02 20:39:36 INFO mapred.LocalJobRunner: Waiting for map tasks
    14/04/02 20:39:36 INFO mapred.LocalJobRunner: Starting task: attempt_local1947041074_0001_m_000000_0
    14/04/02 20:39:36 INFO util.ProcessTree: setsid exited with exit code 0
    14/04/02 20:39:36 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@1cf7491
    14/04/02 20:39:36 INFO mapred.MapTask: Processing split: file:/usr/local/hadoop/project/input1/url.txt:0+68
    14/04/02 20:39:36 INFO mapred.MapTask: numReduceTasks: 1
    14/04/02 20:39:36 INFO mapred.MapTask: io.sort.mb = 100
    14/04/02 20:39:36 INFO mapred.MapTask: data buffer = 79691776/99614720
    14/04/02 20:39:36 INFO mapred.MapTask: record buffer = 262144/327680
    14/04/02 20:39:36 INFO mapred.LocalJobRunner: Map task executor complete.
    14/04/02 20:39:36 WARN mapred.LocalJobRunner: job_local1947041074_0001
    java.lang.Exception: java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:354)
    Caused by: java.lang.RuntimeException: Error in configuring object
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:93)
        at org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:64)
        at org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:117)
        at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:426)
        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:366)
        at org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:223)
        at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
        at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
        at java.util.concurrent.FutureTask.run(FutureTask.java:166)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1146)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
        at java.lang.Thread.run(Thread.java:701)
    Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:622)
        at org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:88)
        ... 11 more
    Caused by: java.lang.NoClassDefFoundError: de/l3s/boilerpipe/BoilerpipeProcessingException
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:270)
        at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:810)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:855)
        at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:881)
        at org.apache.hadoop.mapred.JobConf.getMapperClass(JobConf.java:968)
        at org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:34)
        ... 16 more
    Caused by: java.lang.ClassNotFoundException: de.l3s.boilerpipe.BoilerpipeProcessingException
        at java.net.URLClassLoader$1.run(URLClassLoader.java:217)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:205)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:323)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:268)
        ... 23 more
    14/04/02 20:39:37 INFO mapred.JobClient:  map 0% reduce 0%
    14/04/02 20:39:37 INFO mapred.JobClient: Job complete: job_local1947041074_0001
    14/04/02 20:39:37 INFO mapred.JobClient: Counters: 0
    14/04/02 20:39:37 INFO mapred.JobClient: Job Failed: NA
    Exception in thread "main" java.io.IOException: Job failed!
        at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:1357)
        at webPageToTxt.ConfMain.run(ConfMain.java:33)
        at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
        at webPageToTxt.ConfMain.main(ConfMain.java:40)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:622)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:160)

conf类:软件包webPageToTxt;

The conf class: package webPageToTxt;

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.conf.Configured;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.io.*;
import org.apache.hadoop.mapred.*;
import org.apache.hadoop.util.Tool;
import org.apache.hadoop.util.ToolRunner;


public class ConfMain extends Configured implements Tool{

    public int run(String[] args) throws Exception
    {

          //creating a JobConf object and assigning a job name for identification purposes
          JobConf conf = new JobConf(getConf(), ConfMain.class);
          conf.setJobName("webpage to txt");

          //Setting configuration object with the Data Type of output Key and Value
          conf.setOutputKeyClass(Text.class);
          conf.setOutputValueClass(Text.class);

          //Providing the mapper and reducer class names
          conf.setMapperClass(WebPageToTxtMapper.class);
          conf.setReducerClass(WebPageToTxtReducer.class);

          //the hdfs input and output directory to be fetched from the command line
          FileInputFormat.addInputPath(conf, new Path(args[0]));
          FileOutputFormat.setOutputPath(conf, new Path(args[1]));

          JobClient.runJob(conf);
          System.out.println("configuration is done");
          return 0;
    }


    public static void main(String[] args) throws Exception {
         int res = ToolRunner.run(new Configuration(), new ConfMain(),args);
         System.exit(res);
    }
}

地图类

package webPageToTxt;

import java.io.IOException;
import java.util.Scanner;

import org.apache.hadoop.io.Text;
import org.apache.hadoop.mapred.MapReduceBase;
import org.apache.hadoop.mapred.Mapper;
import org.apache.hadoop.mapred.OutputCollector;
import org.apache.hadoop.mapred.Reporter;

import de.l3s.boilerpipe.BoilerpipeProcessingException;


public class WebPageToTxtMapper extends MapReduceBase implements Mapper<Text, Text, Text, Text>
{
         private Text url = new Text();
         private Text wordList = new Text();
      //map method that performs the tokenizer job and framing the initial key value pairs
      public void map(Text key, Text value, OutputCollector<Text, Text> output, Reporter reporter) throws IOException
      {
            try {
                System.out.println("Prepare to get into webpage");
//              String val = WebPageToTxt.webPageToTxt("http://en.wikipedia.org/wiki/Sun\nhttp://en.wikipedia.org/wiki/Earth");

                String val = WebPageToTxt.webPageToTxt(value.toString());
                System.out.println("Webpage main function implemented");

                Scanner scanner = new Scanner(val);
                while (scanner.hasNextLine()) {
                    String line = scanner.nextLine();
                      // process the line
                    String[] arr = line.split("`", 2);
                    url.set(arr[0]);
                    wordList.set(line);
                    output.collect(url, wordList);
                }
                scanner.close();
            } catch (BoilerpipeProcessingException e) {
                // TODO Auto-generated catch block
                e.printStackTrace();
            }
      }
}

推荐答案

映射器或reduce方法中的打印语句将不会出现在主作业输出中，要使用Jobtracker webUI来查看输出

Print statements in mapper or reduce method will not appear in main job output, for viewing the output use Jobtracker webUI

从Jobid列中选择正确的Jobid ->单击Kind列中的 Map/Reduce ->单击各自Map/Reduce 任务->单击 All Tasklog Clolumn

Choose the correct Jobid from Jobid Column -> Click on Map/Reduce from Kind Column -> Click on respective Map/Reduce task -> Click on All Tasklog clolumn

由于在mapper方法中使用的是de.l3s.boilerpipe.BoilerpipeProcessingException类，因此需要以分布式方式使该类可用，为此，如果使用hadoop命令执行该应用程序，请使用-libjars Generic选项(需要实现ToolRunner类)或仅打包包含类 de.l3s.boilerpipe.BoilerpipeProcessingException 的jar以及Main jar本身.

Since you are using the class de.l3s.boilerpipe.BoilerpipeProcessingException in mapper method you need to make this class available in distributed manner, for that if you are exeuting this application using hadoop command make use of -libjars Generic option (Need to implement ToolRunner Class) or Simply pack the jar containing the class de.l3s.boilerpipe.BoilerpipeProcessingException along with the Main jar itself.

这篇关于Hadoop:java.lang.Exception:java.lang.RuntimeException:配置对象时出错的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！