如何对自定义RecordReader和InputFormat类进行单元测试？ [英] How to do unit testing of custom RecordReader and InputFormat classes?

问题描述

我开发了一个map-reduce程序。我编写了自定义的 RecordReader 和 InputFormat 类。

我为单元使用 MR单元和 Mockito 测试mapper和reducer。

我想知道如何单元测试自定义 RecordReader 和 InputFormat 类？什么是测试这些类最受欢迎的方式？

解决方案

感谢 user7610

已编译且经过测试的 answer

  import org.apache.hadoop.conf.Configuration; 
导入org.apache.hadoop.fs.Path; 
 import org.apache.hadoop.mapreduce.InputFormat; 
 import org.apache.hadoop.mapreduce.RecordReader; 
 import org.apache.hadoop.mapreduce.TaskAttemptContext; 
 import org.apache.hadoop.mapreduce.TaskAttemptID; 
 import org.apache.hadoop.mapreduce.lib.input.FileSplit; 
 import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl; 
 import org.apache.hadoop.util.ReflectionUtils; 
 import java.io.File; 
 
配置conf = new配置（false）; 
 conf.set（fs.default.name，file：///）; 
 
文件testFile = new File（path / to / file）; 
 Path path = new Path（testFile.getAbsoluteFile（）。toURI（））; 
 FileSplit split = new FileSplit（path，0，testFile.length（），null）; 
 
 InputFormat inputFormat = ReflectionUtils.newInstance（MyInputFormat.class，conf）; 
 TaskAttemptContext context = new TaskAttemptContextImpl（conf，new TaskAttemptID（））; 
 RecordReader reader = inputFormat.createRecordReader（split，context）; 
 
 reader.initialize（split，context）; 
  

I have developed one map-reduce program. I have written custom RecordReader and InputFormat classes.

I am using MR Unit and Mockito for unit testing of mapper and reducer.

I would like to know how to unit test custom RecordReader and InputFormat classes? What is the most preferred way to test these classes?

解决方案

thanks to user7610

compiled and somewhat tested version of the example code from the answer

import org.apache.hadoop.conf.Configuration;
import org.apache.hadoop.fs.Path;
import org.apache.hadoop.mapreduce.InputFormat;
import org.apache.hadoop.mapreduce.RecordReader;
import org.apache.hadoop.mapreduce.TaskAttemptContext;
import org.apache.hadoop.mapreduce.TaskAttemptID;
import org.apache.hadoop.mapreduce.lib.input.FileSplit;
import org.apache.hadoop.mapreduce.task.TaskAttemptContextImpl;
import org.apache.hadoop.util.ReflectionUtils;
import java.io.File;

Configuration conf = new Configuration(false);
conf.set("fs.default.name", "file:///");

File testFile = new File("path/to/file");
Path path = new Path(testFile.getAbsoluteFile().toURI());
FileSplit split = new FileSplit(path, 0, testFile.length(), null);

InputFormat inputFormat = ReflectionUtils.newInstance(MyInputFormat.class, conf);
TaskAttemptContext context = new TaskAttemptContextImpl(conf, new TaskAttemptID());
RecordReader reader = inputFormat.createRecordReader(split, context);

reader.initialize(split, context);

这篇关于如何对自定义RecordReader和InputFormat类进行单元测试？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！