如何在hadoop中为java使用combineFileInputFormat？ [英] How to use combineFileInputFormat for java in hadoop?

问题描述

我正在处理大量小文件，并使用combineFileInputFormat来避免执行大量映射任务。它工作正常，但问题是，我需要有每个文件的ID，这样所有的文件的ID是1.我怎么能保持文件的ID？我必须添加一些代码到我的combineFileInputFormat？ / p>

解决方案

配置当前正在处理的文件的配置属性，所以你应该能够从 map.input.file 属性获取当前文件名：

  @Override 
 protected void map（Longwritable key，Text value，Context context）{
 String filePath = context.getConfiguration（）。get（map.input.file ）; 
 
 
 
 
 
如果你使用旧的API（mapred），属性名称是相同。
 
I am working with large number of small files and have used combineFileInputFormat to avoid having a lot of map tasks. it is working properly but the problem is that i need to have the id of each file, in this way all of file's id is 1. how can i keep the ids of files?do i have to add some code to my combineFileInputFormat?
解决方案
The CombineFileRecordReader configures a configuration property for the current file it is processing, so you should be able to obtain the current file name from the map.input.file property:
@Override
protected void map(Longwritable key, Text value, Context context) {
    String filePath = context.getConfiguration().get("map.input.file");
}
If you're using the old API (mapred), the property name is the same.

                        
这篇关于如何在hadoop中为java使用combineFileInputFormat？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！