哪些文件作为输入映射器忽略？ [英] which files ignored as input by mapper?

问题描述

我正在链接多个MapReduce作业，并希望传递/存储一些元信息（例如原始输入的配置或名称）和结果。至少文件_SUCCESS以及目录_logs中的任何接缝都被忽略。

是否有任何文件名模式默认被InputReader忽略？或者这只是一个固定的限制列表？

解决方案

FileInputFormat 以下 hiddenFileFilter

 public boolean accept（Path p）{
 String name = p.getName（）; 
 return！name.startsWith（_）&& ！name.startsWith（ ）; 
} 
};所以如果你使用任何 FileInputFormat （比如如 TextInputFormat ， KeyValueTextInputFormat ， SequenceFileInputFormat ）， （文件名以_或。开头）将被忽略。
 
 
 
您可以使用 FileInputFormat.setInputPathFilter 设置你的自定义 PathFilter 。请记住 hiddenFileFilter 始终处于活动状态。
 
I'm chaining multiple MapReduce jobs and want to pass along/store some meta information (e.g. configuration or name of original input) with the results. At least the file "_SUCCESS" and also anything in the directory "_logs" seams to be ignored.

Are there any filename patterns which are by default ignored by the InputReader? Or is this just a fixed limited list?
解决方案
The FileInputFormat uses the following hiddenFileFilter by default:
  private static final PathFilter hiddenFileFilter = new PathFilter(){
      public boolean accept(Path p){
        String name = p.getName(); 
        return !name.startsWith("_") && !name.startsWith("."); 
      }
    }; 
So if you uses any FileInputFormat (such as TextInputFormat, KeyValueTextInputFormat, SequenceFileInputFormat), the hidden files (the file name starts with "_" or ".") will be ignored.

You can use FileInputFormat.setInputPathFilter to set your custom PathFilter. Remember that the hiddenFileFilter is always active.

                        
这篇关于哪些文件作为输入映射器忽略？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！