Hadoop-> Mapper->我们如何从给定的输入路径中读取每个文件的前N行？ [英] Hadoop-> Mapper->How can we read only Top N rows from each file from given input path?

问题描述

我是Hadoop的新手，我的要求是我需要从每个输入文件中处理前10行。以及如何在读取每个文件的10行后退出映射器。

I am new to Hadoop, My requirement is I need to process only first 10 rows from the each input file. and how to exit mapper after reading 10 rows of each file.

如果有人可以提供一些示例代码，这将是非常有帮助的。

If anyone can provide some sample code , it would be great help.

提前感谢。 p>

thanks in advance.

推荐答案

你可以覆盖你的mapper的run方法，一旦你迭代了map循环，你可以从while循环。这将假设您的文件不可分割，否则您将从每个分割中获取前10行：

You can override the run method of your mapper, and once you've iterated the map loop 10 times you can break from the while loop. This will assume your files are not splitable, otherwise you'll get the first 10 lines from each split:

@Override
public void run(Context context) throws IOException, InterruptedException {
  setup(context);

  int rows = 0;
  while (context.nextKeyValue()) {
    if (rows++ == 10) {
      break;
    }

    map(context.getCurrentKey(), context.getCurrentValue(), context);
  }

  cleanup(context);
}

这篇关于Hadoop-> Mapper->我们如何从给定的输入路径中读取每个文件的前N行？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！