Hadoop FileSystem在执行BufferedReader.close（）时关闭异常 [英] Hadoop FileSystem closed exception when doing BufferedReader.close()

问题描述

在Reduce设置方法中，我试图关闭一个 BufferedReader 对象并获取 FileSystem 关闭的异常。它不会一直发生。这是我用于创建 BufferedReader 的一段代码。

  String fileName =<某些HDFS文件路径> 
 Configuration conf = new Configuration（）; 
 FileSystem fs = FileSystem.get（conf）; 
路径hdfsPath =新路径（文件名）; 
 FSDataInputStream in = fs.open（hdfsPath）; 
 InputStreamReader inputStreamReader = new InputStreamReader（fileInputStream）; 
 BufferedReader bufferedReader = new BufferedReader（inputStreamReader）; 
  

我从bufferedReader读取内容，一旦完成所有阅读，就关闭它。

这是一段读取它的代码

 字符串行; （（line = reader.readLine（））！= null）{
 //做某事
} 
  

>

这段关闭阅读器的代码片段。

  if（bufferedReader！ = null）{
 bufferedReader.close（）; 
 
 $ / code> 

这是我执行<$ $时发生异常的堆栈跟踪c $ c $ bufferedReader.close（）。

I，[2013-11-18T04：56： 51.601135＃25683] INFO - ：
attempt_201310111840_142285_r_000009_0：在
org.apache.hadoop.hdfs.DFSClient.checkOpen（DFSClient.java:565）

I，[2013-11-18T04：56：51.601168＃25683] INFO - ：
attempt_201310111840_142285_r_000009_0：在
org.apache.hadoop.hdfs.DFSInputStream.close（DFSInputStream.java:522）

我，[2013-11-18T04：56：51.601199＃25683]信息 - ：
attempt_201310111840_142285_r_000009_0：at
java.io.FilterInputStream .close（FilterInputStream.java:155）

I，[2013-11-18T04：56：51.601230＃25683] INFO - ：
attempt_201310111840_142285_r_000009_0：在
sun.nio.cs.StreamDecoder.implClose（StreamDecoder.java:358）

我，[2013-11-18T04：56：5 1.601263＃25683] INFO - ：
attempt_201310111840_142285_r_000009_0：在
sun.nio.cs.StreamDecoder.close（StreamDecoder.java:173）

I，[2013-11-18T04：56：51.601356＃25683] INFO - ：
attempt_201310111840_142285_r_000009_0：在
java.io.InputStreamReader.close（InputStreamReader.java:182）

I，[2013-11-18T04：56：51.601395＃25683] INFO - ：
attempt_201310111840_142285_r_000009_0：在
java.io.BufferedReader.close（BufferedReader.java ：497）

我不知道为什么会发生这种异常。这不是多线程的，所以，我不指望有任何形式的竞争条件。你可以帮我理解。

谢谢，

Venk

解决方案

hadoop文件系统API有一个鲜为人知的问题： FileSystem.get 为每个对象返回相同的对象用相同的文件系统调用。所以如果一个人在任何地方关门，他们都会关门。你可以辩论这个决定的优点，但事实就是这样。

因此，如果你试图关闭你的BufferedReader，并试图刷新一些数据，已缓冲，但底层流连接到已关闭的文件系统，则会出现此错误。检查您的代码以查找您正在关闭FileSystem对象的任何其他位置，并查找竞争条件。另外，我相信Hadoop本身在某些时候会关闭FileSystem，所以为了安全起见，您应该只能从Reducer的设置，减少或清理方法（或配置，减少和关闭）中访问它，具体取决于哪个API你正在使用）。

From within the Reduce setup method,I am trying to close a BufferedReader object and getting a FileSystem closed exception. It does not happen all the time. This is the piece of code I used to create the BufferedReader.

    String fileName = <some HDFS file path>
    Configuration conf = new Configuration();
    FileSystem fs = FileSystem.get(conf);
    Path hdfsPath = new Path(filename);
    FSDataInputStream in = fs.open(hdfsPath);
    InputStreamReader inputStreamReader = new InputStreamReader(fileInputStream);
    BufferedReader bufferedReader = new BufferedReader(inputStreamReader);

I read contents from the bufferedReader and once all the reading is done, I close it.

This is the piece of code that reads it

String line;
while ((line = reader.readLine()) != null) {
    // Do something
}

This the piece of code that closes the reader.

    if (bufferedReader != null) {
        bufferedReader.close();
    }

This is the stack trace for the exception that happens when I do a bufferedReader.close().

I, [2013-11-18T04:56:51.601135 #25683] INFO -- : attempt_201310111840_142285_r_000009_0: at org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:565)

I, [2013-11-18T04:56:51.601168 #25683] INFO -- : attempt_201310111840_142285_r_000009_0: at org.apache.hadoop.hdfs.DFSInputStream.close(DFSInputStream.java:522)

I, [2013-11-18T04:56:51.601199 #25683] INFO -- : attempt_201310111840_142285_r_000009_0: at java.io.FilterInputStream.close(FilterInputStream.java:155)

I, [2013-11-18T04:56:51.601230 #25683] INFO -- : attempt_201310111840_142285_r_000009_0: at sun.nio.cs.StreamDecoder.implClose(StreamDecoder.java:358)

I, [2013-11-18T04:56:51.601263 #25683] INFO -- : attempt_201310111840_142285_r_000009_0: at sun.nio.cs.StreamDecoder.close(StreamDecoder.java:173)

I, [2013-11-18T04:56:51.601356 #25683] INFO -- : attempt_201310111840_142285_r_000009_0: at java.io.InputStreamReader.close(InputStreamReader.java:182)

I, [2013-11-18T04:56:51.601395 #25683] INFO -- : attempt_201310111840_142285_r_000009_0: at java.io.BufferedReader.close(BufferedReader.java:497)

I am not sure why this exception is happening. This is not multithreaded and so, I do not expect there to be a race condition of any sort. Can you please help me understand.

Thanks,

Venk

解决方案

There is a little-known gotcha with the hadoop filesystem API: FileSystem.get returns the same object for every invocation with the same filesystem. So if one is closed anywhere, they are all closed. You could debate the merits of this decision, but that's the way it is.

So, if you attempt to close your BufferedReader, and it tries to flush out some data it has buffered, but the underlying stream is connected to a FileSystem that is already closed, you'll get this error. Check your code for any other places you are closing a FileSystem object, and look for race conditions. Also, I believe Hadoop itself will at some point close the FileSystem, so to be safe, you should probably only be accessing it from within the Reducer's setup, reduce, or cleanup methods (or configure, reduce, and close, depending on which API you're using).

这篇关于Hadoop FileSystem在执行BufferedReader.close（）时关闭异常的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！