S3 Java客户端由于"Content-Length分隔的消息正文的提早结束"而失败很多.或"java.net.SocketException套接字已关闭" [英] S3 Java client fails a lot with "Premature end of Content-Length delimited message body" or "java.net.SocketException Socket closed"

问题描述

我有一个在S3上做很多工作的应用程序，主要是从S3下载文件.我看到了很多这类错误，我想知道这是否是我的代码中的东西，或者服务是否真的像这样不可靠.

I have an application that does a lot work on S3, mostly downloading files from it. I am seeing a lot of these kind of errors and I'd like to know if this is something on my code or if the service is really unreliable like this.

我用来从S3对象流中读取的代码如下:

The code I'm using to read from the S3 object stream is as follows:

public static final void write(InputStream stream, OutputStream output) {

  byte[] buffer = new byte[1024];

  int read = -1;

  try {

    while ((read = stream.read(buffer)) != -1) {
      output.write(buffer, 0, read);
    }

    stream.close();
    output.flush();
    output.close();
  } catch (IOException e) {
    throw new RuntimeException(e);
  }

}

此 OutputStream 是 new BufferedOutputStream(new FileOutputStream(file)).我正在使用最新版本的Amazon S3 Java客户端，并且在放弃之前重试此呼叫 4 次.因此，在尝试了4次之后，它仍然失败.

This OutputStream is a new BufferedOutputStream( new FileOutputStream( file ) ). I am using the latest version of the Amazon S3 Java client and this call is retried four times before giving up. So, after trying this for 4 times it still fails.

任何有关如何改善此问题的提示或技巧，将不胜感激.

Any hints or tips on how I could possibly improve this are appreciated.

推荐答案

我刚刚设法克服了一个非常相似的问题.就我而言，我得到的例外是相同的.它发生在较大的文件上，但对于较小的文件则没有，并且在调试器中完全没有发生.

I just managed to overcome a very similar problem. In my case the exception I was getting was identical; it happened for larger files but not for small files, and it never happened at all while stepping through the debugger.

问题的根本原因是AmazonS3Client对象在下载过程中被垃圾回收，这导致网络连接中断.发生这种情况是因为我在每次调用时都构造了一个新的AmazonS3Client对象以加载文件，而首选用例是创建一个持久的客户端对象，该对象可以在调用中保留下来-或至少保证在整个调用过程中都存在下载.因此，一种简单的补救方法是确保保留对AmazonS3Client的引用，以使其不会出现GC.

The root cause of the problem was that the AmazonS3Client object was getting garbage collected in the middle of the download, which caused the network connection to break. This happened because I was constructing a new AmazonS3Client object with every call to load a file, while the preferred use case is to create a long-lasting client object that survives across calls - or at least is guaranteed to be around during the entirety of the download. So, the simple remedy is to make sure a reference to the AmazonS3Client is kept around so that it doesn't get GC'd.

AWS论坛上对我有帮助的链接位于: https://forums.aws.amazon .com/thread.jspa?threadID = 83326

A link on the AWS forums that helped me is here: https://forums.aws.amazon.com/thread.jspa?threadID=83326

这篇关于S3 Java客户端由于"Content-Length分隔的消息正文的提早结束"而失败很多.或"java.net.SocketException套接字已关闭"的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！