从 Amazon S3 读取 Servlet 太慢 [英] Servlet reading from Amazon S3 so slow

问题描述

我需要一个 servlet 来从 Amazon S3 服务器返回文件.只有服务器具有访问凭据，S3 存储桶不公开.我无法改变这一点.有人告诉我使用数据流，但它们太慢了.为了进行测试，我有一个带有缩略图的小项目，当您单击其中一个时，它会打开一个带有完整图像的新选项卡.加载 5mb 图像大约需要一分钟.好慢.

I need a servlet to return files from Amazon S3 servers. Only the server has the credentials to access, the S3 bucket is not public. I cannot change that. I was told to use data streams, but they are so slow. To test, I have a small proyect with thumbnails and when you click on one it opens a new tab with the full image. A 5mb image takes about a minute to load. That slow.

从S3读取并返回数据流的函数:

The function that reads from S3 and returns the data stream:

public void downloadDirectlyFromS3(String s3Path, String fileName, HttpServletResponse response) {
    AmazonS3 s3Client = new AmazonS3Client(new ProfileCredentialsProvider());
    s3Client.setEndpoint(S3ENDPOINT);

    S3Object s3object = s3Client.getObject(new GetObjectRequest(s3Path, fileName));

    byte[] buffer = new byte[5 * 1024 * 1024];

    try {
        InputStream input = s3object.getObjectContent();
        ServletOutputStream output = response.getOutputStream();
        for (int length = 0; (length = input.read(buffer)) > 0;) {
            output.write(buffer, 0, length);
        }
        output.close();
    } catch (FileNotFoundException e) {
        e.printStackTrace();
    } catch (IOException e) {
        e.printStackTrace();
    }
}

推荐答案

我找到了答案.问题是记录器.我们正在使用 log4j 并且它被设置为调试，所以所有的流跟踪都写在控制台中.以防万一它发生在其他人身上，这是他们说在生产中应该避免的链接:https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html#verbose-wire-logging

I found the answer. The problem was the logger. We are using log4j and it was set to debug, so all the trace of the stream was written in the console. Just in case it happens to somebody else, here's the link where they say it should be avoided in production: https://docs.aws.amazon.com/sdk-for-java/v1/developer-guide/java-dg-logging.html#verbose-wire-logging

而且我也开始使用 TransferManager，正如 saravanakumar v 所说，似乎速度稍快一些.

And I also started using the TransferManager as saravanakumar v said, seems to be slightly faster.

例如，我下载了这个 ip-to-country.bin，在调试时查看这个示例响应

for example i had this ip-to-country.bin downloaded look at this sample response while debug was on

2017-05-23 12:06:21,770 Wire (Wire.java:86) DEBUG - http-outgoing-0 << "[0xf0][0x1]BGY[0xb][0x0][0x1]ITY[0xb][0x10][0x1]I"
2017-05-23 12:06:21,824 Wire (Wire.java:72) DEBUG - http-outgoing-0 << "RY[0xb] [0x1]CHY[0xb]0[0x1]DEY[0xb]@[0x1]KZY[0xb]P[0x1]ITY[0xb]`[0x1]ESY[0xb][0x80][0x1]PLY[0xb][0xa0][0x1]GEY[0xb][0xb0][0x1]TJY[0xb][0xc0][0x1]DEY[0xb][0xd0][0x1]CHY[0xb][0xe0][0x1]CZY[0xc][0x0][0x1]GBY[0xc][0x10][0x1]ESY[0xc] [0x1]RUY[0xc]0[0x1]SKY[0xc]@[0x1]RUY[0xc]P[0x1]UZY[0xc]`[0x1]RUY[0xc]p[0x1]MDY[0xc][0x80][0x1]ITY[0xc][0x90][0x1]GBY[0xc][0xa0][0x1]ITY[0xc][0xc0][0x1]UAY[0xc][0xe0][0x1]SAY[0xc][0xf0][0x1]RUY[\r][0x0][0x1]NOY[\r] [0x1]HUY[\r]0[0x1]FRY[\r]@[0x1]DEY[\r]P[0x1]ESY[\r]`[0x1]HUY[\r]p[0x1]ESY[\r][0x80][0x1]GBY[\r][0xa0][0x1]DEY[\r][0xb0][0x1]ATY[\r][0xc0][0x1]DEY[\r][0xd0][0x1]RUY[\r][0xe0][0x1]SEY[0xe][0x0][0x1]NOY[0xe][0x10][0x1]RUY[0xe] [0x1]ESY[0xe]0[0x1]RUY[0xe]@[0x1]CHY[0xe]P[0x1]NGY[0xe]`[0x1]AZY[0xe]p[0x1]DEY[0xe][0x80][0x1]GBY[0xe][0x90][0x1]DEY[0xe][0xb0][0x1]GBY[0xe][0xc0][0x1]RUY[0xe][0xd0][0x1]HRY[0xe][0xe0][0x1]ATY[0xe][0xf0][0x1]RUY[0xf][0x0][0x1]ATY[0xf][0x10][0x1]RUY[0xf] [0x1]ESY[0xf]0[0x1]RUY[0xf]@[0x1]GBY[0xf]P[0x1]FRY[0xf]`[0x1]MTY[0xf]p[0x1]GBY[0xf][0x80][0x1]RUY[0xf][0xa0][0x1]EUY[0xf][0xb0][0x1]KZY[0xf][0xc0][0x1]RUY[0xf][0xd0][0x1]ITY[0xf][0xe0][0x1]BEY[0xf][0xf0][0x1]SEY[0x10][0x0][0x1]FRY[0x10][0x10][0x1]RUY[0x10] [0x1]BEY[0x10]0[0x1]GBY[0x10]@[0x1]MKY[0x10]`[0x1]DKY[0x10]p[0x1]ATY[0x10][0x80][0x1]RSY[0x10][0x90][0x1]ESY[0x10][0xa0][0x1]DEY[0x10][0xb0][0x1]CZY[0x10][0xc0][0x1]SEY[0x10][0xd0][0x1]GBY[0x10][0xe0][0x1]CYY[0x10][0xf0][0x1]ESY[0x11][0x0][0x1]NOY[0x11][0x10][0x1]DEY[0x11] [0x1]PLY[0x11]0[0x1]BGY[0x11]@[0x1]SEY[0x11]P[0x1]LTY[0x11]`[0x1]RSY[0x11]p[0x1]RUY[0x11][0x80][0x1]NLY[0x11][0x90][0x1]TRY[0x11][0xa0][0x1]RUY[0x11]

这篇关于从 Amazon S3 读取 Servlet 太慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！