流式传输java servlet中的大型文件 [英] Streaming large files in a java servlet

问题描述

我正在构建一个需要扩展的java服务器。其中一个servlet将提供存储在Amazon S3中的图像。

I am building a java server that needs to scale. One of the servlets will be serving images stored in Amazon S3.

最近在加载时，我的虚拟机内存不足，而且是在我添加了服务代码之后因此我很确定流式传输更大的servlet响应会导致我的麻烦。

Recently under load, I ran out of memory in my VM and it was after I added the code to serve the images so I'm pretty sure that streaming larger servlet responses is causing my troubles.

我的问题是：有没有最好的方法来解决如何将java servlet编码为从数据库或其他云存储中读取时，将大量（> 200k）响应流回浏览器？

My question is : is there any best practice in how to code a java servlet to stream a large (>200k) response back to a browser when read from a database or other cloud storage?

我考虑将文件写入本地临时驱动器并且然后产生另一个线程来处理流，以便可以重用tomcat servlet线程。这似乎很重。

I've considered writing the file to a local temp drive and then spawning another thread to handle the streaming so that the tomcat servlet thread can be re-used. This seems like it would be io heavy.

任何想法都会受到赞赏。谢谢。

Any thoughts would be appreciated. Thanks.

推荐答案

如果可能，您不应存储要在内存中提供的文件的全部内容。相反，为数据获取InputStream，并将数据复制到Servlet OutputStream中。例如：

When possible, you should not store the entire contents of a file to be served in memory. Instead, aquire an InputStream for the data, and copy the data to the Servlet OutputStream in pieces. For example:

ServletOutputStream out = response.getOutputStream();
InputStream in = [ code to get source input stream ];
String mimeType = [ code to get mimetype of data to be served ];
byte[] bytes = new byte[FILEBUFFERSIZE];
int bytesRead;

response.setContentType(mimeType);

while ((bytesRead = in.read(bytes)) != -1) {
    out.write(bytes, 0, bytesRead);
}

// do the following in a finally block:
in.close();
out.close();

我同意toby，你应该将它们指向S3网址。

I do agree with toby, you should instead "point them to the S3 url."

至于OOM例外，你确定它与提供图像数据有关吗？假设你的JVM有256MB的额外内存用于提供图像数据。在Google的帮助下，256MB / 200KB= 1310.对于2GB额外内存（这些天非常合理的数量），可以支持10,000多个并发客户端。即便如此，1300个同步客户端数量也相当大。这是您遇到的负载类型吗？如果没有，您可能需要在其他地方寻找OOM异常的原因。

As for the OOM exception, are you sure it has to do with serving the image data? Let's say your JVM has 256MB of "extra" memory to use for serving image data. With Google's help, "256MB / 200KB" = 1310. For 2GB "extra" memory (these days a very reasonable amount) over 10,000 simultaneous clients could be supported. Even so, 1300 simultaneous clients is a pretty large number. Is this the type of load you experienced? If not, you may need to look elsewhere for the cause of the OOM exception.

编辑 - 关于：

在这个用例中，图像可以包含敏感数据......

In this use case the images can contain sensitive data...

当我通读S3时几个星期前，我注意到您可以生成可以附加到S3 URL的时间过期密钥。因此，您不必向公众开放S3上的文件。我对该技术的理解是：

When I read through the S3 documentation a few weeks ago, I noticed that you can generate time-expiring keys that can be attached to S3 URLs. So, you would not have to open up the files on S3 to the public. My understanding of the technique is:

1. 初始HTML页面包含指向您的webapp的下载链接


2. 用户点击下载链接


3. 您的网络应用会生成一个S3网址，其中包含一个过期的密钥，比如5分钟。


4. 发送HTTP使用步骤3中的URL重定向到客户端。


5. 用户从S3下载文件。即使下载时间超过5分钟，这仍然有效 - 一旦下载开始，它就可以继续完成。

1. Initial HTML page has download links to your webapp
2. User clicks on a download link
3. Your webapp generates an S3 URL that includes a key that expires in, lets say, 5 minutes.
4. Send an HTTP redirect to the client with the URL from step 3.
5. The user downloads the file from S3. This works even if the download takes more than 5 minutes - once a download starts it can continue through completion.

这篇关于流式传输java servlet中的大型文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！