使用ServletOutputStream在Java servlet中编写非常大的文件而不会出现内存问题 [英] Using ServletOutputStream to write very large files in a Java servlet without memory issues
问题描述
我正在使用IBM Websphere Application Server v6和Java 1.4，并且正在尝试将大型CSV文件写入 ServletOutputStream 供用户下载。目前文件的范围为50-750MB。
I am using IBM Websphere Application Server v6 and Java 1.4 and am trying to write large CSV files to the ServletOutputStream for a user to download. Files are ranging from a 50-750MB at the moment.
较小的文件不会造成太大的问题但是文件较大的文件似乎是写入堆，然后导致OutOfMemory错误并关闭整个服务器。
The smaller files aren't causing too much of a problem but with the larger files it appears that it is being written into the heap which is then causing an OutOfMemory error and bringing down the entire server.
这些文件只能通过HTTPS提供给经过身份验证的用户，这就是我的原因通过Servlet提供它们而不是仅仅将它们粘贴在Apache中。
These files can only be served out to authenticated users over HTTPS which is why I am serving them through a Servlet instead of just sticking them in Apache.
我使用的代码是（在此处删除了一些毛茸茸）：
The code I am using is (some fluff removed around this):
resp.setHeader("Content-length", "" + fileLength);
resp.setContentType("application/vnd.ms-excel");
resp.setHeader("Content-Disposition","attachment; filename=\"export.csv\"");
FileInputStream inputStream = null;
try
{
inputStream = new FileInputStream(path);
byte[] buffer = new byte[1024];
int bytesRead = 0;
do
{
bytesRead = inputStream.read(buffer, offset, buffer.length);
resp.getOutputStream().write(buffer, 0, bytesRead);
}
while (bytesRead == buffer.length);
resp.getOutputStream().flush();
}
finally
{
if(inputStream != null)
inputStream.close();
}
FileInputStream 不似乎导致问题，好像我写入另一个文件或只是完全删除写入内存使用似乎不是一个问题。
The FileInputStream doesn't seem to be causing a problem as if I write to another file or just remove the write completely the memory usage doesn't appear to be a problem.
我是什么我想的是 resp.getOutputStream（）。write 正存储在内存中，直到数据可以发送到客户端。所以整个文件可能会被读取并存储在 resp.getOutputStream（）中导致我的内存问题并崩溃！
What I am thinking is that the resp.getOutputStream().write is being stored in memory until the data can be sent through to the client. So the entire file might be read and stored in the resp.getOutputStream() causing my memory issues and crashing!
我尝试过缓冲这些流，并尝试使用来自 java.nio 的频道，这些似乎都没有对我的内存问题产生任何影响。我还在循环的每次迭代和循环之后刷新 OutputStream 一次，这没有帮助。
I have tried Buffering these streams and also tried using Channels from java.nio, none of which seems to make any bit of difference to my memory issues. I have also flushed the OutputStream once per iteration of the loop and after the loop, which didn't help.
推荐答案
平均体面的servlet容器本身默认每次刷新流量大约2KB。你真的不需要在的在从同一个源顺序传输数据时间隔。例如，Tomcat（和Websphere！）可以配置为HTTP连接器的 OutputStream 上显式调用 flush（） HttpServletResponse bufferSize 属性。
The average decent servletcontainer itself flushes the stream by default every ~2KB. You should really not have the need to explicitly call flush() on the OutputStream of the HttpServletResponse at intervals when sequentially streaming data from the one and same source. In for example Tomcat (and Websphere!) this is configureable as bufferSize attribute of the HTTP connector.
平均体面的servlet容器也只是在中流式传输数据。如果事先未知内容长度（根据 Servlet API规范！）以及客户端是否支持HTTP 1.1。
The average decent servletcontainer also just streams the data in chunks if the content length is unknown beforehand (as per the Servlet API specification!) and if the client supports HTTP 1.1.
问题症状至少表明servletcontainer在刷新之前正在缓冲内存中的整个流。这可能意味着未设置内容长度标头和/或servletcontainer不支持分块编码和/或客户端不支持分块编码（即它使用HTTP 1.0）。
The problem symptoms at least indicate that the servletcontainer is buffering the entire stream in memory before flushing. This can mean that the content length header is not set and/or the servletcontainer does not support chunked encoding and/or the client side does not support chunked encoding (i.e. it is using HTTP 1.0).
要修复其中一个，只需预先设置内容长度：
To fix the one or other, just set the content length beforehand:
response.setHeader("Content-Length", String.valueOf(new File(path).length()));
这篇关于使用ServletOutputStream在Java servlet中编写非常大的文件而不会出现内存问题的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
