Jersey服务文件上传会导致OutOfMemoryError [英] Jersey service file upload causes OutOfMemoryError

问题描述

我正在开发与泽西岛2.0的表单提交服务。表单包含几个文本字段和一个文件字段。我需要提取文件，文件名，文件媒体类型和文件内容类型，并将它们保存在对象存储。
$ b $ pre $ @ $ $ $ $ @ $ $ b $ @ bons @ $（$ MediaType.MULTIPART_FORM_DATA）
@生成（{MediaType.APPLICATION_XML，MediaType.APPLICATION_JSON}）
public class UploadService {
@POST
public BlobDo uploadFile（FormDataMultiPart uploadedBody）{
String accountSid = uploadedBody.getField（ 。帐户-SID）的getValue（）;
String apiToken = uploadedBody.getField（api-token）。getValue（）;
String checksum = uploadedBody.getField（checksum）。getValue（）;

FormDataBodyPart bodyPart = uploadedBody.getField（file）;
MySwiftObject obj = new MySwiftObject（bodyPart.getValueAs（InputStream.class））;
obj.setName（bodyPart.getContentDisposition（）。getFileName（））;
obj.setContentType（bodyPart.getMediaType（）。toString（））;
obj.setContentDisposition（bodyPart.getContentDisposition（）。toString（））;
...
}

pom.xml

 < jersey.version> 2.17< /jersey.version> 
 
< dependency> 
< groupId> org.glassfish.jersey.containers< / groupId> 
< artifactId> jersey-container-servlet-core< / artifactId> 
< /依赖> 
<依赖> 
< groupId> org.glassfish.jersey.media< / groupId> 
< artifactId> jersey-media-json-jackson< / artifactId> 
< / dependency> 
<依赖> 
< groupId> org.glassfish.jersey.test-framework.providers< / groupId> 
< artifactId> jersey-test-framework-provider-inmemory< / artifactId> 
< /依赖> 
<依赖> 
< groupId> org.glassfish.jersey.media< / groupId> 
< artifactId> jersey-media-multipart< / artifactId> 
< /依赖> 
<依赖> 
< groupId> org.glassfish.jersey.core< / groupId> 
< artifactId>球衣服务器< / artifactId> 
< /依赖> 
<依赖> 
< groupId> org.glassfish.jersey.ext< / groupId> 
< artifactId> jersey-bean-validation< / artifactId> 
< /依赖> 
  

表单提交请求

  POST / nbs / v2 / upload HTTP / 1.1 
 Host：127.0.0.1:8080 
 Cache-Control：no-cache 
 Postman-Token：a4c1d4e9-5f71- 2321-3870-e9cac0524f8d 
 Content-Type：multipart / form-data; border = ---- WebKitFormBoundaryA2Z9pPMA7y3da8BG 
 
 ------ WebKitFormBoundaryA2Z9pPMA7y3da8BG 
 Content-Disposition：form-data; name =account-sid
 
 Q45Ppm5ukvdqjTQ6eW0O5ztTXipwnjKQx1p6cf + fbCQ = 
 ------ WebKitFormBoundaryA2Z9pPMA7y3da8BG 
 Content-Disposition：form-data; name =api-token
 
 6397cd691909fdc14cef67dbc1dc2dc3 
 ------ WebKitFormBoundaryA2Z9pPMA7y3da8BG 
 Content-Disposition：form-data; NAME = 文件; filename =screen_4_100155.jpg
 Content-Type：image / jpeg 
 
 ...... Exif..MM。* ............ ................. b ........... j 
 ------ WebKitFormBoundaryA2Z9pPMA7y3da8BG 
 Content-Disposition：form -数据; name =checksum
 
 6a3381b1d16bded4a3dfc325a8bb800e 
 ------ WebKitFormBoundaryA2Z9pPMA7y3da8BG 
  

JVM堆大小

  -Xmx = 1024mb 
   
 $ b $ h 
 $ b 
问题
 
当上传〜50MB文件时，两个临时文件在名为 FileBackedOutputStream7949386530699987086.tmp 和的目录下创建类似的MD5总和 /tmp/tomcat7-tomcat7-tmp  code> MIME8234229766850016150.tmp  
 
 
 
在上传完成之前服务器抛出异常
 
 javax.servlet.ServletException：org.glassfish.jersey.server.ContainerException：java.lang.OutOfMemoryError：Java堆空间
 org.glassfish.jersey.servlet.WebComponent.service（ WebComponent.java:421）
 org.glassfish.jersey.servlet.ServletContainer.service（ServletContainer.java:386）
 org.glassfish.jersey.servlet.ServletContainer.service（ServletContainer.java： ）
 org.glassfish.jersey.servlet.ServletContainer.service（ServletContainer.java:222）
 
并删除文件名称 MIME8234229766850016150.tmp ，但保留另一个。并删除 FileBackedOutputStream .... tmp 文件填充硬盘上的整个空间。
 
 
 
 
 
 
我做了什么
 
 
 
 
 
 
将堆空间增加到7GB，但不能上传〜200MB的文件。 
 
 
在服务器上运行作业以删除旧的临时文件。
     jersey-multipart-config.properties 和content  
 
 
  jersey.config.multipart.bufferThreshold  MIME [random numbers] =         .tmp 不再被创建，但 FileBackedOutputStream [random number] .tmp 仍然挂在硬盘上，除非tomcat重新启动。
 
 
 
 
 
 
问题
 
 
 
 
 
 
泽西岛处理大文件（也许1GB），而不会在我的硬盘上留下临时文件？最好的情况是不要使用硬盘驱动器，并通过内存传输小块。
  
 
 
为什么我得到堆溢出 if输入流是由文件支持？
 
 
 
 $ b 
我读的材料
 h1> 
 
 
 
 
到目前为止我已经找到了关闭的解释。 阅读 
 
 
这个人有类似的问题，但在客户端。 阅读 
 
 
可能包含解决方案，但无法理解答案。 阅读 
 
 
非常接近我的问题，但无法解决问题。 阅读 
 
 
  bufferThreshold 的想法是从这里取得的。 阅读 
 
 
 
 
解决看起来问题1是通过在< servlet>  web.xml 中添加以下行来解决的> tag  
 
 
 < multipart-config> 
< location> / tmp< / location> 
< max-file-size> 1000000000< / max-file-size> 
< max-request-size> 1500000000< / max-request-size> 
< file-size-threshold> 0< / file-size-threshold> 
< / multipart-config> 
  
删除了 jersey-multipart-config.properties 文件。  
 
 
现在我可以上传超过200Mb的文件。没有更多的临时文件被创建。
 
 
 
但我仍然无法解释问题＃2。
I'm developing form submission service with Jersey 2.0. The form includes several text fields and one file field. I need to extract file, file name, file media type and file content type and save them in object store.
@Path("upload")
@Consumes({MediaType.MULTIPART_FORM_DATA})
@Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON})
public class UploadService {
    @POST
    public BlobDo uploadFile(FormDataMultiPart uploadedBody) {
        String accountSid = uploadedBody.getField("account-sid").getValue();
        String apiToken = uploadedBody.getField("api-token").getValue();
        String checksum = uploadedBody.getField("checksum").getValue();

        FormDataBodyPart bodyPart = uploadedBody.getField("file");
        MySwiftObject obj = new MySwiftObject(bodyPart.getValueAs(InputStream.class));
        obj.setName(bodyPart.getContentDisposition().getFileName());
        obj.setContentType(bodyPart.getMediaType().toString());
        obj.setContentDisposition(bodyPart.getContentDisposition().toString());
   ...
}
pom.xml
<jersey.version>2.17</jersey.version>

<dependency>
    <groupId>org.glassfish.jersey.containers</groupId>
    <artifactId>jersey-container-servlet-core</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.media</groupId>
    <artifactId>jersey-media-json-jackson</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.test-framework.providers</groupId>
    <artifactId>jersey-test-framework-provider-inmemory</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.media</groupId>
    <artifactId>jersey-media-multipart</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.core</groupId>
    <artifactId>jersey-server</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.ext</groupId>
    <artifactId>jersey-bean-validation</artifactId>
</dependency>
form submission request
POST /nbs/v2/upload HTTP/1.1
Host: 127.0.0.1:8080
Cache-Control: no-cache
Postman-Token: a4c1d4e9-5f71-2321-3870-e9cac0524f8d
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryA2Z9pPMA7y3da8BG

------WebKitFormBoundaryA2Z9pPMA7y3da8BG
Content-Disposition: form-data; name="account-sid"

Q45Ppm5ukvdqjTQ6eW0O5ztTXipwnjKQx1p6cf+fbCQ=
------WebKitFormBoundaryA2Z9pPMA7y3da8BG
Content-Disposition: form-data; name="api-token"

6397cd691909fdc14cef67dbc1dc2dc3
------WebKitFormBoundaryA2Z9pPMA7y3da8BG
Content-Disposition: form-data; name="file"; filename="screen_4_100155.jpg"
Content-Type: image/jpeg

......Exif..MM.*.............................b...........j
------WebKitFormBoundaryA2Z9pPMA7y3da8BG
Content-Disposition: form-data; name="checksum"

6a3381b1d16bded4a3dfc325a8bb800e
------WebKitFormBoundaryA2Z9pPMA7y3da8BG
JVM heap size
-Xmx=1024mb




The problem

When uploading ~50MB file two temporary files with similar MD5 sums are created in the directory /tmp/tomcat7-tomcat7-tmp with name FileBackedOutputStream7949386530699987086.tmp and MIME8234229766850016150.tmp

Before upload is complete server throws exception
javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: Java heap space
    org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:421)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:386)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:222)
and deletes the file with name MIME8234229766850016150.tmp but leaves the other one. And undeleted FileBackedOutputStream....tmp files fill the whole space on hard drive.



What I did


Increase heap space to 7GB, but ~200MB files can't be uploaded.
Run job on server to delete old temporary files.
Created file with name jersey-multipart-config.properties and content

jersey.config.multipart.bufferThreshold = -1
The file MIME[random numbers].tmp is no longer created, but FileBackedOutputStream[random number].tmp still hangs on hard drive unless tomcat is restarted.



Question


How can Jersey handle large files (maybe 1GB) without leaving temporary files on my hard disk? Best case would be not to use hard drive at all and transfer small chunks through memory.
Why do I get heap overflow if input stream is backed with files?




Materials I read


The closes explanation I found so far. read
This guy has similar problem but on the client side. read
May contain solution, but couldn't understand the answer. read
Pretty close to my problem but can't work it out. read
bufferThreshold idea was taken from here. read

解决方案
It seems that problem #1 was solved by adding following lines in my web.xml under the <servlet> tag
<multipart-config>
        <location>/tmp</location>
        <max-file-size>1000000000</max-file-size>
        <max-request-size>1500000000</max-request-size>
        <file-size-threshold>0</file-size-threshold>
</multipart-config>
and removed jersey-multipart-config.properties file. 

Now I can upload files that are over 200Mb. No more temporary files are created.

But I still can't explain problem #2.

                        
这篇关于Jersey服务文件上传会导致OutOfMemoryError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！