Jersey 服务文件上传导致 OutOfMemoryError [英] Jersey service file upload causes OutOfMemoryError

查看：34
本文介绍了Jersey 服务文件上传导致 OutOfMemoryError的处理方法，对大家解决问题具有一定的参考价值，需要的朋友们下面随着小编来一起学习吧！

问题描述

我正在使用 Jersey 2.0 开发表单提交服务.该表单包括几个 text 字段和一个 file 字段.我需要提取文件、文件名、文件媒体类型和文件内容类型并将它们保存在对象存储中.

@Path("上传")@Consumes({MediaType.MULTIPART_FORM_DATA})@Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON})公共类 UploadService {@邮政公共 BlobDo uploadFile(FormDataMultiPart uploadBody) {String accountSid = uploadBody.getField("account-sid").getValue();String apiToken = uploadBody.getField("api-token").getValue();字符串校验和 = UploadBody.getField("校验和").getValue();FormDataBodyPart bodyPart = uploadBody.getField("file");MySwiftObject obj = new MySwiftObject(bodyPart.getValueAs(InputStream.class));obj.setName(bodyPart.getContentDisposition().getFileName());obj.setContentType(bodyPart.getMediaType().toString());obj.setContentDisposition(bodyPart.getContentDisposition().toString());...}

pom.xml

<jersey.version>2.17</jersey.version><依赖性><groupId>org.glassfish.jersey.containers</groupId><artifactId>jersey-container-servlet-core</artifactId></依赖><依赖性><groupId>org.glassfish.jersey.media</groupId><artifactId>jersey-media-json-jackson</artifactId></依赖><依赖性><groupId>org.glassfish.jersey.test-framework.providers</groupId><artifactId>jersey-test-framework-provider-inmemory</artifactId></依赖><依赖性><groupId>org.glassfish.jersey.media</groupId><artifactId>jersey-media-multipart</artifactId></依赖><依赖性><groupId>org.glassfish.jersey.core</groupId><artifactId>球衣服务器</artifactId></依赖><依赖性><groupId>org.glassfish.jersey.ext</groupId><artifactId>jersey-bean-validation</artifactId></依赖>

表单提交请求

POST/nbs/v2/上传 HTTP/1.1主机:127.0.0.1:8080缓存控制:无缓存邮递员令牌:a4c1d4e9-5f71-2321-3870-e9cac0524f8d内容类型:multipart/form-data；边界=----WebKitFormBoundaryA2Z9pPMA7y3da8BG------WebKitFormBoundaryA2Z9pPMA7y3da8BG内容处置:表单数据；名称="帐户-sid"Q45Ppm5ukvdqjTQ6eW0O5ztTXipwnjKQx1p6cf+fbCQ=------WebKitFormBoundaryA2Z9pPMA7y3da8BG内容处置:表单数据；名称="api-token"6397cd691909fdc14cef67dbc1dc2dc3------WebKitFormBoundaryA2Z9pPMA7y3da8BG内容处置:表单数据；名称=文件"；文件名="screen_4_100155.jpg"内容类型:图片/jpeg......Exif..MM.*......................b..........j------WebKitFormBoundaryA2Z9pPMA7y3da8BG内容处置:表单数据；名称="校验和"6a3381b1d16bded4a3dfc325a8bb800e------WebKitFormBoundaryA2Z9pPMA7y3da8BG

JVM 堆大小

-Xmx=1024mb

<小时>

问题

上传 ~50MB 文件时，会在目录 /tmp/tomcat7-tomcat7-tmp 中创建两个具有相似 MD5 和的临时文件，名称为 FileBackedOutputStream7949386530699987086.tmp 和 >MIME8234229766850016150.tmp

上传完成前服务器抛出异常

<上一页>javax.servlet.ServletException:org.glassfish.jersey.server.ContainerException:java.lang.OutOfMemoryError:Java堆空间org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:421)org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:386)org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:222)

并删除名称为 MIME8234229766850016150.tmp 的文件，但保留另一个文件.并且未删除的 FileBackedOutputStream..tmp 文件填满了硬盘上的整个空间.

<小时>

我做了什么

  1. 将堆空间增加到 7GB，但无法上传 ~200MB 文件.

  2. 在服务器上运行作业以删除旧的临时文件.

  3. 创建的文件名为 jersey-multipart-config.properties 和内容

    jersey.config.multipart.bufferThreshold = -1

文件 MIME[random numbers].tmp 不再创建，但 FileBackedOutputStream[random number].tmp 仍然挂在硬盘上，除非重新启动 tomcat.

<小时>

问题

  1. Jersey 如何在不将临时文件留在硬盘上的情况下处理大文件(可能是 1GB)?最好的情况是根本不使用硬盘驱动器并通过内存传输小块.

  2. 如果输入流由文件支持，为什么会出现堆溢出?

<小时>

我读过的资料

  • 到目前为止我找到的关闭解释.阅读
  • 这家伙有类似的问题，但在客户端.阅读
  • 可能包含解决方案，但无法理解答案.阅读
  • 非常接近我的问题，但无法解决.阅读
  • bufferThreshold 的想法取自这里.阅读

解决方案

似乎问题 #1 是通过在 <servlet> 下的 web.xml 中添加以下行来解决的 标签

<multipart-config><位置>/tmp</位置><最大文件大小>1000000000</最大文件大小><最大请求大小>1500000000</最大请求大小><文件大小阈值>0</文件大小阈值></多部分配置>

并删除了 jersey-multipart-config.properties 文件.

现在我可以上传超过 200Mb 的文件.不再创建临时文件.

但我仍然无法解释问题 #2.

I'm developing form submission service with Jersey 2.0. The form includes several text fields and one file field. I need to extract file, file name, file media type and file content type and save them in object store.

@Path("upload")
@Consumes({MediaType.MULTIPART_FORM_DATA})
@Produces({MediaType.APPLICATION_XML, MediaType.APPLICATION_JSON})
public class UploadService {
    @POST
    public BlobDo uploadFile(FormDataMultiPart uploadedBody) {
        String accountSid = uploadedBody.getField("account-sid").getValue();
        String apiToken = uploadedBody.getField("api-token").getValue();
        String checksum = uploadedBody.getField("checksum").getValue();

        FormDataBodyPart bodyPart = uploadedBody.getField("file");
        MySwiftObject obj = new MySwiftObject(bodyPart.getValueAs(InputStream.class));
        obj.setName(bodyPart.getContentDisposition().getFileName());
        obj.setContentType(bodyPart.getMediaType().toString());
        obj.setContentDisposition(bodyPart.getContentDisposition().toString());
   ...
}

pom.xml

<jersey.version>2.17</jersey.version>

<dependency>
    <groupId>org.glassfish.jersey.containers</groupId>
    <artifactId>jersey-container-servlet-core</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.media</groupId>
    <artifactId>jersey-media-json-jackson</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.test-framework.providers</groupId>
    <artifactId>jersey-test-framework-provider-inmemory</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.media</groupId>
    <artifactId>jersey-media-multipart</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.core</groupId>
    <artifactId>jersey-server</artifactId>
</dependency>
<dependency>
    <groupId>org.glassfish.jersey.ext</groupId>
    <artifactId>jersey-bean-validation</artifactId>
</dependency>

form submission request

POST /nbs/v2/upload HTTP/1.1
Host: 127.0.0.1:8080
Cache-Control: no-cache
Postman-Token: a4c1d4e9-5f71-2321-3870-e9cac0524f8d
Content-Type: multipart/form-data; boundary=----WebKitFormBoundaryA2Z9pPMA7y3da8BG

------WebKitFormBoundaryA2Z9pPMA7y3da8BG
Content-Disposition: form-data; name="account-sid"

Q45Ppm5ukvdqjTQ6eW0O5ztTXipwnjKQx1p6cf+fbCQ=
------WebKitFormBoundaryA2Z9pPMA7y3da8BG
Content-Disposition: form-data; name="api-token"

6397cd691909fdc14cef67dbc1dc2dc3
------WebKitFormBoundaryA2Z9pPMA7y3da8BG
Content-Disposition: form-data; name="file"; filename="screen_4_100155.jpg"
Content-Type: image/jpeg

......Exif..MM.*.............................b...........j
------WebKitFormBoundaryA2Z9pPMA7y3da8BG
Content-Disposition: form-data; name="checksum"

6a3381b1d16bded4a3dfc325a8bb800e
------WebKitFormBoundaryA2Z9pPMA7y3da8BG

JVM heap size

-Xmx=1024mb

The problem

When uploading ~50MB file two temporary files with similar MD5 sums are created in the directory /tmp/tomcat7-tomcat7-tmp with name FileBackedOutputStream7949386530699987086.tmp and MIME8234229766850016150.tmp

Before upload is complete server throws exception

javax.servlet.ServletException: org.glassfish.jersey.server.ContainerException: java.lang.OutOfMemoryError: Java heap space
    org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:421)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:386)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:335)
    org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:222)

and deletes the file with name MIME8234229766850016150.tmp but leaves the other one. And undeleted FileBackedOutputStream....tmp files fill the whole space on hard drive.

What I did

  1. Increase heap space to 7GB, but ~200MB files can't be uploaded.

  2. Run job on server to delete old temporary files.

  3. Created file with name jersey-multipart-config.properties and content

    jersey.config.multipart.bufferThreshold = -1

The file MIME[random numbers].tmp is no longer created, but FileBackedOutputStream[random number].tmp still hangs on hard drive unless tomcat is restarted.

Question

  1. How can Jersey handle large files (maybe 1GB) without leaving temporary files on my hard disk? Best case would be not to use hard drive at all and transfer small chunks through memory.

  2. Why do I get heap overflow if input stream is backed with files?

Materials I read

  • The closes explanation I found so far. read
  • This guy has similar problem but on the client side. read
  • May contain solution, but couldn't understand the answer. read
  • Pretty close to my problem but can't work it out. read
  • bufferThreshold idea was taken from here. read

解决方案

It seems that problem #1 was solved by adding following lines in my web.xml under the <servlet> tag

<multipart-config>
        <location>/tmp</location>
        <max-file-size>1000000000</max-file-size>
        <max-request-size>1500000000</max-request-size>
        <file-size-threshold>0</file-size-threshold>
</multipart-config>

and removed jersey-multipart-config.properties file.

Now I can upload files that are over 200Mb. No more temporary files are created.

But I still can't explain problem #2.

这篇关于Jersey 服务文件上传导致 OutOfMemoryError的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！

查看全文
相关文章
其他开发最新文章
热门教程
热门工具
登录 关闭
扫码关注1秒登录
发送“验证码”获取 | 15天全站免登陆