Java 堆空间不足以在 AWS S3 上上传文件 [英] Java Heap Space is insufficient to upload files on AWS S3

问题描述

我正在尝试使用 Java-AWS API 在 AWS S3 上上传文件.问题是我的应用程序无法上传大型文件，因为堆已达到其限制.错误:java.lang.OutOfMemoryError:Java 堆空间

I'm trying to upload a file on AWS S3 by using Java-AWS API. The problem is my application is unable to upload large sized files because the heap is reaching its limit. Error: java.lang.OutOfMemoryError: Java heap space

我个人认为扩展堆内存不是永久的解决方案，因为我必须上传高达 100 GB 的文件.我该怎么办?

I personally think extending heap memory isn't a permanent solution because I have to upload file upto 100 gb. What should I do ?

这是代码片段:

        BasicAWSCredentials awsCreds = new BasicAWSCredentials(AID, Akey);
        AmazonS3 s3Client = AmazonS3ClientBuilder.standard()
        .withRegion(Regions.fromName("us-east-2"))
        .withCredentials(new AWSStaticCredentialsProvider(awsCreds))
        .build();

        InputStream Is=file.getInputStream();

        boolean buckflag = s3Client.doesBucketExist(ABuck);
        if(buckflag != true){
           s3Client.createBucket(ABuck);
        }
        s3Client.putObject(new PutObjectRequest(ABuck, AFkey,file.getInputStream(),new ObjectMetadata() ).withCannedAcl(CannedAccessControlList.PublicRead));

推荐答案

我强烈建议在 ObjectMetadata 上使用 setContentLength()，因为:

I strongly recommend to setContentLength() on ObjectMetadata, since:

..如果未提供，库将必须缓冲输入流的内容以进行计算.

..If not provided, the library will have to buffer the contents of the input stream in order to calculate it.

(..这会导致足够大"的文件出现 OutOfMemory.)

(..which predictably will lead to OutOfMemory on "sufficient large" files.)

来源:PutObjectRequest javadoc

应用于您的代码:

 // ...
 ObjectMetadata omd = new ObjectMetadata();
 // a tiny code line, but with a "huge" information gain and memory saving!;)
 omd.setContentLength(file.length());

 s3Client.putObject(new PutObjectRequest(ABuck, AFkey, file.getInputStream(), omd).withCannedAcl(CannedAccessControlList.PublicRead));
 // ...

这篇关于Java 堆空间不足以在 AWS S3 上上传文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！