通过流将多部分文件上传到Amazon S3时内存使用率很高? [英] High memory usage when uploading a multipart file to Amazon S3 via streaming?

问题描述

我的Java Spring应用程序中下面的方法直接流式传输文件并将其上传到Amazon S3存储桶.我研究过，使用流将使大文件(对于我的用例而言，> 100MB视频)的上传更加节省内存.当使用25MB的文件测试该方法时，我的Java Spring应用程序在Kubernetes集群设置中的内存使用量增加了200MB！我还尝试了一个200MB的文件，内存再次飙升至2GB.没有引发内存不足异常，但内存使用率并未下降.为什么会这样?

The method below in my Java Spring application directly streams and uploads a file to an Amazon S3 bucket. I have researched that using streams will make the uploading of large files (> 100MB videos for my use case) to be more memory efficient. When testing the method with a 25MB file, the memory usage of my Java Spring application in a Kubernetes cluster setup spiked up by 200MB! I also tried a file that was 200MB and the memory spiked up again to ~2GB. There were no out of memory exceptions thrown but the memory usage does not drop back down. Why does this happen?

public void uploadFile(MultipartFile file, String saveFileName) {
        try {
            ObjectMetadata metadata = new ObjectMetadata();

            if (file.getContentType() != null){
                om.setContentType(file.getContentType());
            }

            metadata.setContentLength(file.getSize());

            saveFileName = saveFileName.replaceAll(" ", "");

            InputStream stream = file.getInputStream();

            PutObjectRequest request = new PutObjectRequest(bucketName, saveFileName, stream, metadata);
            request = request.withMetadata(om);

            s3client.putObject(request);

            stream.close();
        } catch (AmazonClientException | IOException exception) {
            // handle exception
        }
    }

推荐答案

处理大文件上传有多种方法.

There are multiple ways to handle large file uploads.

1. 使用后台任务将字节数组写入磁盘并上传到S3 也许.
2. 存储在内存中并直接上传对象(不好，不好) 选项，除非您设置的文件上传限制很低)

1. Write byte array to disk and upload to S3 using a background task maybe.
2. Store in memory and upload the object directly (bad bad option, unless you set a very low file upload limit)

看看此 git repo，了解如何实现上述方法

take a look at this git repo on how the above methods can be achieved

我在这里看不到您的用例.但是，如果您还要处理UI，请考虑使用

I don't see your use case here. But if you are handling the UI as well consider uploading the files directly from the UI using pre-signed S3 URLs.

这篇关于通过流将多部分文件上传到Amazon S3时内存使用率很高?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！